Benchmarking in Neuro-Symbolic AI
Abstract
Neural-symbolic (NeSy) AI has gained a lot of popularity by enhancing learning models with explicit reasoning capabilities. Both new systems and new benchmarks are constantly introduced and used to evaluate learning and reasoning skills. The large variety of systems and benchmarks, however, makes it difficult to establish a fair comparison among the various frameworks, let alone a unifying set of benchmarking criteria. This paper analyzes the state-of-the-art in benchmarking NeSy systems, studies its limitations, and proposes ways to overcome them. We categorize popular neural-symbolic frameworks into three groups: model-theoretic, proof-theoretic fuzzy, and proof-theoretic probabilistic systems. We show how these three categories have distinct strengths and weaknesses, and how this is reflected in the type of tasks and benchmarks to which they are applied.
How to cite
@inproceedings{manhaeve2024benchmarking,
author = {Robin Manhaeve and Francesco Giannini and Mehdi Ali and
Damiano Azzolini and Alice Bizzarri and Andrea Borghesi and
Samuele Bortolotti and Luc De Raedt and Devendra Singh Dhami and
Michelangelo Diligenti and Sebastijan Dumancic and Boi Faltings and
Elisabetta Gentili and Alfonso Gerevini and Marco Gori and
Tias Guns and Martin Homola and Kristian Kersting and
Jens Lehmann and Michele Lombardi and Luca Salvatore Lorello and
Emanuele Marconato and Stefano Melacci and Andrea Passerini and
Debjit Paul and Fabrizio Riguzzi and Stefano Teso and
Neil Yorke-Smith and Marco Lippi},
editor = {Wang-Zhou Dai},
title = {Benchmarking in Neuro-Symbolic {AI}},
booktitle = {Learning and Reasoning - 4th International Joint Conference on
Learning and Reasoning, {IJCLR} 2024, Nanjing, China,
September 20-22, 2024, Proceedings},
series = {Lecture Notes in Computer Science},
volume = {16059},
pages = {238--249},
publisher = {Springer},
year = {2024},
url = {https://doi.org/10.1007/978-3-032-09087-4_17},
doi = {10.1007/978-3-032-09087-4_17}
}