Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark
Enrico Mensa, Lorenzo Zane, Calogero Jerik Scozzaro, Matteo Delsanto, Tommaso Milani, Daniele Paolo Radicioni
cs.CL, cs.AI
Submitted: 2026-08-05
Journal ref: Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), pages 722-734
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) have transformed computational linguistics and achieved remarkable performance across numerous natural language processing tasks, yet significant gaps persist in
Terminology
Abstract
Large Language Models (LLMs) have transformed computational linguistics and achieved remarkable performance across numerous natural language processing tasks, yet significant gaps persist in understanding how these systems process culturally embedded linguistic expressions. This paper introduces ProverbIT, a novel Italian benchmark comprising 100 multiple-choice questions designed to evaluate LLMs' ability to complete Italian proverbs. We assess 13 frontier models, including Large Reasoning Models (LRMs) and traditional LLMs, across three tasks: proverb completion, multiple-choice selection with correct answers, and multiple-choice selection without correct answers. Our evaluation reveals surprising results: while nearly all models demonstrate knowledge of the proverbs through successful completion tasks, performance drops dramatically when transitioning to multiple-choice formats without correct answers, with even state-of-the-art reasoning models showing substantial degradation. Through detailed Chain-of-Thought analysis of two LRMs, we uncover that models exhibit a strong bias toward selecting literal synonyms and frequently mention correct proverb endings during reasoning without successfully identifying their absence from the given options. These findings suggest that current LLMs rely heavily on memorized patterns rather than deeper semantic understanding of culturally grounded expressions, highlighting important limitations in their reasoning capabilities for figurative language comprehension.
Sources
- Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models
- The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Training Verifiers to Solve Math Word Problems
- FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- On the Measure of Intelligence
- ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems
- Evalita-LLM: Benchmarking Large Language Models on Italian
- Memorization or Reasoning? Exploring the Idiom Understanding of LLMs
- Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Model
- GPT-4o System Card
- DeepSeek-V3 Technical Report
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Qwen3 Technical Report
- Gemma 3 Technical Report
- Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
- None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks
- Reasoning Models Don't Always Say What They Think
- "My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering