Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval

cs.LG, cs.AI, cs.CL, cs.PL

Submitted: 2026-10-01

Updated: 2026-10-01

Comments: 30 pages (preprint)

Code: https://github.com/Nickil21/AlgoREval

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Sources

• Program Synthesis with Large Language Models
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs
• The Llama 3 Herd of Models
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• The Stack: 3 TB of permissively licensed source code
• StarCoder 2 and The Stack v2: The Next Generation
• Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibility
• Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Where LLM Agents Fail and How They can Learn From Failures

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks