EngTrace: A Symbolic Benchmark for Verifiable Process Evaluation of Engineering Reasoning
EngTrace is a symbolic benchmark designed to rigorously evaluate Large Language Models (LLMs) for safety-critical engineering tasks. It generates 1,350 test cases with gold-standard reasoning traces to stress-test generalization across diverse physical scenarios. The methodology verifies reasoning through automated checks and a multi-model tribunal, revealing that frontier models struggle with arithmetic execution while open-weights models fail conceptually.