Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
OOLONG introduces a benchmark testing how models reason over massive amounts of text by analyzing individual chunks and aggregating those analyses to answer complex questions. It has two parts: OOLONG-synth uses synthetic data for controlled testing, while OOLONG-real uses real conversational data. The study shows that models struggle significantly with this task as context length increases.