From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring
cs.CL, cs.AI
Submitted: 2026-09-24
Updated: 2026-09-24
Comments: Accepted as a full paper to FLINS-ISKE 2026
DOI: 10.1007/978-981-92-2480-7_20
License: http://creativecommons.org/licenses/by/4.0/
The gist: Science, technology, and innovation policies are crucial for competitiveness, yet their diversity and scale make them difficult to map and monitor consistently.
Terminology
Abstract
Science, technology, and innovation policies are crucial for competitiveness, yet their diversity and scale make them difficult to map and monitor consistently. Existing approaches rely heavily on manual survey efforts, which are costly and challenging to scale across countries. Large language models (LLMs) enable new possibilities for extracting and structuring information from long and unstructured policy documents. This paper presents an application of LLMs as "AI respondents" for generating structured survey responses from policy texts. We develop a data extraction pipeline based on long-context in-context learning to map information from public web sources into predefined survey categories, including policy instruments, target groups, and thematic areas. The pipeline integrates a validation step using a secondary LLM to assess relevance and evidence, alongside comparisons with human-provided responses. Using a multi-country dataset, we evaluate the alignment between LLM-generated and human-generated outputs through overlap measures and cross-validation. Results show that LLMs achieve high agreement for structured indicators (84-95%), while differences remain in free-text fields, where models tend to provide more detailed procedural descriptions. These findings highlight the potential of hybrid human-AI workflows for policy monitoring, improving both efficiency and scalability while maintaining the need for human validation and contextual interpretation.
Sources
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- The Llama 3 Herd of Models
- A Survey on LLM-as-a-Judge
- Which is better? Exploring Prompting Strategy For LLM-based Metrics
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- GPT-4o System Card
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
- Large Language Models for Data Annotation and Synthesis: A Survey
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering