From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

The paper presents the Grip on LLMs framework, a systematic evaluation suite for Dutch governmental use of large language models (LLMs), developed in collaboration with domain experts from the City of Amsterdam. The framework identifies six evaluation dimensions—factuality, honesty, social bias, energy consumption, cost, and training data transparency—through an advisory board process, user research, and a survey of 429 chatbot users. The authors evaluate more than 30 multilingual and Dutch-specific models across these dimensions, using benchmarks such as tinyBenchmarks (MMLU, ARC, TruthfulQA) for factuality, a newly developed HONESTCITYBENCH for honesty, Dutch BBQ and BZK Social Bias for bias, and SARI and BERTScore for simplification and summarisation use cases.

Key findings include: no single model excels across all dimensions, and that trade-offs are unavoidable: higher quality consistently comes at greater environmental impact and financial cost, while bias remains largely independent of both. The paper also finds that factuality (whether a model answers correctly) and honesty (whether a model acknowledges what it does not know) are governed by distinct properties, with high factuality not implying high honesty. For example, GPT-5 leads on factuality but scores lowest on honesty, while Mistral Small 24B combines top-tier factuality with the joint-highest honesty score. The authors release a publicly accessible, user-friendly model overview designed for non-technical audiences, available at https://amsterdam.github.io/grip-on-llms/nl/, to make these findings actionable for stakeholders from engineers to policymakers.

Improvements to AI systems:

1. Implement a dual-factuality-honesty calibration module. Current LLMs optimize for answer correctness but not for calibrated uncertainty expression. The improved system will have a separate honesty head that is trained to detect and verbalize knowledge gaps (e.g., I don't know or I am uncertain) independently of the factuality head. This prevents high-factuality models from overconfidently hallucinating on out-of-distribution queries.

2. Add a dynamic trade-off governor for deployment. The system will expose a configurable resource budget (energy and cost) that automatically selects the optimal model variant per query type. For low-stakes tasks (e.g., summarization), it routes to a small, efficient model; for high-stakes factual queries (e.g., legal or medical), it escalates to a larger model—but only after checking the honesty score of the small model to avoid silent failures.

3. Integrate a bias-aware response filter. The improved system will run every generated response through a real-time bias detector (trained on Dutch BBQ and BZK Social Bias benchmarks) that flags and rephrases outputs containing stereotypical associations, independent of the model's factuality. This ensures bias mitigation is applied post-hoc, since bias is shown to be orthogonal to quality.

4. Build a transparency-aware training data audit tool. The system will include a lightweight, queryable metadata layer that reports whether a given response relies on training data that is publicly documented, licensed, or unknown. This allows users to reject outputs derived from opaque data sources, directly addressing the training data transparency dimension.

5. Enable energy-aware response generation. The system will estimate the energy cost of each inference (based on model size and token count) and, for non-critical requests, automatically switch to a distilled or quantized version that reduces energy consumption by up to 40% without significant factuality loss, as per the paper's trade-off findings.

What the improved AI system can do:

• For a citizen asking a Dutch government chatbot about a tax rule: It will answer factually if confident, but if uncertain, it will explicitly say I am not sure and provide a link to a human expert, rather than guessing.

• For a policymaker querying a large document corpus: It will use a small model for summarization (saving energy/cost) but automatically escalate to a high-factuality model for any numeric or legal claims, with the honesty module flagging any unverifiable statements.

• For a city employee drafting public communications: It will automatically rephrase any biased or stereotypical phrasing (e.g., gender or ethnic bias) before output, regardless of the base model's bias score.

• For an auditor: It will generate a per-response data provenance card showing whether the training data for that response is transparent, partially transparent, or unknown, enabling compliance with EU AI Act transparency requirements.

• For a system administrator: It will provide a live dashboard of energy consumption per query and automatically switch to a low-power mode during off-peak hours, balancing cost and performance based on the trade-off curves from the paper.

Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of public administration and the linguistic requirements of non-English contexts. We present the "Grip on LLMs" framework, a systematic evaluation suite for Dutch governmental use developed in collaboration with domain experts from a major Dutch municipal organisation. Through an advisory board process, user research, and a survey of the users of a civil-servant chatbot, we identify six evaluation dimensions (factuality, honesty, social bias, energy consumption, cost, and training data transparency) and operationalise them into a benchmark suite covering more than 30 multilingual and Dutch-specific models. Our results reveal that no single model excels across all dimensions, and that trade-offs are unavoidable: higher quality consistently comes at greater environmental impact and financial cost, while bias remains largely independent of both. We further find that factuality (whether a model answers correctly) and honesty (whether a model acknowledges what it does not know) are governed by distinct properties, with high factuality not implying high honesty. To make these findings actionable for non-technical audiences, we release a publicly accessible, user-friendly model overview designed for the full range of stakeholders involved in governmental LLM selection, from engineers to policymakers.