FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India

The paper addresses a critical barrier to social benefit access in India: In India, almost every social benefit starts with a form, yet the people who need these benefits most are often unable to read or write. Reaching them requires a spoken conversation. The authors note that more than half the women in the lowest wealth quintile cannot read at all (citing NFHS-5), so enrollment falls to frontline health workers who sign up beneficiaries one conversation at a time. This is an inefficient use of their capacity, as Documentation already consumes much of their day and dozens of overlapping reporting systems add to this burden, requiring the same data to be re-entered across apps and paper registers. This caps enrollment at worker capacity rather than need.

The paper frames the opportunity: Voice-based services have already demonstrated population-scale reach in low- and middle-income countries (LMICs). For example, the outbound prerecorded-call service Kilkari reached more than 10 million subscribers. LLM-powered voice agents now make real-time spoken interaction practical, creating opportunities where A mother can call a number and enroll by speaking naturally with an agent in her own words and language, with nothing to read, type, or install.

The authors identify why this is technically hard: "Voice agents work well with a cooperative speaker in a quiet room, but performance degrades with background noise, telephone-channel distortion, underrepresented regional accents, unusual speaking rates, mispronunciations, and disfluencies. Additional challenges include Hindi-English code-mixing, structured values like phone numbers and dates, which can be spoken in many ways but must resolve to a single value, and the compounding nature of form-filling calls: callers hesitate, they correct themselves mid-answer, and the form branches on earlier answers, so an incorrectly captured field can send the agent down the wrong path."

The authors "built FormBharo ('fill the form' in Hindi), a voice agent that fills a structured form over a phone call under tight latency and cost budgets by pairing Large Language Models (LLMs) with deterministic, rule-based validation and flow control. The design principle is stated clearly: The design principle is to use LLMs only where they add value: interpreting user inputs, turning unstructured answers into structured fields, phrasing the next question naturally to be spoken back to the user, and deciding when the form is complete. Everything else stays rule-based: validation, retries, branching, and choosing the next question."

The architecture works as follows: "Once the Voice Activity Detection (VAD) model detects that the caller has stopped speaking, an STT model transcribes the audio. The EXTRACT LLM reads the transcript together with the conversation history and extracts every field answered in that turn, so the agent does not re-ask questions already answered. The form has fields of several types: free text, number, date, boolean, and categorical. For categorical fields, EXTRACT returns the index of the chosen option rather than the option text. In parallel, EXTRACT generates a short acknowledgement that the TTS model speaks back to reduce perceived latency."

The rule-based layer is central to robustness: "The rule-based layer validates each extracted value against field-specific built-in guards, such as a specified minimum length, 10 digits for a phone number, or a date restricted to the past. When a value is missing or fails validation, it checks whether the maximum number of retries for that field is reached. If retries remain, the validation error is passed to REPLY LLM along with the conversation history, which asks the question again. Once retries are exhausted for a required field, the rule-based layer ends the call. EXTRACT also detects when the caller declines or does not know the answer to an optional field (the skip flag). The rule-based layer also handles branching: The form has several branches conditioned on the values of earlier fields. For example, gestational age is asked only if the caller says she is pregnant... This decision is injected into the conversation history of REPLY LLM as a tool call. A second LLM, REPLY, phrases the question naturally for the TTS model to speak back to the caller, or ends the call if the form is complete. Interruptions are handled: if the user starts speaking while the agent's inference is running, the current turn is terminated immediately."

The system is being piloted in rural Maharashtra, India, enrolling low-income, Hindi-speaking mothers in an antenatal and postnatal care program in collaboration with ARMMAN, a nonprofit in India that operates large-scale mobile-health programs for maternal and child health among underserved communities.

The enrollment form used in the pilot has 12 fields (Table 1): Name, District, Clinic name, Pregnant?, and conditionally Gestational age (if pregnant) or Child's name and Child's DOB (if not), Phone linked to clinic?, conditionally Number linked to clinic, Phone linked to WhatsApp?, conditionally WhatsApp number, and Aadhaar last 4 digits. On repeated failure, the agent either skips to the next field (skip) or ends the call (end).

The authors "openly release FormVoiceAgentBench, a new benchmark pairing human-recorded Hindi audio with 3,760 multi-turn conversation tests across 960 simulated calls, built on the enrollment form from our pilot without exposing any real caller's data." The benchmark statistics (Table 2): 5 simulated users, 12 form fields, 4 acoustic conditions per utterance, 380 unique audio recordings, 240 calls from unique call scripts, 960 calls with acoustic variations, 3,760 unit tests total (1,880 for EXTRACT and 1,880 for REPLY).

The benchmark construction: "We define five simulated user profiles, each with fixed personal details... Each form field has three representations: the value, the desired form entry and the ground truth for measuring form completion ('Sunita Devi'); the reference transcript, a naturally spoken rendering of the value, phrased as a caller would say it in a real conversation ('My name is Sunita Devi'); and the recording of that transcript. Values and transcripts are generated sequentially by an LLM, under field-specific constraints defined with ARMMAN: names and district names with phonetically hard spellings, and numbers with many spoken variations that make them hard to transcribe. District names were deliberately selected from outside the pilot state for adequate stress-testing." For optional fields, two linguistic variants are generated: one with the value, one where she says she does not know it. OpenAI's GPT-5.5 was used for generation.

Audio collection details: "Each transcript was recorded to mimic the conditions in a public clinic across three acoustic variations: background noise (ambient chatter and nearby speakers), microphone distance (close or far, chosen at random), and speaking pace (fast or slow, chosen at random). Combined with the ideal acoustic condition, this gives four recordings per transcript. Five native Hindi speakers were selected to match the target demographic: all female, aged 18–35, drawn from two states (Uttar Pradesh and Maharashtra) to cover differences in accent and colloquialisms. Human supervisors listened to every clip to ensure the recordings met the requirements."

Call assembly: "We assemble each call by stitching together a simulated user's transcripts with one recording per field. Since the form branches, a simulated user can follow several paths, and each path becomes its own call... Each call uses a single acoustic condition throughout (e.g. noisy environment or speaking slowly), since a caller's environment does not change midcall. Each simulated user follows 2 × 3 × 4 × 2 = 48 distinct paths: two pregnancy branches (pregnant or not), three branches for whether the calling number is linked to the clinic (linked, not linked and correct value provided, or not linked but not able to remember the linked number), four choices for whether the WhatsApp number is the same as the calling number (same, not same and answered, not same but unable to recall the WhatsApp number, cannot recall if they are the same), and two branches for the Aadhaar digits (answered or not known)." This yields 5 × 48 = 240 calls, and ×4 acoustic conditions = 960 calls.

Unit tests evaluate components in isolation. For STT, the paper notes that "Word Error Rate (WER) is commonly used for comparing STT models. However, it is a poor metric for transcription quality on Indic languages and for agents: it counts every surface difference as an error, even when the meaning is unchanged. They therefore also report LLM-WER... which discards mismatches that an LLM classifies as semantically equivalent or phonetically similar and recomputes WER over the genuine errors."

For EXTRACT: For each turn, we create a unit test using the latest user response paired with the preceding conversation history as input and the expected form values as the ground truth. Extracted values for "closed-ended fields like numbers, dates, and booleans are scored by computing the exact match with the expected values. The skip flag is scored the same way. Outputs for open-ended fields like names are harder to rate since the expected value can have many phonetic forms ('Lakshmi' versus 'Laxmi'), so we use a calibrated binary LLM judge." Deduplication yields 1,880 unit tests.

For REPLY: unit tests are prepared similarly but "REPLY additionally receives a tool call carrying the decision of the rule-based layer as an input too... For the unit tests, we construct this tool call by passing the expected extraction values through the rule-based validation and flow-control step, so that measuring response accuracy can be isolated from extraction errors. REPLY is graded across five independent dimensions: 1) asking the right question, 2) adherence to Hindi, 3) single line response, 4) no acknowledgment (since EXTRACT handles that), and 5) not echoing the caller's answer back. This yields 1,880 unit tests: 920 require a reply, while the remaining 960 check whether REPLY ends the call correctly."

Integration tests chain the components, so the output of one feeds into the next, and the agent is judged on completing the form, not on any single turn. First, reference transcripts are replaced with transcripts produced by the STT models, so transcription errors propagate to the LLMs. Then each call is evaluated as a whole: "We start from an empty form and go through the call's turns in order. At each turn, we write the values extracted for that turn's unit test into the form, as the agent would in a live call. The form after the last turn is the predicted final form. Comparing this against the expected values gives the form-completion accuracy: the fraction of form fields captured correctly at the end of the call. Finally, EXTRACT and REPLY are chained: To build the tool call for REPLY, we pass EXTRACT's actual output through the rule-based validation and flow control, instead of the expected extraction values used in the unit tests. Thus extraction accuracy is measured for every STT × EXTRACT pair, and response accuracy for every STT × EXTRACT × REPLY combination."

The paper benchmarks five STT models and 11 LLMs across accuracy, latency (p95), and cost. The temperature is set to 0 for non-reasoning models. The reasoning models use 'medium' reasoning effort for EXTRACT and 'low' for REPLY.

Speech-to-Text. The two metrics disagree: Nova-3 has the best WER yet the second-worst LLM-WER. Ranking by LLM-WER, Chirp 3 has the best LLM-WER but costs at least twice as much as any other model, while Scribe v2 performs close to Chirp 3 at a fraction of the cost. GPT-4o-transcribe is the least accurate. The authors eliminate those two, leaving Scribe v2, Saaras v3, and Nova-3 for integration tests.

Extraction accuracy. "Frontier models saturate extraction accuracy on reference transcripts as inputs with the top models scoring almost perfectly: GPT-5.5 leads at 99.79%, with Gemini 3.5 Flash (99.36%) and the two Claude models (99.15%) just behind. Robustness to STT errors varies: the median drop is modest, from 0.42 percentage points to 3.14. But weaker models degrade much more: GLM-5.1 collapses by 35 points with Scribe v2 transcripts. Importantly, The extraction accuracy leaderboard changes when real transcripts are used: GPT-5.5 is no longer the winner."

Form completion. The rule-based layer recovers extraction errors. On reference transcripts, Gemini 3 Flash achieves 95.96% per-turn extraction accuracy but 100% form-completion accuracy. The authors explain: "Per-turn extraction evaluates the output of EXTRACT LLM, whereas form completion evaluates the values ultimately stored in the form after the rule-based layer processes it. All the extraction errors for this model arise from a type mismatch: a numeric field being returned as a string. The rule-based layer normalizes it before storing the value, preserving 100% form-completion accuracy. This highlights a benefit of our hybrid design which enables smaller models to perform better end-to-end even if their per-turn inference is not perfect."

Crucially: "The best model end-to-end differs from the best model per-turn. Under reference transcripts, GPT-5.5 leads extraction accuracy (99.79%), whereas Gemini 3 Flash and Gemini 3.5 Flash tie for the highest form-completion accuracy at 100%."

Real-speech transcripts cause larger degradation end-to-end than per-turn: "median form-completion accuracy drops by 7.67 percentage points with Saaras v3, 7.38 points with Scribe v2, and 13.56 points with Nova-3... compared with median per-turn extraction drops of only 0.42–3.14 points. The largest model-specific decline is ∼41 points for GLM-5.1 with Scribe v2. The authors explain: Although the rule-based layer recovers some extraction errors, uncorrected errors can accumulate across turns to produce a larger degradation end-to-end."

The selection is done sequentially because each downstream component consumes the outputs of the components before it.

STT selection: "Scribe v2 provides the strongest downstream performance. It achieves the highest median extraction accuracy (97.29%) and form-completion accuracy (91.84%), while producing the smallest median drops relative to the model performance on reference transcripts."

EXTRACT selection: With Scribe v2 fixed, the deployment constraints are p95 latency below 5 s and form-completion accuracy above 90%. This excludes Gemini Pro, despite its leading form-completion accuracy, and Gemini 2.5 Flash, leaving behind six candidates. Among those, "Claude Sonnet 4.6 achieves the highest form-completion accuracy, Mistral Medium 3.5 has the lowest p95 latency (1.75 s), and GPT-5.4-mini is the cheapest (0.0011). No model leads all three axes, and all six candidates lie on the Pareto frontier. The authors use weighted-sum scalarization: they min–max normalize the three axes, reverse latency and cost scales so that higher values are always preferred," and rank by U = w a·ã + w l·l̃ + w c·c̃.

"Under equal weights, Mistral Medium 3.5 ranks highest (U = 0.81)... Mistral combines the lowest latency with 91.84% form-completion accuracy and a cost of 0.0080 per turn. Claude Sonnet 4.6, the most accurate candidate, scores lower (U = 0.51) because it is slower and more expensive. The authors assign latency a lower weight (w l = 0.1) since The hard latency constraint already excludes models that are too slow for deployment. Sweeping w a from 0.5 to 0.9, Only two models lead across this range: Gemini 3.5 Flash for 0.50 ≤ w a ≤ 0.61, and Claude Sonnet 4.6 for 0.62 ≤ w a ≤ 0.90. The authors select Gemini 3.5 Flash: it trails Claude Sonnet 4.6 by only 0.51 percentage points on form-completion accuracy while responding faster and costing less than half as much p…

Based on FormBharo, here are concrete improvements to AI voice-agent systems and what the improved system can do.

• Use a hybrid architecture instead of an end-to-end LLM pipeline: LLMs only interpret user speech into structured fields and generate natural replies; rule-based code handles validation, retries, branching, and form-completion decisions.

• Extract every field answered in a single user turn from conversation history, not just the field associated with the last question. Return categorical fields as option indices, and generate a spoken acknowledgement in parallel to reduce perceived latency.

• Detect and honor skips: if the caller says “I don’t know” or declines an optional field, set a skip flag instead of re-asking until failure.

• Add field-specific deterministic validators: 10-digit phone numbers, dates constrained to the past, Aadhaar last-4 digits, minimum name lengths, and exact option-list membership. On invalid input, retry up to a field-specific limit; then either skip the field or end the call if the field is required.

• Encode form branching as tool calls injected into the reply LLM’s context. For example, only ask gestational age after a “pregnant” answer, and only ask clinic number after “phone is linked to clinic.”

• Evaluate STT models with LLM-WER instead of raw WER for Indic languages, because WER penalizes harmless phonetic differences. Select the STT model based on downstream extraction/form-completion accuracy rather than transcription similarity.

• Split component selection: choose STT first, then EXTRACT, then REPLY. Apply hard constraints (e.g., p95 latency 90%), then use a Pareto/weighted-sum ranking across accuracy, latency, and cost.

• Use multi-turn integration tests with the STT output feeding the LLM chain, measuring final form-completion accuracy instead of per-turn extraction accuracy. This captures error accumulation and also reveals when rule-based normalization cancels errors.

• Build the benchmark from human-recorded audio of simulated users covering every form branch, with acoustic variations (noise, microphone distance, speaking pace) and optional-field “don’t know” responses, without exposing real user data.

• For open-ended fields like names, use a calibrated LLM judge tolerant of phonetic variants (e.g., “Lakshmi” vs. “Laxmi”).

• Support interruption handling: if the user speaks during inference, immediately terminate the current turn and process the new input.

• Let a low-income, non-literate Hindi-speaking mother enroll in a social program by calling a number and speaking naturally, with nothing to read, type, or install.

• Operate reliably from a noisy public clinic: it tolerates background chatter, distant microphones, fast or slow speech, regional accents, and Hindi-English code-mixing.

• Complete a 12-field structured enrollment form conditionally, asking only relevant follow-ups and skipping optional fields when the caller doesn’t know or declines.

• Recover from transcription and extraction errors via rule-based normalization, allowing smaller, cheaper models to match frontier-model accuracy end-to-end.

• Meet deployment budgets: form completion accuracy above 90% with p95 latency below 5 seconds and per-turn cost low enough for LMIC-scale deployment.

• Avoid re-asking answered fields, confirm captured values with natural acknowledgements, retry invalid inputs up to a safe limit, and gracefully end the call when required fields still fail.

• Scale enrollment beyond frontline-worker capacity by shifting the burden of documentation from health workers to an automated conversational agent, with guardrails in place to confirm or correct data before final submission.