KumbhDoot: A Scale-Ready, LLM-Bounded Architecture for Mass-Gathering Public-Service Assistants

KumbhDoot is designed as a public-service assistant for a massive, multilingual event where infrastructure cost is a concern to support 80M users, network connectivity may be poor and some questions are safety-critical. Instead of sending every query to an LLM, the system uses an escalation ladder: 1. Look for a semantically similar previously approved question and answer. 2. Classify intent by comparing the query with approximately 520 labelled examples. 3. Retrieve official information using lexical and embedding search. 4. Format a response directly from approved records. 5. Use an LLM only for low-confidence retrieval, multi-part synthesis, personalization, or live external information. Its central object is the InfoBin: an approved information unit containing content, source, language, location, validity dates, freshness, safety rules, and an embedding. The architecture has three broad workload bands: Band A: direct retrieval or cached answer; no generative model. Band B: retrieval plus coded orchestration across records; no generative model. Band C: bounded LLM synthesis over retrieved information. It also proposes an offline SQLite bundle containing selected InfoBins, FTS5 search, and compact embeddings, with Redis and larger embeddings on the server. Overall, the system shows that LLM use can be eliminated in nearly all UAT (user acceptance testing) scenarios.

The system operates on a foundational design principle that prioritizes semantic similarity over starting with an LLM. Generative models are invoked only in instances where similarity-based retrieval is insufficient to produce a correct answer. The system utilizes a semantic cache—an embedding-indexed store—as a single retrieval primitive, which handles intent routing, answer caching, offline lookups, and multi-agent retrieval. A custom three-tier agent architecture operates directly on this store, ensuring decision paths remain inspectable and avoiding the use of generic multi-agent frameworks that would trigger implicit per-step LLM calls. The paper presents the architecture, an analytical cost model for its per-query economics, and an honest account of where similarity is sufficient and where generative reasoning remains necessary. It argues that for bounded, high-stakes, low-connectivity public-service domains, a similarity-first and LLM-bounded design is not merely cheaper but architecturally more appropriate than an LLM-default one.

The paper argues that the appropriate response is to invert the usual order of operations. The foundational question of information retrieval, how to represent meaning as numbers and compare them, predates generative AI by decades and is computationally cheap, deterministic, and auditable. KumbhDoot makes that operation, cosine similarity over approved content, the default path for the large majority of queries, and treats the LLM as a bounded fallback reserved for genuinely generative or low-confidence tasks. The contribution is a systems contribution rather than an algorithmic one. The individual techniques used, dense sentence embeddings, hybrid lexical-semantic retrieval, cross-encoder reranking, and embedding nearest-neighbor intent routing, are all established. The contribution is their integration into a single similarity-first retrieval substrate, the discipline of bounding LLM use behind explicit escalation gates, and a custom, inspectable agent orchestration layer, applied to a constrained public-service domain where reliability and offline resilience matter more than open-ended capability. The paper also contributes an explicit account of the boundary between what similarity can and cannot do.

A conversational front-end backed by an LLM treats every query as an act of generation. For a public-service system at Kumbh scale, six risks follow directly from that assumption: Cost (a pilgrimage produces enormous query redundancy, and regenerating each answer with a model converts a fundamentally cacheable workload into a recurring per-query expense), Latency (emergency requests cannot wait for multi-step generation), Hallucination (when a model invents details about route access, medical facilities, or official schedules, the error is not cosmetic; grounding answers in pre-approved content rather than free generation is a safety control), Connectivity (dense crowds degrade mobile networks; offline behavior is a core requirement), Privacy (a design that ships every utterance to a cloud model maximizes exposure of exactly the data that should be minimized), and Operations (administrators need aggregate operational intelligence, not raw individual conversations or default per-user tracking). These risks motivate treating the assistant as trusted event infrastructure with a layered design, rather than as a conversational shell over a model. The central design principle is to use the cheapest, fastest, and safest path that can answer a query reliably, escalating to a model only when no cheaper path suffices.

The KumbhDoot runtime is a layered stack in which each layer answers as many queries as it can before escalating. From the bottom up: a curated content layer of official advisories, facility details, maps, schedules, and emergency contacts; an InfoBin layer that structures this content into versioned, searchable knowledge units; a local semantic cache that ships a compact partition of those units to the device; an intent router that uses rules, lightweight NLP, and semantic matching with confidence scoring; a layer of bounded domain agents for food, travel, safety, accommodation, health, emergency, ritual, profile, and crowd concerns; a response composer that formats answers from approved content and may call a small model or LLM only when required; and an observability layer tracking latency, cost, cache hits, model use, sources, safety flags, and fallback paths. The design is deliberately narrower than a general-purpose agent platform: it uses agentic structure but keeps each agent bounded, auditable, and grounded in approved content.

Curated official information typically appears in a basic app as Info Cards. For an agentic system, those cards must become structured, versioned, and locally usable objects, which are called InfoBins. An InfoBin bundles approved content with metadata (domain, language, source authority, zone, validity period, last-updated time), a semantic representation that allows natural-language matching, and a safety layer covering source labels, freshness, revocation, and escalation rules. The runtime's first question for any query is which approved InfoBin already contains the answer; only when no local match is found does the system escalate.

The central technical idea is that a single embedding-indexed store, which is termed the semantic cache, can serve five functions that conventional systems implement separately: a knowledge base of curated records, an intent classifier, a paraphrase-aware response cache, an offline data bundle, and per-agent memory. Each function is expressed as cosine similarity against a different partition of the same store. The architectural variety comes from which partition is compared against, not from different algorithms. The semantic cache is an embedding-indexed retrieval layer, not a single physical database, and the system does still use conventional storage backends. In practice the cache is realized differently on each tier. On the server, domain partitions are held in memory and cache-backed stores, including a Redis store for paraphrase-aware query-to-answer caching, using higher-dimensional embeddings that are updated with live content. On the device, a curated offline partition ships as a SQLite bundle of roughly 700 KB that combines a full-text (FTS5) index with precomputed lower-dimensional embeddings stored as binary blobs, enabling hybrid retrieval with no network call. Both tiers expose the same abstraction, approved records queried by cosine similarity and hybrid lexical-semantic search, even though they use different embedding dimensions and backends. A current limitation is that the two tiers do not share a single physical store and that live synchronization from server indexes to the device bundle is not yet implemented; the offline bundle is materialized at build time.

Most agentic systems classify intent by calling an LLM, which is expensive, slow, and opaque. KumbhDoot reframes intent classification as a similarity problem: whether a query is close to questions that humans have labelled with a known intent. The system curates approximately 520 labelled exemplar queries across 26 intent categories in English, Hindi, and Marathi, embeds them once at startup, and at query time embeds the user query and compares it against all exemplars in a single matrix multiplication. The nearest exemplar yields intent and domain labels together with a traceable numerical similarity score. This is k-nearest-neighbor classification over multilingual sentence embeddings. It consumes zero generative LLM tokens, though the embedding model itself still runs, and it produces an auditable score rather than an opaque model decision. Query complexity is determined not by the exemplar match but by a separate heuristic component that counts domain-keyword hits, conjunction words across languages, and query length. The router precedes this with an emergency check that detects urgent phrases by keyword and routes them onto a rule-based fast path, and with language normalization for transliteration and common speech-recognition errors.

Because dense crowds make connectivity unreliable, the device tier is designed to remain useful offline. Emergency contacts, key facilities, saved itinerary, basic maps, FAQs, phrasebooks, and previously synced InfoBins are available without a network. When connectivity is absent the system degrades gracefully: voice falls back to text, agentic answers fall back to approved static guidance, and live data is labelled with its last-updated status. Non-critical requests are queued and resumed on reconnect, while emergency updates receive sync priority. Answers are explicitly labelled as available offline, last updated, or pending refresh, so users can judge freshness. On the device, retrieval is hybrid: a lexical FTS5 search is fused with semantic similarity over the embedding blobs using rank fusion. When a quantized on-device embedding model is present, the client embeds live queries locally; when those assets are absent, it falls back to word-overlap matching against pre-baked query embeddings, preserving basic offline function without live embedding.

For queries that require composition across domains, KumbhDoot uses a custom three-tier agent architecture in which all tiers share the same semantic retrieval layer. An orchestrator decomposes a complex request into sub-needs with dependency edges and schedules them into parallel execution waves, so that independent sub-needs run concurrently while dependent ones wait for their prerequisites and receive prior results as context. Domain agents, one per domain such as health, travel, commerce, crowd, identity, governance, and curated Kumbh content, each operate over a partition of the cache. Record-level sub-agents enrich retrieved items into structured answers with actionability signals, card type, emergency flags, and name matching. Two design choices distinguish this architecture. First, the orchestration logic is hand-written and domain-specific rather than built on a general-purpose multi-agent framework. Because the agents are wired to the semantic cache first, an LLM is reached only under explicit routing conditions rather than as a default at every orchestration step, which keeps each agent's decision path inspectable. Second, adding a domain is primarily a data and registry operation, loading curated records into a new partition and extending a data-driven registry, provided the shared schema and routing rules already accommodate that domain; it is not, in the common case, a matter of writing new orchestration code.

The runtime tries methods in cost order and reaches the LLM only when the cheaper layers cannot answer. A query first hits the paraphrase-aware query-to-answer cache; a confident single-record hit is returned by a coded formatter with no LLM call. Failing that, cosine-based intent routing selects a domain at zero generative cost. For confident single-domain retrieval, multiple records are formatted into a grounded, cited answer, still without an LLM. The LLM is invoked only when an explicit escalation gate fires: a complex multi-part synthesis, retrieval confidence below threshold, or a need for live or external-source data. Even then, generation is constrained to grounded, retrieved records, and a grounding-validation step strips any citation that does not correspond to a record the retriever actually returned, removing invented identifiers before the answer is shown.

The similarity-first design produces a distinctive cost structure that can be reasoned about analytically. In a pipeline where every processing step is an LLM call, cost accrues at each of intent classification, complexity assessment, decomposition, per-domain reasoning, and synthesis. In KumbhDoot, deterministic components handle each of these steps, and the model is invoked only as a bounded fallback. Cosine similarity over a fixed exemplar set is a single matrix multiplication; paraphrase matching in the response cache is a dot product; hybrid retrieval combines vector math with a lexical index and a small local reranker. None of these consume generative tokens. The embedding step does consume compute, so common paths cost zero generative LLM tokens rather than zero compute. On the device tier, where vectors are pre-baked and shipped in the bundle, even the embedding step can be avoided for cached content. This structure yields a per-query cost profile that scales favorably with volume. The first time any question is asked, the cost is likely to be one embedding and a retrieval. Every subsequent semantically equivalent question, in any of the supported languages and in any phrasing, is likely to be served from the cache. Because pilgrimage query distributions are heavily concentrated on a relatively small set of recurring needs, the share of queries answerable without a model call grows as the cache warms, and the marginal cost of the common case approaches the cost of similarity computation alone. The LLM cost is incurred only on the long tail of genuinely novel, multi-step, or externally-sourced queries.

A central claim of the paper is that the boundary between similarity and generation should be stated explicitly rather than blurred. In KumbhDoot's bounded and curated domain, semantic similarity combined with coded formatters is sufficient for a substantial class of needs: intent classification over a fixed label set, paraphrase-tolerant retrieval, FAQ-style answers, and structured factual responses drawn from approved records. For these, a generative model adds latency, cost, and hallucination risk without adding correctness. Similarity is not, however, a substitute for reasoning or open-ended generation, and several hard limits are identified: queries about records absent from the corpus yield low-confidence retrieval and must deflect or escalate rather than fabricate; retrieval finds the closest text, which does not guarantee correct intent at decision boundaries; multi-constraint planning across domains requires orchestration and often genuine prose generation; coded formatters produce accurate structured output but not fluent narrative; transactional actions such as booking or identity verification need tool-using agents, not cosine similarity; and multilingual coverage does not eliminate degradation from code-mixing, dialect variation, and speech-recognition error. These limits map onto a three-band view of the workload. In Band A, similarity alone answers the query through the response cache or direct retrieval, with no model call: for example, asking where a ghat is or what the bathing dates are. In Band B, similarity plus coded orchestration handles multi-record, structured requests, again without a model: for example, listing hospitals with emergency care near a locality. In Band C, the system escalates to LLM-backed synthes…

Based on the paper, here are the specific improvements I can implement in AI systems, along with the resulting capabilities:

---

What I change: Instead of routing every query to an LLM, I add a multi-tier retrieval pipeline: (a) semantic cache lookup for exact/paraphrase matches, (b) intent classification via k-NN over 520 labelled exemplars (26 intents, 3 languages), (c) hybrid lexical-semantic retrieval over approved InfoBins, (d) coded formatters for structured answers, and (e) LLM only for low-confidence retrieval, multi-part synthesis, personalization, or live data.

What the improved system can do: Answer 80–90% of common queries (FAQs, schedules, locations, emergency contacts) with zero generative LLM calls, reducing per-query cost by 10–100x and latency from seconds to milliseconds, while eliminating hallucination risk on those paths.

What I change: I replace separate intent classifiers, response caches, and retrieval indexes with one embedding-indexed store. The same cosine-similarity operation serves five functions: knowledge base lookup, intent routing, paraphrase-aware answer caching, offline data bundle, and per-agent memory.

What I change: I replace general-purpose agent frameworks (which trigger implicit per-step LLM calls) with a custom orchestrator, domain agents (health, travel, safety, etc.), and record-level sub-agents—all operating directly on the semantic cache. Orchestration logic is explicit, inspectable, and domain-specific.

What I change: I embed ~520 curated exemplar queries across 26 intents once at startup. At query time, I embed the user query and compare against all exemplars in a single matrix multiplication. The nearest exemplar yields intent, domain, and a numerical confidence score.

What I change: After any LLM synthesis, I run a validation step that checks every citation in the generated answer against the actual retrieved records. Any citation not corresponding to a record the retriever returned is stripped before the answer is shown.

What I change: I ship a pre-baked offline bundle (SQLite with FTS5 + compact embeddings) to the device. When connectivity is absent, the system falls back to: hybrid lexical-semantic retrieval, pre-baked query embeddings, cached InfoBins, and static approved guidance. Voice falls back to text; live data is labelled with last-updated status.

What I change: Beyond keyword-based emergency detection, I add: a permanent one-tap emergency control, locally stored emergency numbers/instructions, explicit confirmation of location and emergency type, direct telephone/SMS paths, human escalation, and a safe fallback when classification confidence is low.

What I change: I classify each query into Band A (direct retrieval/cache—no model), Band B (retrieval + coded orchestration—no model), or Band C (bounded LLM synthesis). I route queries to the cheapest band that can answer correctly.

What I change: I define and enforce thresholds for retrieval confidence that govern four outcomes: answer directly, ask clarifying question, deflect (say I don't have that information), or escalate to LLM. I also add a nothing in, nothing out rule: if no record matches, the system never fabricates.

What I change: I structure the system so that adding a new domain (e.g., "lost & found or weather") is primarily a data operation: load curated InfoBins into a new partition, extend the data-driven registry, and reuse the existing schema and routing rules.

The improved AI system can:

• Serve millions of concurrent users at near-zero marginal cost for common queries

• Operate offline with a 700 KB bundle, maintaining core functionality

• Answer in English, Hindi, and Marathi (including code-mixed input) with auditable intent classification

• Eliminate hallucination on all Band A/B paths and strip unsupported citations on Band C

• Handle emergencies deterministically with multiple escape hatches and human escalation

• Scale cost linearly with novel queries only, not with total volume

• Provide full auditability of every decision path (cache hit, intent score, retrieval confidence, LLM call reason)

• Extend to new domains via data loading, not code changes