Koa-decision: Fast and Consistent Structured Decision Making with Generative LLMs

The gist The Koa-action framework proposes a method for low-latency atomic actions by formulating classification as constrained generation with single-token outputs, achieving frontier accuracy at sub-second speeds.

How it works

Koa-action treats classification as a constrained generation task with single token outputs to reduce latency to O(1) (Page 2). This is achieved by introducing atomic label tokens and applying supervised fine-tuning (Page 2). The framework defines a set of atomic special tokens, such as [ok], for each class ck, which are augmented to the model's vocabulary (Page 4). The model is trained to map input text and vision directly to the correct special token (Page 4).

Key mechanisms

The core mechanism involves treating classification as a constrained generation task where the model is guided to produce a single special token corresponding to the target label (Page 3.2). This design eliminates multi-token decoding overhead because it guarantees that the output decode[s] as exactly one token (Page 4). The training objective is defined by minimizing a cross-entropy loss over this single prediction, where All preceding tokens (system/user prompts and any in-context descriptions) are masked out of the loss (Page 4).

Randomized label assignments

To prevent memorization of static token–label associations, the method employs randomized label assignments during preprocessing by shuffling the correspondence between classes and control tokens across training instances (Page 3.2). This encourages the model to rely on the contextual descriptions of labels provided in the prompt, rather than memorizing token identities (Page 3.2). Furthermore, this randomization allows for zero-shot adaptation by reassigning label tokens at inference time without retraining (Page 2).

Extension to multi-label classification

The framework extends naturally to multi-label settings by changing the training supervision from single-label cross-entropy to a distribution matching objective (Page 3.3). Instead of supervising toward a one-hot target, the model is trained to match a soft target distribution that spreads probability mass across the control tokens of all labels relevant to an instance (Page 3.3). This allows multi-label prediction in a single forward pass without additional generation steps or architectural changes (Page 3.3).

Performance and validation

Koa-action demonstrates frontier-level accuracy at a fraction of the latency (Page 2). On the production intent-routing benchmark, Koa-action reaches "85.5% accuracy—competitive with the strongest frontier models (Claude-4.8-Opus, Gemini-Pro-3.1) and ahead of GPT-5 and Gemini-2.5-Pro—while answering in about half a second—several-fold faster than every frontier model (up to ∼7.5× at the median) under identical serving conditions (Page 2). The latency is independent of the task," with its median latency remaining around 0.53 s across different tasks like intent-routing, SST-2 sentiment, and voice turn-taking / endpointing (Page 9).

Generalization and efficiency

The framework exhibits strong generalization, maintaining high accuracy across datasets of varying difficulty (Page 14). The P50 latency remains essentially flat as the label space grows, indicating that the design achieves scalable inference efficiency while handling tasks of increasing complexity (Page 14). The system is also robust to changes in label sets at inference time, where accuracy remains stable across random permutations of control-token assignments (Page 15).

Conclusion

Koa-action is a unified single-token classification framework that integrates single-step constrained generation, randomized label assignment, and multimodal inputs into one deployable interface (Page 2). It achieves frontier accuracy at sub-second time with a near-flat tail and taskindependent predictability (Page 5). This combination of frontierlevel accuracy, several-fold lower latency, and taskindependent predictability is what makes the framework practical under real production constraints (Page 5).

1. Koa-action framework allows classification to be treated as a constrained generation task with single-token outputs, which reduces classification to a deterministic one-step decoding problem. This enables an AI system to perform fast, single-step decisions such as classification, semantic endpointing, Boolean checks, and scoring with predictable latency.

2. The system can achieve frontier-level accuracy while maintaining significantly lower latency; specifically on the intent-routing benchmark, Koa-action reaches "85.5% accuracy—competitive with the strongest frontier models (Claude-4.8-Opus, Gemini-Pro-3.1) and ahead of GPT-5 and Gemini-2.5-Pro—while answering in about half a second—several-fold faster than every frontier model (up to ∼7.5× at the median)."

3. The AI system supports zero-shot adaptation to new tasks by reassigning label tokens at inference time, meaning it can handle label additions, merges, and deprecations—common in production intent inventories—a prompt-side edit rather than a retraining event.

4. The framework is robust against token memorization because it employs randomized label assignments during training, ensuring the model rely[s] on the contextual descriptions of labels provided in the prompt, rather than memorizing token identities.

5. The system can handle multi-label classification by training to match a soft target distribution that spreads probability mass across the control tokens of all labels relevant to an instance, allowing it to select the set of labels the model assigns sufficient probability to in a single pass.

Industry applications often demand low-latency classification, yet current large language model (LLM) approaches remain poorly suited for latency-critical applications. Existing prompting and constrained decoding produce verbose, multi-token outputs that require expensive token-by-token generation, while encoder-based models achieve faster inference but sacrifice task flexibility. We propose Koa-decision, a framework for low-latency atomic actions -- fast, single-step decisions such as classification, semantic endpointing, Boolean checks, and scoring -- formulated as constrained generation with single-token outputs. By introducing atomic label tokens and applying supervised fine-tuning, our method reduces classification to a deterministic one-step decoding problem. Across standard benchmarks, Koa-decision delivers competitive accuracy with consistently low and stable latency. On a production intent-routing benchmark, Koa-decision reaches 85.5% accuracy -- competitive with the strongest frontier models (Claude-4.8-Opus, Gemini-Pro-3.1) and ahead of GPT-5 and Gemini-2.5-Pro -- while answering in about half a second, several-fold faster than every frontier model (up to 7.5x at the median) under identical serving conditions. Against the dedicated single-token system Jev/TypeSafe, Koa-decision is competitive on accuracy and faster at the median, while also handling multimodal inputs and multi-label outputs that single-label text systems do not.