Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents

summary

The gist

LLM agents are increasingly used for long-horizon tasks requiring sequential reasoning and tool use, making runtime intervention crucial for improving reliability without retraining.

In short

COTA introduces a tiny auxiliary model that doesn't solve tasks but compares potential actions. Instead of correcting errors, it ranks local alternatives to guide a larger LLM agent during runtime intervention. This shows that simple pairwise comparison is enough for effective, constructive guidance without needing a complex task-solving critic.

Key concepts

Comparison-Only Mechanism: This framework restricts the auxiliary model's job solely to comparing two actions at the same state. It doesn't try to solve the overall problem or generate a full correction. Its only function is to judge which of several proposed actions is locally superior based on a specific quality metric.
Same-Prefix Counterfactuals: This is the training method used to teach the tiny comparator how to compare actions effectively. It involves creating training data by taking an action and then comparing sibling actions from the exact same starting state, ensuring the model learns relative differences between choices.
Constructive Intervention: The goal is for runtime intervention to provide more than just a simple 'stop' or 'go' signal. COTA aims for advice that gives the original agent a useful direction by showing which local alternatives are better, enabling the agent to replan intelligently instead of just receiving a binary warning.
Domination Rate: This refers to the true probability that one action is significantly better than another from a given state. The comparator model is trained to estimate this rate, allowing it to determine if an action proposal is truly poor by checking if enough alternatives defeat it.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 26 episodes
Flow Matching — explained in 22 episodes
Quantum Fisher Information (QFI) — explained in 21 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Sample Complexity — explained in 20 episodes
Federated Learning — explained in 18 episodes

This episode discusses

The paper

Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents · Read on arXiv

National University of Singapore

LLM agents are emerging as an important paradigm for real-world tasks that require reasoning, tool use, and sequential decision-making. As these agents operate over longer horizons, runtime intervention offers a way to improve reliability without retraining the underlying actor. Effective intervention must provide a useful direction for recovery besides a warning. Existing approaches often rely on an expert solver or a critic that generates task-specific corrections, incurring either the cost of another capable solver or the capacity demands of a task-capable critic. We introduce Comparison-Only Tiny Advisor (COTA) for constructive runtime intervention, which reduces the learned intervention role to local action comparison. A lightweight comparator judges the actor's proposal against available alternatives, and preferred alternatives are returned as non-binding advice for replanning. The comparator is trained from same-prefix counterfactual branches. Across WebShop, ALFWorld, and tau 3-Retail with three LLM actors, COTA instantiated with a 0.5B comparator consistently improves the original actor and achieves the strongest overall performance--cost trade-off among the compared methods. These results suggest that effective runtime intervention need not itself be a task-solving problem: the intervention role can be separated from task solving and handled by a lightweight model specialized for local comparison.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Don't Solve, Just Compare".

Jane: LLM agents are increasingly used for long-horizon tasks requiring sequential reasoning and tool use, making runtime intervention crucial for improving reliability without retraining.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: To start, let's look at the title itself: "Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents." That really captures the essence of what they’re doing—limiting the auxiliary model to comparison rather than solving.

Jane: Exactly, Tom. The authors are Yanze Jiang and colleagues, and they’re pointing out that we need a way to give agents a useful direction when their proposed action is bad during runtime intervention, but existing methods often require another full solver or critic.

Lu: It’s interesting how they framed the two main hurdles they have to overcome: first, the intervention has to be more than just a simple warning, and second, it needs to consider the long-term consequences instead of just surface plausibility.

Meng: That distinction about long-term consequence versus surface plausibility is what makes me think about safety; we need interventions that actually make sense in the context of the whole plan, not just looking good on paper.

Lalam: If we can get this small comparison mechanism to work, it opens up a path for more reliable agents that don't rely on needing a massive correction module every time things drift off course. That feels like a big step toward real-world deployment.

The paper's summary: Tom: So, what’s the actual mechanism they propose? They introduce Comparison-Only Tiny Advisor, or COTA, which is a comparison-only framework where the tiny comparator judges whether sampled alternatives lead to better continuations than the actor’s proposal.

Jane: That means at any decision point after an agent proposes an action, they sample some alternative actions and use this tiny comparator to rank them against the original proposal. If at least R of those alternatives defeat it, they suggest intervention.

Lu: The core mathematical idea is that the comparator learns a function Cθ(st, aA, aB) which estimates whether action A leads to better continuation than action B in state st.

Meng: So the tiny model isn't trying to find the best path from scratch; it's just checking if option A beats option B right now based on what it has learned. That seems much less demanding computationally than solving the whole task again.

Lalam: It’s smart because it keeps the task-solving power with the original, larger actor while offloading the comparison judgment to this very focused little model. It's like giving a brilliant generalist a tiny expert referee for local disputes.

The paper's improvements: Tom: The paper suggests two big improvements for this approach. First, constructive intervention must provide more than a binary warning; it needs to give the actor something actionable to replan from.

Jane: They also emphasize that this comparison should reflect the long-term consequences of the current decision rather than just how plausible that single action looks right now.

Lu: The training method they use, same-prefix counterfactuals, is clever because it helps them train the comparator on relative continuation quality between actions from the same state without mixing in different historical context.

Meng: That training setup sounds rigorous; ensuring we get good pairwise supervision for local differences is tricky when you’re dealing with long sequences of decisions. I wonder how stable that learned comparison function is when the agent moves into a completely new area of the task space.

Lalam: The statistical guarantees they provide are what really sell this, showing that even with just a tiny model, we can have a high probability that the intervention gate will work correctly when things get tricky. That level of reliability is what makes me optimistic about its application in complex agent systems.

Conclusion: Tom: So, to wrap up the paper "Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents," the main point is that we can achieve constructive runtime intervention by narrowing the auxiliary model’s role from solving a task to comparing local alternatives.

Jane: It shows that this tiny comparator framework can be sufficient to guide substantially larger actors, proving that effective intervention doesn't have to rely on another full task solver or critic.

Lu: The paper demonstrates that candidate diversity in the comparison mechanism is crucial because repeated sampling alone doesn't give you a wide enough local set of alternatives for good intervention.

Meng: I think the efficiency gain mentioned, averaging "one point three eight times the end-to-end episode time of the original actor," shows this isn't just a theoretical curiosity; it has practical overhead that’s manageable in real systems.

Lalam: Ultimately, COTA achieves what they set out to do: it provides constructive runtime intervention without needing another task solver, and it performs well across all nine evaluation settings compared to the strongest actors we tested.

Tom: It’s a solid piece of work, and I think this comparison-only approach is going to be a key tool for building more reliable AI agents in the future.

More episodes

← Home