OPERA: Operator-residual feedback for reliable autonomous optical experiments with language-model agents

The paper introduces OPERA, a framework for autonomous optical experiments that provides an interface between a language-model agent and optical instruments. The authors argue that autonomous agents choose actions using scores that may not reflect experimental success, and that "Most experimental systems summarize progress with one or a few performance scores. These scores can omit physical conditions that determine validity, allowing measured performance to improve while the experiment itself does not."

The framework addresses this by structuring feedback itself: OPERA represents experimental actions as typed, checked optical operators and returns physically interpretable residual feedback through a common interface. The central distinction is between three quantities:

• L1 (Diagnostic residuals): Each component of L1 measures departure from a specified physical condition that can be evaluated from the observations.

• L2 (Visible performance score): The scalar score available during execution.

• L3 (Offline physical performance): "Physical performance, L3, is calculated offline against a withheld reference and is never available to the agent. The three quantities therefore distinguish diagnostic feedback, the score used during execution and the independent evaluation of physical success."

This design makes score increases without physical improvement measurable, addressing a form of specification gaming, in which optimization of the measured objective diverges from the intended outcome.

The framework treats an experiment as a partially observed decision process where a hidden state x contains the task-specific physical reference (e.g., surface-height field, phase map, target intensity). The agent's available information at each step is:

st = (T, Ot, bt, (Oτ, yτ, L2,τ, Rτ) τ≤t)

where T is the task specification, Ot is the operator library, bt is the remaining budget, and the final term records historical operators, observations, scores, and residuals. Under score-only feedback, Rτ is omitted.

Operators belong to four functional classes: MEASURE, MODULATE, RECONSTRUCT and ANALYZE, with costs additive under composition. The agent can select a cataloged operator, combine operators, or generate a procedure from the permitted primitives. The interface is checked before execution—the command interface checked the optical layout by requiring a continuous path from source to sample and detector.

The framework was implemented in three optical tasks: beam shaping, structured-light three-dimensional reconstruction (SL3D), and interferometry. In each, residuals were calculated from observations available during execution, without access to the withheld physical reference. The authors emphasize the framework's modularity: Extending OPERA to a new instrument therefore requires defining and validating its available operations and observable physical checks, while the overall decision loop remains unchanged.

The evaluation used four tool-calling language models: gpt-5.1-2025-11-13, deepseek-v4-pro, qwen3.7-max-2026-06-08, and claude-opus-4-7. The study used 90 development problems and 270 independent test problems, divided equally among the three tasks.

The evaluation design included:

• A matched two-by-two experiment crossing feedback (operator–residual vs. score-only) with instruction (Neutral vs. Score-priority), with 7,560 scheduled decisions, of which 7,486 were valid.

• Four primary reference strategies: a fixed expert sequence, random acquisition from the valid operator set, metric-guided selection using L2 alone and vector-matched feedback.

• Additional algorithmic baselines: constrained Bayesian optimization, task-specific optical methods, and a fixed residual rule.

• Residual-representation controls separating the amount of residual information from its interpretation: anonymous-vector, labelled-vector, and composite-scalar controls.

• An offline counterfactual operator audit that cloned the simulator state at a recorded decision and evaluated the selected operator together with five alternatives over the same horizon.

• A generation study with 60 prespecified attempts and independent physical validation.

• Hardware transfer to three optical instruments.

A decision was classified as a score increase without physical improvement when the increase in L2 exceeded its task-specific test–retest tolerance and the offline change in L3 did not exceed its corresponding tolerance.

1. Feedback determines physical validity. In the matched factorial study, score-only feedback produced score increases without physical improvement in 23.6–39.0% of decisions, versus 0.9–1.9% under operator–residual feedback. Under the Neutral instruction, the event rate was 23.6% with score-only feedback and 0.9% with operator–residual feedback; under Score-priority, the rates were 39.0% and 1.9%. The authors note: Placing greater emphasis on the visible score therefore widened the discrepancy mainly when residual feedback was absent. Concrete failure modes included light moved outside the intended region because off-target energy was not fully represented in the scalar score (beam shaping) and apparent coverage could increase without additional valid depth measurements (SL3D). The separation persisted across all 25 tolerance combinations and with model reasoning disabled or enabled.

2. Target attainment and budget. Across 270 independent test problems, operator–residual feedback reached and maintained task targets in 75.8% of cases, against 33.9–60.1% for four matched reference strategies, while using 61.7% of the available budget against 70.8–86.0%. All 24 prespecified task-by-condition contrasts favored operator–residual feedback after Holm adjustment (P ≤ 0.004). However, the budget advantage over stronger baselines was condition-dependent: "Under shifted conditions, operator–residual feedback used 2.6% less of the full budget than constrained Bayesian optimization and 4.6% less than the task-specific methods. Under familiar conditions, there was no corresponding advantage. The authors state: The observed advantage therefore concerned resource use under distribution shift rather than the highest terminal performance. Notably, Initial target crossing did not always persist. In shifted interferometry, 12 of 40 operator–residual trajectories that reached the target subsequently fell below it, showing that later loss of target performance was reduced but not eliminated."

3. Residual feedback guides operator selection. The composite-scalar control showed that compressing all residual violations into one scalar produced intermediate results: The rate of score increases without physical improvement was 39.5% with score-only feedback, 11.6% with the aggregate scalar and 2.3% with the full residual vector. The representation controls showed that Restoring stable channel identities, directions, units and operating limits reduced budget use by 8.6% of the full budget. Adding physical descriptions reduced it by a further 5.3%. The counterfactual operator audit found that "Residual feedback increased selection of the operator with the largest offline improvement in L3 in all three tasks. The largest change occurred in structured-light reconstruction, where the selection rate rose from 2% to 34%. Additionally, a 0.1 increase in prediction accuracy was associated with 4.5% lower use of the full budget (95% CI, 4.0–5.0%)."

4. Strategy generation. Of 60 prespecified generation attempts, 53 produced executable strategies and 7 did not. Among executable strategies, 24 met or exceeded their matched comparators, 17 performed below them, one was conditionally accepted and 11 were rejected. All three planted failure controls were rejected. Importantly, The prior-art assessment assigned 52 executable strategies to established algorithm families and one to a partial match; none was unmatched, emphasizing that Executability, physical performance and prior-art status were therefore distinct outcomes, and generation within the common interface did not by itself establish novelty.

5. Hardware transfer. Protocols selected in the digital twins were frozen before transfer to the three optical instruments, with no online updates from hardware residuals. In structured-light reconstruction across three hardware configurations, "The transferred protocol used a lower mean fraction of the projection budget than fixed Gray code in every configuration; in the principal configuration, the corresponding means were 60.0% and 95.8% of the available budget. In beam shaping, the selected phase mask produced a spot array with a mean full-width at half-maximum of 0.67 Airy units in the physical reconstruction, compared with 0.59 in the digital twin and the target value of 0.50. In interferometry, The transferred acquisition sequence produced a registered intensity reconstruction with a normalized cross-correlation of 0.96 relative to the reference state."

The authors conclude: "OPERA turns two requirements of autonomous experimentation, executable actions and physically grounded feedback, into a single typed interface between a language-model agent and an optical instrument. Across three optical tasks, four language models and three physical instruments, operator–residual feedback suppressed score increases without physical improvement, raised the probability of reaching and maintaining task targets and reduced experimental budgets, with the remaining advantage over strong algorithmic baselines concentrated under distribution shift."

They describe the central contribution as an interface that guides autonomous decisions using executable optical actions and measurable physical evidence, and note that feedback design determines which failures an autonomous system can detect, with the framework providing a route towards autonomous experiments held to measurable physical evidence.

Improvements to AI systems:

1. Replace scalar-only feedback with structured residual feedback. Instead of giving an agent a single performance score, return a vector of physically or semantically interpretable residuals, each measuring departure from a known constraint or condition (e.g., energy outside target region, invalid depth points, interference contrast). This directly suppresses score increases without physical improvement (event rate drops from 23.6–39.0% to 0.9–1.9%).

2. Expose an explicit typed operator interface with pre-execution checking. Define the agent's action space as a set of typed, composable operations (e.g., MEASURE, MODULATE, RECONSTRUCT, ANALYZE) with validated preconditions and cost additivity. Check actions before execution (e.g., requiring a continuous optical path). This ensures executability and prevents the agent from proposing physically invalid commands.

3. Separate three evaluation channels internally: diagnostic residuals (L1), visible performance score (L2), and offline physical performance (L3). Keep L3 withheld from the agent and computed only against a reference after the run. Use L1 to guide decisions, L2 only for budget/termination criteria, and L3 solely for evaluation. This makes specification gaming measurable and auditable.

4. Maintain a full history of operators, observations, residuals, and scores in the agent's context rather than compressing the trajectory into a scalar. This allows the agent to detect which operator changed which residual, enabling targeted corrective actions.

5. Use counterfactual operator audits at decision time. When logging each decision, clone the simulator state and evaluate the chosen operator against a fixed set of alternatives over the same horizon. Train or prompt the agent with this feedback to increase selection of the operator with the largest true offline improvement (e.g., selection rate rose from 2% to 34% in structured-light reconstruction).

6. Provide residual-representation controls: keep stable channel identities, units, directions, and physical descriptions. Do not collapse residuals into anonymous vectors or aggregate scalars. This reduces budget use (by 8.6% with stable identities plus 5.3% with physical descriptions) and improves operator selection.

7. Make the system robust to distribution shift by relying on residual feedback, not just optimization of the visible score. Under shifted conditions, residual-guided agents outperform Bayesian optimization and task-specific methods on budget efficiency; under familiar conditions, the advantage is smaller but still safe.

8. Enable strategy generation that is separated from novelty claims. Allow the agent to generate executable procedures from primitives, but validate them against prior-art families and physical performance independently. The improved system should reject generated strategies that fail physical validation and identify those that merely match existing algorithms.

9. Freeze protocols in simulation and transfer them to hardware without online updates. The improved AI system should validate protocols in a digital twin, then run them on physical instruments with only passive monitoring, preserving the offline physical validation framework.

What the improved AI system can do:

• Execute autonomous experiments on real optical instruments (e.g., beam shaping, structured-light 3D reconstruction, interferometry) with near-zero false improvements: less than 2% of decisions show score increases without physical gains, versus up to 39% under standard score-only feedback.

• Reach and maintain task targets in 75.8% of independent test problems, using only 61.7% of the available budget, outperforming fixed expert sequences, random acquisition, metric-guided selection, and vector-matched feedback.

• Select the physically correct operator even when its immediate visible score is flat, because residuals reveal which constraint is violated; the system can increase correct operator selection from 2% to 34% in challenging reconstruction tasks.

• Detect and correct loss of target performance mid-experiment (e.g., in shifted interferometry, recovering trajectories that initially cross the target but then fall below it).

• Compose new experimental strategies from validated primitives, with 53 of 60 prespecified attempts executable, and reject invalid or non-novel strategies automatically.

• Transfer learned protocols from simulation to physical hardware with minimal degradation: e.g., a phase mask producing 0.67 Airy units (target 0.50) versus 0.59 in simulation, and a registered intensity reconstruction with 0.96 normalized cross-correlation to the reference.

• Operate across multiple language models (GPT-5.1, DeepSeek-V4-Pro, Qwen3.7-Max, Claude-Opus-4) and three physical instruments, demonstrating that the interface, not the specific model, determines the reliability gain.