Failure-Aware Long-Form Translation: Design and Implementation of a Recoverable LLM Translation System

This paper describes a recovery protocol for a long-form translation system that addresses the problem that "A long-form translation request can succeed at the API layer and still produce an unusable result. The output may be empty, truncated, filtered, dominated by source or prompt material, or interrupted after producing text worth keeping."

The system's design is motivated by the observation that "provider-level success is not application-level success. A non-empty response can be truncated, contain large source-language residue, or omit most of a multi-paragraph input. A stream can terminate cleanly at the transport layer with a length or filtering finish reason that makes the content unusable. Conversely, part of an interrupted stream may be safe to retain. Long documents amplify the cost of getting these distinctions wrong: full restart increases latency and cost, while blindly retaining an unverified suffix can misalign subsequent text and make omissions harder to detect."

The paper makes three contributions: "1. a streaming architecture that distinguishes buffered, visible, boundary-safe, and committed text while recovery is still in progress; 2. an operational failure matrix and bounded routing policy derived from the deployed implementation; and 3. a sanitized executable artifact with public tests and fixed cases for release, retention, attempts, events, and provenance."

The system uses a hierarchy of text units: "The client tracks document-level completion by chunk, while the server may recover within a chunk at complete paragraph or sentence boundaries. Terms such as token, network delta, sentence, paragraph, chunk, and committed document progress therefore need to remain distinct."

The design uses three distinct states for text: "Buffered text is not yet visible. Visible text has passed the leading guard but may still be replaced if a later whole-output check fails. Committed text has passed completion checks and is recorded as a completed chunk. Within an interrupted attempt, a boundary-safe prefix contains only complete paragraphs and, conservatively, complete sentence-like units that can be aligned to available source units."

The first containment mechanism is a leading-window guard: "The streaming translator initially accumulates content instead of immediately calling the client-visible delta callback. At the recorded snapshot, the hold window is 64 characters. Each arriving fragment extends the buffered prefix. If a known prompt marker appears, the attempt raises an unusable-output error before any candidate content is emitted. Once the window is large enough, a source-heavy check runs; failure similarly prevents release. If checks pass, the accumulated prefix is flushed once and subsequent deltas flow normally."

After assembly, the system performs completion validity checks: "the system interprets provider completion metadata as an application signal. A content-filter finish enters the policy-specific path; a length finish is an unusable truncation; and a successful terminal event with no content is an empty-output failure. After assembly, prompt markers, severe source residue, and catastrophic omission are checked again."

The system also has an omission guard: "a model may return a fluent, non-empty ending while silently dropping most input paragraphs. The current implementation applies it only to sources with at least four content blocks and 400 visible characters. It rejects output only when paragraph retention is below 0.70 and the output-to-source character ratio is below 0.15."

For recovery after interruption, the system derives a safe state: "It keeps completed paragraphs. For the latest paragraph, it may also retain complete sentence-like units ending in punctuation, bounded by the number of complete source sentences. It drops the unterminated output fragment and reconstructs the remaining source. The server sends replace with the retained text and the separator needed to join the next output."

The routing policy uses a stable model order: "For a model tier and session identifier, the server deterministically shuffles active models. Every chunk in the task therefore sees a stable order unless operational state changes. Quota-classified errors mark a model temporarily exhausted; unusable outputs rotate without globally marking the model, because the failure may be input-specific."

The system has bounded recovery parameters: "At the recorded snapshot, the handler budget is 240 seconds and the streaming phase receives 170 seconds of that budget. Six consecutive failures without a new boundary-safe prefix end stream recovery; retained progress resets the counter because the remaining source has changed."

After model rotation is exhausted, the system uses a layered fallback: "the handler preserves any boundary-safe prefix and applies a layered fallback to the remaining source. The order is: a still-eligible compatibility model, an optional configured external machine-translation provider, and an emergency machine-translation path. Each layer receives only a bounded portion of the shared handler budget... The recorded caps are 35 seconds for the compatibility layer and 20 seconds for the external machine-translation layer, with 10 seconds reserved for emergency machine translation."

The frontend handles typed stream events: "The frontend parses meta, model, reasoning, delta, restart, replace, done, and error. A restart announces a new attempt but does not change visible text. A replace resets the current chunk to a server-derived boundary-safe prefix. A delta appends text. The done payload supplies the final combined text and a typed fallback origin."

The evaluation uses a sanitized companion artifact: "The artifact excludes the production product, user data, acquisition adapters, accounts, community features, database code, credentials, and deployment configuration. It uses only the Python standard library. Its 38 public tests pass in an isolated run and cover guard boundaries, multilingual residue, omission thresholds, resume-state derivation, reconnect validation, event transitions, attempt bounds, deadlines, layered fallback, and provenance."

The fixed offline conformance set results show: Across the fixed set, every configured label and each scripted attempt, event, and provenance invariant was reproduced. Specifically, the leading window contained 0 guarded visible characters versus 235 without the hold window for four early-invalid prefixes, and the interruption recovery retained 31 characters retained versus 0 compared to full restart.

The paper discusses design lessons from development history. One key change was limiting sentence rescue: "An early resume path tried to preserve complete sentences whenever a stream was interrupted. In practice, a sentence could be retained without a reliable paragraph anchor and then aligned to the wrong part of the source... Sentence rescue was consequently limited to cases with an established paragraph prefix and disabled for moderation-related interruptions."

Another lesson involved prompt echo detection: "Prompt echo was initially checked only after the stream ended. In one recorded response, a provider returned the task and source tags themselves, so the interface displayed them before the final validator rejected the response. In the same internal trace, holding the first 64 characters prevented that exposure and delayed first visible text by about 1.5 seconds."

The omission guard was added after an internal incident: "a 30-paragraph chunk returned one fluent paragraph and passed every existing non-emptiness, prompt-echo, and residue check. Because the client treated non-empty completion as success, repeated manual retries were marked complete. The saved document then contained 600 translated paragraphs for a source document with 629 paragraphs."

The paper acknowledges limitations: "This study is tied to one deployed system and its development history. The matrix describes failures observed or anticipated in that system; it does not measure their prevalence across products, providers, or domains. Several guards are language-specific heuristics. Paraphrased prompt leakage, subtle omission, and semantic inconsistency can pass them... A boundary-safe prefix is structurally aligned, not semantically certified."

The conclusion states: "Long-form translation does not fail only when a request crashes. It can also fail when a response appears successful for the wrong reason: a non-empty result may conceal missing paragraphs, expose prompt material, or end after producing text that is worth keeping only at a safe boundary. These cases lead to one design consequence: generating text cannot be the same as committing it. The reported system therefore treats release, recovery, and fallback as separate decisions. It holds uncertain prefixes before they reach the reader, re-derives the portion that can survive an interruption, and keeps the origin of a fallback attached to the completed chunk."

Improvements to AI Systems Based on This Paper:

1. Implement a three-tier text commitment protocol (buffered → visible → committed) in any streaming generation system. The AI system will hold output in a buffer until it passes content-safety and source-fidelity checks, then release it to the user, and only mark it as final after completion validation—preventing exposure of prompt echoes, source residue, or truncated outputs.

2. Add a leading-window guard (e.g., 64 characters) before emitting any visible text. The improved system will accumulate initial output, scan for known prompt markers or source-heavy patterns, and reject the attempt before the user sees any content. This reduces visible invalid characters from 235 to 0 in early-failure cases, as demonstrated in the paper.

3. Introduce an omission guard for long-form generation. For inputs with ≥4 content blocks and ≥400 characters, the system will compute paragraph retention and output-to-source character ratio. If retention <0.70 and ratio <0.15, it will flag the output as unusable—catching cases where a model returns one fluent paragraph while silently dropping 29 others.

4. Implement boundary-safe recovery after interruption. Instead of full restart or blind suffix retention, the system will keep only complete paragraphs and sentence-like units ending in punctuation, aligned to source sentence counts. It will drop the unterminated fragment, send a replace event to the client, and resume from the reconstructed source—retaining 31 characters versus 0 with full restart in the paper’s fixed test set.

5. Use a typed event stream with explicit recovery semantics (meta, model, reasoning, delta, restart, replace, done, error). The improved system will allow the frontend to distinguish between a new attempt (restart, no visible change) and a safe-state rollback (replace), enabling seamless user experience during provider failures.

6. Apply a bounded routing policy with stable model ordering. For each session, the system will deterministically shuffle active models per chunk, mark models as temporarily exhausted on quota errors, and rotate on unusable outputs without globally blacklisting (since failures may be input-specific). This maximizes success probability while minimizing latency and cost.

7. Enforce layered fallback with strict time budgets. After exhausting model rotations, the system will preserve the boundary-safe prefix and try: (a) a compatibility model (≤35s), (b) an external MT provider (≤20s), (c) an emergency MT path (≤10s)—all within a shared 240s handler budget. Each fallback layer receives only bounded time, preventing runaway costs.

8. Add completion-metadata interpretation as an application signal. The system will treat provider finish reasons as semantic: content-filter → policy path, length → unusable truncation, success-with-no-content → empty-output failure. This converts transport-level events into actionable recovery decisions.

9. Limit sentence rescue to cases with established paragraph anchors. The improved system will only retain partial sentences when a paragraph prefix already exists and will disable this rescue entirely for moderation-related interruptions—preventing misalignment of text to the wrong source segment.

10. Attach provenance to every completed chunk. The system will record the fallback origin (model tier, compatibility, external MT, emergency MT) in the final output metadata, enabling downstream auditing, debugging, and quality tracking across heterogeneous providers.

A long-form translation request can succeed at the API layer and still produce an unusable result. The output may be empty, truncated, filtered, dominated by source or prompt material, or interrupted after producing text worth keeping. This report describes a recovery protocol developed for a deployed translation system with heterogeneous inputs and provider APIs. It delays the first visible release behind a 64-character window, validates the assembled output, and uses typed stream events to distinguish replacement from continuation. Interrupted work is retained only when a paragraph or sentence prefix can be re-derived from the source. Further attempts follow a stable model order and a shared deadline before entering a provenance-marked fallback path. A sanitized companion artifact implements the protocol and passes 38 public tests. Its fixed cases reproduce all 14 configured completion labels, contain four early-invalid prefixes before any of their 235 characters become visible, retain 31 boundary-safe characters across four interrupted streams, and satisfy the attempt, event, and provenance rules in two end-to-end scenarios. These results are executable checks of the published control flow. Translation quality and detector performance on naturally occurring outputs require a different evaluation.