The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora

The paper introduces and measures a previously unrecorded training variable: the notation of pre-training corpora, defined as the distribution, across a corpus, of how documents' arrangement is written down. The central claim is that structural markup (headings, list markers, etc.) acts as a reliable cue that removes the need for models to infer discourse boundaries, and that the field has never measured how much of this notation corpora carry.

The paper defines a new statistic, clean-window survival S(W), as the fraction of non-overlapping W-token windows containing no structural markup. It also measures the distribution of clean runs (maximal markup-free spans). A census of thirteen public corpora shows a steep gradient: survival falls to 0.153 in the olmOCR slice of Dolma 3 (re-measured at 0.271), where 74.0% of documents carry markup, against 0.889 in C4. The paper finds that the scarce resource is not unmarked text; it is long unmarked text, with web corpora having 99th-percentile clean runs of 2,400–6,500 tokens while books reach 109,045. A pre-registered supply test finds that only institutional sources (UK Hansard, S(8,192)=0.930) supply the missing format, while consumer sources (YouTube transcripts) fail on length and library scans fail on density.

A pre-registered converter study tested whether converters fabricate structure on prose. The prediction failed: imposition loss was significantly negative (-0.029 [-0.047, -0.010] at W=8,192), meaning converters slightly under-mark rather than invent boundaries. The paper concludes: A faithful cue is a trustworthy one, so accuracy in conversion is not a mitigation of the problem in this paper. It is the problem.

A reader-side experiment across five base models (0.6B to 8.2B, two pipelines) used three arms: marked (Markdown headings), flat (heading text as bare lines), and silent (announcement deleted). Results show that deleting a structural announcement makes the following prose measurably harder to predict (R1 satisfied at 5 of 5, e.g., +0.466 [+0.159, +0.770] millinats per byte at 0.6B), while swapping its notation moves nothing (R3 satisfied, tight null at 5 of 5). The paper concludes: the operative cue is the announcement, not the sigil. A whole-book curve on Moby-Dick shows the deficit is local: +0.0335 nats per byte over the first 16 tokens after a boundary, converging to zero by 1,000 tokens.

A writer-side experiment asked whether base models impose the marked register. Under greedy decoding, 84–100% of continuations collapsed into repetition loops (77 of 1,328 survived). The pre-registered nulls held: models do not impose markup above the authored baseline (I1 not satisfied, bounds below 11.1 tokens per 1,000), and they do not restore deleted announcements (I3, exact zero at all readers, with a temperature-1.0 secondary arm confirming).

The paper ships a format called the pure frame: paragraphs in authored order, every structural announcement deleted, every deletion recorded in a reversible sidecar. A validator passes a file only if it contains zero announcements. The pure pack over 27 works deletes 16,502 tokens (0.713%), achieving S(8,192)=1.000 over 267 windows and S(32,768)=1.000 over 57. The recommended mixture is over announcement presence, not notation: the same work present both marked and pure, so that at the same underlying event the announcement is sometimes there and sometimes not.

The paper identifies the long-context training stage (under 5% of tokens, e.g., 50B of 6.03T for Olmo 3) as the one place where the training sequence is the document, and notes that this slot is currently filled entirely with olmOCR-converted PDFs. It proposes a data card for notation with fields including extractor identity, flags, S(W), and sampling provenance. The central hypothesis is stated as a bet: that a model trained on the same arrangement with and without its announcements forms a discourse channel separable from the surface channel that renders it. The paper concludes: choose format operators by the capability they train, not by the fidelity they preserve.

Improvements to AI systems:

1. Add a notation-aware data filter to pretraining pipelines. During corpus curation, compute S(W) at multiple W values (e.g., 256, 1,024, 8,192) per document and per shard. Reject or down-weight documents with S(8,192) < 0.5 if the goal is long-range discourse modeling; up-weight documents with S(8,192) > 0.9 for clean-prose benchmarks. This directly addresses the scarce resource is long unmarked text finding.

2. Implement a pure frame preprocessor for long-context training stages. Before the final <5% of training tokens, convert all documents to the pure frame format: strip every structural announcement (headings, list markers, etc.), record deletions in a reversible sidecar, and validate with a zero-announcement checker. Train on a mixture of marked and pure versions of the same document (e.g., 50/50) to force the model to learn discourse boundaries from content alone, not from markup cues.

3. Add a notation card to every training shard's metadata. Include extractor identity (e.g., olmOCR, PDF parser), flags for markup presence, S(W) values, and sampling provenance. Use this card at evaluation time to stratify downstream task performance by notation density, revealing where the model relies on markup vs. true discourse understanding.

4. Modify the loss function during long-context fine-tuning to penalize over-reliance on markup. Specifically, add a regularization term that measures the difference in next-token prediction loss between marked and pure versions of the same document. Minimize this difference to encourage the model to form a discourse channel separable from the surface channel, as hypothesized.

5. Build a boundary-aware evaluation suite. For any new model, test on prose with and without structural announcements (marked vs. pure) at multiple context lengths. Report the announcement deficit (loss increase when announcements are deleted) as a standard metric, alongside perplexity. A model with a low deficit (<0.01 nats/byte at W=8,192) is considered robust to notation variation.

6. Add a writer-side guardrail during decoding. After greedy or sampled generation, run a detector that flags repetition loops (as seen in 84–100% of continuations). If a loop is detected, re-generate with a forced pure frame prompt (no markup) or insert a soft structural boundary (e.g., a paragraph break) to break the loop, improving long-form coherence.

7. Create a format operator selection API for downstream tasks. When a user asks for a summary, essay, or code, the system chooses whether to output marked (with headings) or pure (plain paragraphs) based on the task's need for discourse structure vs. surface readability. The API uses the paper's finding that the operative cue is the announcement, not the sigil to decide: if the task requires hierarchical reasoning, keep announcements; if it requires dense prose, strip them.

8. Implement a clean-run length predictor for retrieval-augmented generation. Before retrieving documents, estimate the longest clean run (maximal markup-free span) in each candidate. For queries requiring long coherent passages (e.g., legal analysis, narrative synthesis), rank documents by their 99th-percentile clean run length, not just relevance score. This leverages the finding that books have clean runs up to 109,045 tokens vs. web's 2,400–6,500.

9. Add a converter fidelity check to any OCR or PDF-to-text pipeline. Before using converted text for training, measure imposition loss (as defined in the paper) on a held-out prose sample. If the loss is significantly negative (under-marking), re-run with a different extractor or apply a markup-restoration step. This prevents silent degradation of discourse cues in training data.

10. Design a mixed-notation curriculum for continual learning. During the final training stage, present the same document in three arms (marked, flat, silent) in a randomized order, as in the reader-side experiment. This teaches the model to predict prose equally well regardless of whether a boundary is announced, reducing sensitivity to notation and improving generalization to unseen formats.

How a document's arrangement is written down, its notation, is a training variable that no dataset card records. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts. What corpora carry: a census of thirteen public corpora, where survival falls to 0.153 in a vision-converted PDF slice against 0.889 in C4; the scarce resource is not unmarked text but long unmarked text; a pre-registered supply test finds what remains institutional, not consumer. Our own pre-registered prediction failed: converters do not fabricate structure on prose, and that null forced the reliability mechanism that survives it. What readers use: across five base models spanning 0.6B to 8.2B and two pipelines, deleting a structural announcement makes the following prose measurably harder to predict, while swapping its notation moves nothing. That zero does not make notation unimportant; it relocates the variable: the operative cue is the announcement, not the sigil. What writers impose: a bounded null. Base models do not impose the marked register above the authored baseline, and handed prose with every announcement deleted they do not put one back, at a rate indistinguishable from zero against an authored reference of zero. We ship the format those measurements imply: the pure frame, paragraphs in authored order, every announcement deleted into a reversible sidecar, mixed against the marked copy over announcement presence rather than notation. Choose format operators by the capability they train, not by the fidelity they preserve, and record extractor identity and survival on data cards.