Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

The paper identifies a fundamental limitation of Transformer positional embeddings: Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to syntactic structure. While the Transformer architecture underpins modern LLMs, "several linguistic simplifications were introduced in the underlying architecture to improve scalability. One key choice was the use of positional embeddings (PE) to encode each token's location within a sequence. While this approach preserved computational efficiency and scalability, it reduced positional information to a simple ordering mechanism—sufficient to indicate where tokens appear, but insufficient to capture the syntactic relations that form the scaffolding for downstream semantics."

The authors illustrate the problem with an example: for the request Please move this large file to another folder, the move predicate must be associated with its arguments (file, folder), yet the relative distance between move and folder is large (six words), even though a direct oblique syntactic dependency connects the verb to its modifier. Prior work injecting hierarchical inductive bias typically does so by constraining self-attention using constituency or dependency structure, often via quadratic token-token interactions and hard-coded attention masks, which requires considerable changes in the transformer architecture and increases computational overhead.

The authors introduce SiPE, which "augment[s] a model's positional pathway with a lightweight prior derived from linearized dependency trees—coarse directional indicators relating each token to its syntactic governor—learned during pre-training via an auxiliary indicator-prediction objective alongside the LM loss. The key design principle is that SiPE composes with all three dominant positional encoding families—absolute (Liu et al., 2019), relative (He et al., 2021; Dai et al., 2019), and rotary (Warner et al., 2024)—without modifying self-attention or any other Transformer component."

The syntactic signal comes from Hexatagger (Amini et al., 2023), a parsing-as-tagging dependency parser that "assigns each token two types of discrete syntactic labels (analogous to position or subword IDs): (a) the position of its syntactic governor (which can further be decomposed into two subtypes) and (b) the dependency relation. Hexatagger maps a projective dependency tree to a binary head tree, producing terminal tags and, plus non-terminal tags L, R, L, and R—yielding a fixed projective tag inventory (T=2 terminals, N=5 non-terminals). The authors omit dependency relation labels, finding that terminal/non-terminal hexatags alone provide sufficient syntactic inductive bias to improve downstream language modeling performance."

At position p, the prior contributes mp · (ETτp + ENνp), where ET ∈ RT×d and EN ∈ RN×d are learned tables... mp ∈ 0,1 is the first-subword mask, equal to 1 only at each word's first subword so the hexatag is injected once per word. Hexatags are assigned first-subword only so that the total tag embedding signal injected per word is invariant to its subword length ki.

A central contribution is isolating where and how the prior should enter the model, finding it depends on the architecture.

For Transformer-XL, the paper studies five injection sites. The winning configuration is Position-Pathway Injection (PP-SiPE)—leave the input residual clean and route the tag straight into the position term. The attention score splits into a content term AC and a position term BD, where BD depends only on i−j, it carries no information about which tokens occupy positions i and j or their syntactic role. SiPE supplies that missing signal by forming a tag-query alignment ci,j,n = 1/√d head ⟨qi,n + vn, WE(ETτj + ENνj)⟩n and coupling it multiplicatively: e A(l)i,j,n = AC(l)i,j,n + (1 + c(l)i,j,n) · BD(l)i,j,n.

The authors explain why multiplicative coupling wins: "its correction term c · BD gates by offset alignment, applying the syntactic adjustment in proportion to how strongly the query already attends to that offset. Where the query-offset alignment is strong (BD large), the tag exerts a large pull; where it is near zero, the tag exerts almost none, so syntax modulates an existing positional preference rather than acting in isolation." This outperforms input-side injection, fully-disentangled injection (adding c as an independent third logit term), attention-bias projections (adapting Shaw et al. 2018 with key-side and value-side biases), and the joint combination. Notably, Injecting the same hexatag signal as both a positional and attentional bias is thus redundant rather than complementary; for Transformer-XL, the prior is best supplied once, through the positional pathway.

For encoders, the simpler input-embedding injection is better: the prior is added directly to the input embeddings, composing with each encoder's native positional mechanism. For RoBERTa (absolute): xp = etokp + pp + esegp + mp · (ETτp + ENνp). For DeBERTa-v3 (disentangled relative attention, no input position embedding): the tag prior becomes the only positionally-localized signal at the input layer: h(0)p = etokp + esegp + mp · (Eτp + ENνp). For ModernBERT (RoPE): added at the input residual exactly as in DeBERTa... without the segment term. A positional-pathway variant for RoPE applies a per-layer phase offset ∆θp,t = [W(l)δ(ETτp + ENνp)]t added to the rotary angle before rotation, so the tag enters as a syntactic phase shift on top of the positional one.

On Transformer-XL (all SiPE variants use TXL as the decoder), compared to the vanilla Transformer-XL baseline (BLiMP 75.30, PPL 18.63, SyntaxGym 73.09):

• PP-SiPE (positional pathway): BLiMP 74.01 ↓ -1.71, PPL 16.95 ↑ +9.02, SyntaxGym 80.60 ↑ +10.29 — the best SyntaxGym score, achieving the best SyntaxGym score of any configuration of our method (80.60, +10.3 relative improvement over the no-syntax baseline) while cutting perplexity by 9.0% (relative).

• Input-side injection: SyntaxGym 76.97; fully-disentangled: 78.72; per-layer attention bias: 78.10; combined SiPE + attention bias: 77.97.

The paper notes this carries through to GLUE, improving the TXL macro average from 68.17 to 73.78, a dramatic +8.2% relative gain (Table 6), with SiPE improv[ing] every task on GLUE.

On BLiMP: +1.41% (RoBERTa), +2.27% (DeBERTa-v3), +2.32% (ModernBERT) (Table 3). Under continued pre-training on BLLIP-LG (out-of-domain), these gains grow... rising to +1.87%, +3.06%, and +4.21% respectively, indicating the prior generalizes beyond its original distribution rather than overfitting. On GLUE (Table 4), SiPE improves every architecture on the macro average—RoBERTa +0.75%, DeBERTa-v3 +0.59%, ModernBERT +1.0% relative, with gains at large scale too (Table 5).

A key contribution is positioning SiPE between two extremes of prior work: "joint syntactic LMs (TG, Pushdown LM, PLM) recover p(x) by marginalizing over ≈300 candidate parses; parser-free methods (TreeReg, Tree-Planted Transformers) inject syntax only during training and discard the parser at runtime, making inference cheap but sacrificing some syntactic knowledge. SiPE conditions on a single parse, retaining most of the benefit of full marginalization at 1/300th of its inference cost. Empirically, SiPE surpasses parser-free approaches... cutting perplexity by roughly a quarter against the strongest such baseline (16.95 vs. 22.30 for TreeReg) and by more than half against the Tree-Planted Transformer variants (16.95 vs. 45.5–47.7) while matching or exceeding their syntactic generalization."

A sweep of the entry layer k for PP-SiPE in Transformer-XL finds: syntax is most impactful at the very first layer: full injection (k=1) is clearly best, and skipping just the first layer (k=2) already causes a sharp drop in syntactic generalization: SyntaxGym falls from 80.6 to 73.5. More broadly, syntactic information matters most in the lower layers — for autoregressive models the positional pathway should carry syntactic structure from the earliest layer onward.

SiPE adds two small embedding tables for terminal and non-terminal tags, ET ∈ R2×d and EN ∈ R5×d, totaling 7d parameters per pathway (∼5K at d = 768, ∼7K at d = 1024), well under 0.01% of the LM's total parameters. The injection adds no asymptotic cost over self-attention, and at inference, a single auxiliary parser pass supplies the hexatags.

On BLiMP Causative examples where SiPE is correct and the base model is not, the encoders redistribut[e] verb→object attention to mirror syntactic adjacency: RoBERTa+SiPE places higher verb→object attention than its base model in 90% of cases... ModernBERT+SiPE does so in 85% of cases, while Transformer-XL surfaces the prior only weakly in its attention maps and instead converts it into the largest downstream semantic (GLUE) gains of any model we train (only 55% of cases shift attention). "In over 80% of the subset of 20 examples we examine, the object is nonadjacent to the verb, indicating a redirection of attention weights to the syntactically relevant object, even when it is linearly distant from the verb."

The paper acknowledges several limitations: the approach assumes access to hexatag annotations at inference time and inherits the hexatagger's errors. The study considers only coarse-grained syntactic priors derived from directional terminal and non-terminal tags (DEPREL labels did not yield meaningful gains on GLUE). The injection-site study is broad but not exhaustive. Most significantly: fast autoregressive text generation with the decoder is an open problem because "generating each new token requires re-tagging the sequence produced so far... it is not compatible with standard KV-caching: as the sequence grows, the parser may revise the tags of earlier tokens, changing their injected representations and invalidating the cached keys and values. Finally, experiments are limited to small models, English text, and the pre-training budgets used here... its unclear how the relative benefit of an explicit syntactic prior will scale as model and data scale increase."

The paper's five contributions: (1) introducing SiPE with a lightweight dependency-tree-derived prior learned via an auxiliary indicator-prediction objective; (2) demonstrating composition with all three dominant PE families without modifying self-attention; (3) showing encoder gains on BLiMP (+2.3% relative for ModernBERT) and GLUE at small and large scale; (4) showing the positional-pathway recipe is strongest for autoregressive decoders (SyntaxGym 80.60, PPL reduced 9.0%, GLUE +8.2% relative); and (5) establishing a new Pareto frontier between syntactic supervision and inference cost by conditioning on a single parse at inference rather than marginalizing over ≈300 trees or discarding syntax at runtime.

Here are concrete improvements you can make to an AI system based on this paper, and what the improved system will be able to do.

• Add two small learned tag tables: E T for terminal tags (2 entries) and E N for non-terminal tags (5 entries), totaling only 7d extra parameters (under 0.01% of the model).

• At first-subword positions only, add E T[terminal tag] + E N[non-terminal tag] to the existing input embeddings.

• Pre-train with an auxiliary objective that predicts the token’s hexatag from the model’s hidden state, jointly with the language modeling loss.

What the improved system can do:

• Improve syntactic generalization on BLiMP by up to +2.32% (ModernBERT), +2.27% (DeBERTa-v3), and +1.41% (RoBERTa).

• Improve GLUE macro-average performance by +0.59% to +1.0% relative across encoder architectures.

• Preserve the original Transformer architecture and inference speed; no attention masks, no quadratic token-token interactions, and no parser at runtime beyond one cheap tagging pass.

• For decoder-only models such as Transformer-XL, do not add the tag embeddings to the input residual.

• Instead, form a tag-query alignment:

c(i,j,n) = (1/√d head) ⟨q i + v n, W E(E T τ j + E N ν j)⟩

and apply it multiplicatively to the position term:

eA(i,j,n) = AC(i,j,n) + (1 + c(i,j,n)) · BD(i,j,n)

where BD is the existing relative-position attention logit.

• Cut perplexity by 9.0% relative compared to the vanilla Transformer-XL baseline (18.63 → 16.95).

• Improve SyntaxGym syntactic generalization by +10.3 points (73.09 → 80.60).

• Improve GLUE macro-average by +8.2% relative (68.17 → 73.78), improving every GLUE task.

• Keep injection computationally free: the extra term costs no asymptotic complexity over self-attention.

• For encoders (RoBERTa, DeBERTa-v3, ModernBERT), add the hexatag prior directly to the input embedding.

• For decoders with relative positional encodings, inject through the positional pathway as described above.

• For transformers using rotary positions (RoPE), additionally support a positional-pathway variant that applies a per-layer phase offset derived from the tag embeddings before rotation.

• Achieve consistent gains regardless of the underlying positional encoding family (absolute, relative, or rotary).

• Avoid the need to redesign attention; the prior composes with all three dominant PE families.

• Improve robustness under continued pre-training on out-of-domain data: encoder gains grow from +1.41% to +1.87% (RoBERTa), +2.27% to +3.06% (DeBERTa-v3), and +2.32% to +4.21% (ModernBERT) on BLiMP, showing the prior generalizes rather than overfitting.

• Replace expensive joint syntactic language modeling (which marginalizes over 300 candidate parses) with a single Hexatagger parse.

• Keep the parser only as a pre-processing step; at inference, condition on the single fixed hexatag sequence.

• Retain most of the benefit of full parse marginalization at roughly 1/300th of the inference cost.

• Beat parser-free methods by a large margin: perplexity 16.95 vs. 22.30 (TreeReg) and 45.5–47.7 (Tree-Planted Transformers), while matching or exceeding their syntactic generalization.

• Sit on a better Pareto frontier than both marginalizing syntactic LMs and parser-free approaches.

• In Transformer-XL, set the injection entry layer to the first layer (k=1), not later layers.

• Skipping the first layer already drops SyntaxGym from 80.6 to 73.5.

• Achieve strong syntactic generalization in lower layers, where syntactic structure is most impactful.

• This gives better downstream task performance while allowing later layers to specialize on higher-level semantics.

• With the SiPE prior, encoders learn to redistribute attention from linearly adjacent tokens to syntactically governed tokens even when far apart in the sequence.

• For example, attention from a verb to its object increases in 90% of tested cases with RoBERTa+SiPE and 85% with ModernBERT+SiPE, even when the object is nonadjacent to the verb.

• Handle long-distance syntactic dependencies better, especially when the head and dependent are separated by many intervening words.

• Improve performance on tasks requiring subject-verb agreement, filler-gap dependencies, and long-range argument attachment.

• Train the model to predict each token’s terminal/non-terminal hexatag from its contextual representation.

• Do not require dependency relation labels — directional hexatags alone are sufficient.

• Gains are achieved with minimal added parameters and no added attention cost.

• The auxiliary objective can be added to existing pre-training scripts with only small code changes, and it works across RoBERTa, DeBERTa-v3, ModernBERT, and Transformer-XL.

This improved system is most useful for non-interactive or offline settings (e.g., evaluation, classification, encoder tasks, or pre-training) because fast autoregressive generation is currently incompatible with KV-caching: the parser may revise earlier tokens' tags as the sequence grows, invalidating cached keys/values. An improved system that targets interactive generation would first need to address this re-tagging problem, for example by freezing tags once generated or using incremental parsing.

Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to syntactic structure. We introduce Syntax-informed Positional Embeddings (SiPE), which learns a lightweight syntactic prior from dependency parses during pretraining and injects it across all three dominant PE families (absolute, relative, rotary), for both encoders and decoders, leaving self-attention and the rest of the architecture untouched. We isolate where and how the prior should enter the model, and find it depends on the architecture: for autoregressive decoders that use relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score, outperforming injection into the input embeddings, into self-attention, or into the positional and attention terms jointly---while for encoders it is best added directly to the input embeddings, composing with each encoder's native positional mechanism. We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to 10.3% while simultaneously reducing perplexity by 9.0% over a base model with no syntactic supervision---a metric nearly every existing syntax-injection method instead degrades. Crucially, these gains extend beyond syntactic generalization: SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to 8.2% over a model trained without it. Unlike existing syntactic language models that marginalize over many parses at inference or discard syntax at runtime, SiPE conditions on a single parse, establishing a new Pareto frontier between syntactic supervision and inference cost.