UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Following

Large language models (LLMs) are increasingly expected to follow long lists of constraints in complex instructions, and synthesizing instructions from a reference document (i.e., back-translation) is a widely used method to measure/enhance LLMs’ ability to follow complex instructions. However, this method introduces a critical loophole: the constraint synthesis model copies text from the reference as a very specific constraint and the evaluated LLM trivially satisfies the constraint by copying its text in the response. To address these issues, we propose UNSPECIFIC, a novel framework that synthesizes constraints common to two similar reference articles to reduce copy-pasting, selectively hardens only trivially satisfied constraints to balance difficulty and naturalness, and evaluates satisfaction on both the generated article and its summary to penalize superficial instruction following. Consequently, we built the UNSPECIFIC benchmark on news, story, and blog domains to analyze the copy-pasting behavior of LLMs. Our results show that our synthesized constraints are not only more challenging (e.g., the satisfaction rate of GPT-5 Mini drops from 90% to 78%) and natural (LLM win-rate gap improves by 30%) from a human perspective but also mitigate the copy-pasting. We also find that a large portion of constraints are satisfied superficially (i.e., not satisfied in the core narrative of the article). The code and datasets are released at https://github.com/JeetDSharma/UNSPECIFIC.

Improvements to AI systems:

1. Constraint synthesis via intersection, not extraction: Instead of generating constraints by copying from a single reference, the system now synthesizes constraints that are common to two distinct reference articles on the same topic. This forces the model to abstract away surface-level phrasing and focus on semantic requirements (e.g., “mention the cause of the event” rather than “include the exact date ‘March 14, 2023’”).

2. Selective hardening of trivial constraints: The system detects constraints that are trivially satisfied by verbatim copying (e.g., a specific name or statistic) and automatically rewrites them into harder, more abstract forms (e.g., “describe the protagonist’s motivation” instead of “include the name ‘Alice’”). This balances difficulty without making all constraints unnatural.

3. Dual-level satisfaction evaluation: The system now checks constraint satisfaction not only on the generated article but also on a summary of that article. This penalizes superficial instruction-following where the model inserts the required text in a peripheral sentence but fails to integrate it into the core narrative. The improved system can flag when a constraint is met only at the surface level.

4. Copy-paste mitigation mechanism: The system explicitly measures and reduces the overlap between the generated response and the reference documents used for constraint synthesis. It can now detect when a model is “cheating” by copying long spans from the reference and either penalize that behavior or force the model to paraphrase.

What the improved AI system can do:

• Generate more robust instruction-following benchmarks that are resistant to gaming via memorization or copying.

• Evaluate LLMs more accurately by distinguishing between genuine comprehension and superficial text matching.

• Train LLMs to follow complex instructions more deeply—ensuring that constraints are integrated into the narrative structure, not just appended as disconnected facts.

• Provide human-aligned difficulty calibration—producing constraints that are challenging yet natural, as judged by human preference (30% win-rate improvement).

• Identify specific failure modes in LLMs (e.g., “constraint satisfied in the article but missing from the summary”) to guide targeted fine-tuning or prompting strategies.

• Serve as a diagnostic tool for developers to see which types of constraints (e.g., abstract vs. concrete) cause LLMs to fall back on copy-pasting, enabling better training data design.

Large language models (LLMs) are increasingly expected to follow long lists of constraints in complex instructions, and synthesizing instructions from a reference document (i.e., back-translation) is a widely used method to measure/enhance LLMs' ability to follow complex instructions. However, this method introduces a critical loophole: the constraint synthesis model copies text from the reference as a very specific constraint and the evaluated LLM trivially satisfies the constraint by copying its text in the response. To address these issues, we propose UNSPECIFIC, a novel framework that synthesizes constraints common to two similar reference articles to reduce copy-pasting, selectively hardens only trivially satisfied constraints to balance difficulty and naturalness, and evaluates satisfaction on both the generated article and its summary to penalize superficial instruction following. Consequently, we built the UNSPECIFIC benchmark on news, story, and blog domains to analyze the copy-pasting behavior of LLMs. Our results show that our synthesized constraints are not only more challenging (e.g., the satisfaction rate of GPT-5 Mini drops from 90% to 78%) and natural (LLM win-rate gap improves by 30%) from a human perspective but also mitigate the copy-pasting. We also find that a large portion of constraints are satisfied superficially (i.e., not satisfied in the core narrative of the article). The code and datasets are released at https://github.com/JeetDSharma/UNSPECIFIC.