Language Models Struggle to Use Representations Learned In-Context

arXiv:2602.04212 · cs.CL, cs.AI · Submitted 2026-02-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Language Models Struggle to Use Representations Learned In-Context".

Jane: The gist Open-weights language models struggle to deploy in-context representations learned from context to complete simple downstream tasks like next-token prediction and adaptive world modeling,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we’re looking at this paper called Language Models Struggle to Use Representations Learned In-Context. The main idea here is that while language models can learn representations from context—like seeing a pattern in a prompt—they still struggle to use those learned representations when they have to do actual tasks later on.

Jane: Right, so the big question is, if an AI learns a new way to understand something just because you showed it in the prompt, can it actually use that understanding when you ask it something totally different? That’s what this paper is really digging into.

Lu: Exactly. The authors are testing whether these in-context representations are flexible enough to handle different tasks after they've been learned from the context, which is a big step toward building systems that can adapt to new situations upon deployment.

Meng: From my side, I’m thinking about deployment—if a model learns something new in context but then fails on a simple downstream task like predicting the next word, it just means it’s not robust enough for real-world use where things get messy.

Lalam: I see what they're saying is that even when the latent representations encode completely novel semantics from the context, they often end up being inert when a different instruction comes along.

Tom: That’s a strong claim. They first test this on next-token prediction and then move into this more complex idea called adaptive world modeling to see if those same representations can be deployed flexibly.

Jane: So, the paper sets up two main tests: one for simple prediction and another for a task that requires building an internal model of how things connect in a space. This is where they really try to push the limits of in-context learning.

Lu: They use prompts with two parts—one part defines the context semantics, like specific words or structures, and the other part asks for something based on those semantics, and they find that even when representations reflect novel topology information, it’s hard to actually use that topology later on >

Meng: And they found that open-weights models struggle with this deployment. That means if you take a model like a smaller version of something publicly available, it doesn't automatically get the benefit of the context learning for practical applications >

Lalam: They show evidence that in both tasks, these in-context representations are largely inert, even though they seem to encode the new semantics perfectly within their latent states >

Tom: That’s a key finding because it challenges the idea that simply encoding information in a hidden state is enough for flexible deployment. The abstract specifically mentions this difficulty with open-weights LMs trying to complete simple downstream tasks like next-token prediction and adaptive world modeling >

Jane: It suggests that the representations learned in context aren't just stored; they aren't easily accessible or usable when the model needs to apply them to a new problem, which is a problem for building truly adaptable AI systems >

Paper summary: Lu: The authors point out that even when they look at closed-source, state-of-the-art reasoning models on the adaptive world modeling task, they still cannot reliably use the novel patterns presented in context >

Meng: So even those top performers aren't automatically solving this deployment hurdle. That tells me we need to think about how to make these learned representations more usable for actual agents >

Lalam: The overall conclusion of this paper is that in-context semantics encoded in hidden representations are often inert, or unable to be used for either next-token prediction or adaptive world modeling >

Tom: So the title itself really sums up the issue: Language Models Struggle to Use Representations Learned In-Context. It’s about the gap between learning something new and actually being able to use it when things get complicated >

Jane: When we think about this, it means that just seeing a pattern in a prompt isn't enough for an AI to gain a flexible, useful understanding that it can carry forward into novel situations >

Lu: The limitation they highlight is that in-context semantics encoded in hidden representations may not be flexibly deployable, but the paper doesn't offer a solution directly; instead, they suggest targeted training regimens or mechanistic interventions could make these semantics available to earlier model layers >

Meng: That sounds like heavy training work. I’m wondering how much effort it would take to retrofit existing models so that their in-context learning isn't just theoretical but practical for deployment >

Lalam: Also, they mention that it’s difficult to understand the relationship between the externalized chains-of-thought produced by reasoning models and their hidden token representations, which adds another layer of complexity when we try to make these things work >

Tom: So if we look at what this means for AI development right now, it suggests that simply scaling up context length or prompt complexity isn't the answer; we need a fundamental change in how those learned internal representations are structured >

Jane: It pushes the research toward architectures and training strategies that support flexible deployment of representations learned in-context, which is what they hope to inspire for building truly adaptable artificial agents >

Lu: The paper builds on previous work that looked at whether world models afford other manipulations, suggesting that the current view isn't quite complete >

Meng: For production, this means we can’t just rely on a model "remembering" context; we need mechanisms to actively manipulate those learned structures for different purposes >

Lalam: The core finding remains that representations learned in context are often inert when faced with novel tasks, which is a cautionary note for how we view the power of in-context learning >

Tom: So, this paper shows us that while in-context learning is impressive, we still have a long way to go before language models can reliably use those learned concepts for complex reasoning outside of the exact context they were given >

Conclusion: Tom: So, we’ve seen how these language models learn stuff just by looking at context, but now we're talking about whether they can actually use that stuff later on in different situations.

Jane: Exactly. This paper is titled "Language Models Struggle to Use Representations Learned In-Context." The authors are showing that this isn't just a theoretical problem; it happens when you ask the model to do real tasks after it’s learned from a prompt.

Lu: What they're doing is testing if these in-context representations are flexible enough for things like next-token prediction and adaptive world modeling. They’re seeing if the patterns they learn in context actually stick around when the task changes.

Meng: And what they found is pretty definitive—open-weights models struggle to deploy those learned semantics, even when they've encoded the right information in their hidden states for things like grid topology. That’s a problem for production deployment.

Lalam: I agree with Meng. I see it too; the representations are often inert when faced with novel examples, whether that’s a simple next-token prediction or trying to model an adaptive world. They aren't reliably using the context they saw before.

Tom: That means we can’t just assume that if a model learns something new in a prompt, it automatically gains the ability to use it for anything else. It stops being like a flexible system and more like a static data store.

Jane: Right. The authors are suggesting that this gap between learning and using is real, even for models that are already pretty high performing. They're pointing toward needing training regimens or architectural changes to make these learned representations actually available for deployment in novel contexts.

Lu: And they highlight that we still don’t fully understand the connection between those externalized reasoning chains and the hidden token representations, which makes it hard to fix this right now. It points toward a bigger need for new ways of structuring these internal models.

Meng: So, if this is true, then simply increasing context length isn't the answer; we need something that lets the model manipulate those learned concepts rather than just storing them passively. That’s what I’m thinking for shipping models at scale.

Lalam: I’m quite sure about that conclusion. The core finding is that in-context semantics are often inert, and they aren't useful for either next-token prediction or adaptive world modeling without some kind of mechanism to activate them properly.

Tom: This paper isn't just about a model failing a test; it’s a warning about the current way we think models should learn and adapt. It suggests that in-context learning is impressive, but deployment requires more than just memorizing patterns.

Jane: It really does. And this leads us nicely into what might actually fix this, like those targeted training regimens the authors mention next.

Brown University · Google Research · New York University Google DeepMind

cs.CL, cs.AI

Submitted: 2026-02-04

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: The gist Open-weights language models struggle to deploy in-context representations learned from context to complete simple downstream tasks like next-token prediction and adaptive world modeling,

Key concepts

In-Context Representation Learning
This is the process where a model learns useful patterns directly from the text provided in a prompt without any formal training on that specific task. The researchers tested if these learned patterns could be reliably used later for different goals, like predicting the next word or understanding a new world.
Adaptive World Modeling (AWM)
This is a novel task designed to test if an LLM can use in-context semantics to solve problems that require understanding and manipulating a dynamic environment. The models were shown to struggle when trying to apply novel, context-defined rules or topologies they learned from the prompt.
Inert Representations
This refers to representations (the internal data structures the model creates from context) that are present in the model's hidden states but cannot be effectively used by the model to solve a specific problem. The study found that even when these representations contain new meanings, they remain largely unused for tasks like next-token prediction.
Flexible Deployment
This is the ability of a model to take information learned in one context (like a prompt) and apply it reliably to solve a completely different, subsequent task. The paper concludes that current LLMs struggle with this flexibility, indicating they need better training methods for such adaptability.

Terminology

Summary

The gist

Open-weights language models struggle to deploy in-context representations learned from context to complete simple downstream tasks like next-token prediction and adaptive world modeling, even when those representations encode novel semantics in their latent states.

In-Context Representation Learning

The study first assesses whether open-weights LMs can use in-context representations for next-token prediction, and then probes models using a novel task, adaptive world modeling. The researchers first evaluate whether open-weights LMs can use in-context representations for next-token prediction, and then probe models using a novel task, adaptive world modeling. They investigate this question using prompts which crucially have two components: one which defines the in-context semantics (colored words in Figure 2) and a separate component which requires the model to refer to these semantics to complete a task. The findings indicate that in-context representations present in the hidden states of an LM are largely inert, even when they encode novel semantics in their latent representations.

Next-Token Prediction Task

In Experiment 1, the researchers tested whether in-context representations are flexibly deployable by varying whether the random walk is presented in a user message (Instruction Condition) or a prefilled model response (Prefilled Condition). They found that models reliably predict a valid next-word in the random walk when the random walk is prefilled, but struggle when it is presented in the user prompt. The analysis showed that representations that are learned in-context are largely inert.

Adaptive World Modeling Task

The second task, Adaptive World Modeling (AWM), was designed to probe whether LMs can flexibly deploy novel, in-context semantics to solve simple tasks. For AWM, the researchers found that open-weights models struggle with the task, even when their latent representations encode the underlying grid topology. Specifically, LMs cannot reliably use the topology information encoded in them when confronted with the few-shot examples.

Reasoning Models and Externalized Chains

The study also assessed closed-source, state-of-the-art reasoning models on the AWM task, demonstrating that even the most performant LMs cannot reliably leverage novel patterns presented in-context. These models utilize long chains of thought (CoTs) to externalize intermediate reasoning steps, and they encode novel, in-context semantics in a manner that sometimes facilitates their use when solving downstream tasks, but they do so imperfectly.

Conclusion

Overall, the work suggests that LMs cannot reliably deploy representations of in-context defined semantics to solve downstream tasks like next-token prediction or adaptive world modeling. The conclusion is that in-context semantics encoded in their latent representations are often inert, or unable to be used for either next-token prediction or adaptive world modeling.

Limitations and Future Directions

A limitation noted is that the work reveals that in-context semantics encoded in hidden representations may not be flexibly deployable, but do not offer a solution. The authors speculate that targeted training regimens or mechanistic interventions that make these semantics available to earlier model layers can partially address this issue. Furthermore, it is difficult to understand the relationship between the externalized chains-of-thought produced by reasoning models and their hidden token representations. The paper concludes that the creation of truly adaptable artificial agents requires training regimens and architectures that support flexible deployment of representations learned in-context.

Related Work

The research is situated at the intersection of characterizing the internal world models implemented by language models and characterizing the capabilities and limitations of in-context learning. Many prior works rely on a model’s ability to track state throughout context, which is a precursor to the in-context representation learning under study. The study also addresses the canonical view that a world model is not just a look-up table but affords other manipulations.

Acknowledgments

The authors thank Andrew Lampinen, Martin Wattenberg, Asma Ghandeharioun, Effie Li, Eghbal Hosseini, Yasaman Bahri, Julian Zimmert, and the members of the Computation and Psycholinguistics Lab at NYU for their helpful feedback. They also thank Rachel Goepner for proofreading the manuscript.

References

The paper cites numerous works related to world models, in-context learning, and reasoning chains, including studies by Li et al. (2023; 2025a), Vafa et al. (2024, 2025), and various papers on chain-of-thought prompting. The references also include foundational works on language models as few-shot learners and implicit representations in neural language models. The work builds upon previous investigations into world models by probing internal representations or generalization behavior and characterizing ICL’s algorithmic properties. The final section lists specific citations for various model architectures and techniques used in the experiments, such as Gemini 2.5 (Comanici et al., 2025) and GPT-5 (Team, 2025). The paper also references work on reasoning models like DeepSeek-AI et al. (2025) and the concept of thought anchors.

Appendix Details

The appendix contains detailed analyses, such as analysis of relative Dirichlet Energy between tokens in the random walk and tokens in the few-shot examples, analysis of meta-learning versions of the task, and ablation studies on reasoning chains to analyze their impact on performance. These appendices provide deeper evidence regarding why representations learned in-context are not being flexibly deployed.

Optimal Context Lengths

Table 1 reports optimal context lengths from the analysis of in-context representation learning, showing varying results across different dimensions and world sizes. The table also presents naive baseline strategy accuracies for next-token prediction. The results indicate that optimal context lengths vary depending on the model and topology being tested.

Naive Solution for One Step Rules

Table 2 presents the accuracies achieved by a naive baseline strategy for adaptive world modeling tasks, which involves randomly sampling an attested adjacent state for each possible query example. This baseline strategy achieves accuracies presented in Table 2. The results show that even with perfect information about the topology, or a simple heuristic solution to adaptive world modeling tasks with a one-step rule, there is still a performance gap compared to explicit knowledge.

Measuring Relative Dirichlet Energy

Section E presents an analysis of the relative Dirichlet Energy between tokens in the random walk and tokens in the few-shot examples in an adaptive world modeling task. This analysis computes normalized Dirichlet Energy as described in Section 2 to report the ratio of these values. The results show that representations in the few-shot learning examples are degraded relative to their counterparts in the random walk.

Metalearning Results

Figure 14 presents the results of a metalearning version of the task, which consists of three task presentations, and shows that this approach yields poor performance. The results from this metalearning prompt accuracy analysis are shown in Figure 14.

Long Context Adaptive World Modeling

Figure 15 presents adaptive world modeling results with constant long context length, using gemma-3-27b-it, and shows that this does not improve performance. The results from this analysis are shown in Figure 15.

Frontier Model Next-Token Prediction Results

Figure 16 presents next-token prediction performance for frontier models, finding that aside from Gemini-2.5-Flash, models achieve good performance at next-token prediction. The results are presented in Figure 16.

Improvements for AI systems

  1. Bold Header: Flexible Deployment Mechanism for In-Context Semantics

The system can be improved to ensure that representations of novel semantics that are defined in-context are not just encoded but also flexibly deployable. This means the model must be able to deploy these representations to accomplish goals, moving beyond the current state where they are largely inert when faced with novel tasks like Adaptive World Modeling.

  1. Bold Header: Enhanced Reasoning Chain Utilization

For closed-source, state-of-the-art reasoning models, the system should be tuned to better leverage externalized reasoning steps. This is achieved by ensuring that externalized reasoning chains can compensate for otherwise-inert hidden representations, allowing them to encode novel, in-context semantics in a manner that sometimes facilitates their use when solving downstream tasks.

  1. Bold Header: Topology-Aware Representation Encoding

The system should be trained or fine-tuned to better reflect the latent state space topology. This involves improving the fidelity of encoded structures by increasing the Relative Dirichlet Energy between tokens, ensuring that representations in few-shot examples encode the underlying topology with lower fidelity than the representations in the random walk.

  1. Bold Header: Explicit Topology Grounding for Downstream Tasks

To improve performance on novel tasks, models should be explicitly informed of task structures. This is supported by findings showing that when the explicit description of the underlying state space is [provided], LMs achieve ceiling accuracy on a few-shot learning task, suggesting that providing this strong clue about the underlying state space topology enables better deployment.

Abstract

Though large language models (LLMs) have enabled great success across a wide variety of tasks, they still appear to fall short of one of the loftier goals of artificial intelligence research: creating an artificial system that can adapt its behavior to radically new contexts upon deployment. One important step towards this goal is to create systems that can induce rich representations of data that are seen in-context, and then flexibly deploy these representations to accomplish goals. Recently, Park et al. (2024) demonstrated that current LLMs are indeed capable of inducing such representation from context (i.e., in-context representation learning). The present study investigates whether LLMs can use these representations to complete simple downstream tasks. We first assess whether open-weights LLMs can use in-context representations for next-token prediction, and then probe models using a novel task, adaptive world modeling. In both tasks, we find evidence that open-weights LLMs struggle to deploy representations of novel semantics that are defined in-context, even if they encode these semantics in their latent representations. Furthermore, we assess closed-source, state-of-the-art reasoning models on the adaptive world modeling task, demonstrating that even the most performant LLMs cannot reliably leverage novel patterns presented in-context. Overall, this work seeks to inspire novel methods for encouraging models to not only encode information presented in-context, but to do so in a manner that supports flexible deployment of this information.

Sources

Related papers