Retrieval-Based In-Context Learning: A Domain Adaptation Framework

In-context retrieval (ICR) is a retrieval-based form of in-context learning (ICL) in which demonstrations are retrieved from a source database based on similarity to the query, rather than sampled independently. In this work, we formulate ICR as a type of domain adaptation problem, where the source distribution P of the database may differ from the target distribution Q of the test query-label pair. We investigate the performance of ICR under a flexible class of distributional shifts that substantially extends prior work, and establish theoretical guarantees that quantify the benefits and pitfalls of this learning paradigm. Our theory is verified by experiments on synthetic and language tasks.