Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shift

Deployed AI text detection approaches are supervised, relying on training-time access to labeled datasets of both human-written and AI-generated text. This approach is vulnerable to three types of distribution shifts that occur continually post-deployment, and for which labeled data is often unavailable: adversarial humanization, new LLMs being released, and unidentifiable temporal drift in human writing after widespread LLM usage. Simultaneously, existing approaches do not leverage a key signal of LLM usage: inference-time homogeneity. We propose a test-time adaptation (TTA) approach that adapts to distribution shifts by leveraging homogeneity among unlabeled samples observed at inference time. Empirically, we find that state-of-the-art supervised detectors systematically fail when they encounter distribution shifts in AI-generated and human writing, both adversarial and natural, while TTA with semi-supervised learning is largely robust; e.g., the commercial model Pangram detects just 27.0% of our adversarial AI-generated text, compared to 93.3% for our test-time approach. We establish that TTA is a promising framework for AI text detection in the wild.