The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort

summary

The gist

Spracklen et al.

In short

This study replicates previous findings to measure how often large language models invent non-existent software package names from PyPI and npm. While the range of hallucination rates narrowed, a universal set of 127 packages is consistently invented across all tested models. This means one registration can attack users of multiple frontier AI providers simultaneously.

Key concepts

Package Hallucination
This occurs when an LLM generates a software package name that does not actually exist on public registries like PyPI or npm. This creates a security risk because malicious actors can use these invented names to trick users into downloading harmful, fake packages.
Universal Attack Surface
This is a specific set of 127 package names that every model tested in the study invented, regardless of the model's architecture. Identifying this set allows security teams to target a single name that could compromise users across all five evaluated AI providers.
Rate Compression
The wide difference in hallucination rates between commercial and open-source models has significantly reduced. This suggests that the quality of training data regarding package references is becoming saturated, leading to more consistent, lower hallucination rates across different model types.
Model Asymmetry
This refers to differences observed between specific model families, such as Python versus JavaScript generation rates or how different versions within the Anthropic family perform. Understanding these asymmetries helps tailor prompt strategies to mitigate risks specific to certain model architectures.

Terminology used across episodes

This episode discusses

The paper

The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort · Read on arXiv

Aleksandr Churilov

Package hallucinations, LLM-generated references to packages that do not exist, enable slopsquatting: an attacker registers the hallucinated name. We replicate Spracklen et al. (USENIX Security '25) on five 2026 frontier models over 199,840 Python and JavaScript generations. This version reports a preregistered live revalidation that replaces the static package lists of earlier versions with two independent checks against the live PyPI and npm registries. Hallucination rates are 3.6-5.0% of package mentions (2.2-3.4% of generations): the 2026 frontier sits at the level of 2024's commercial models, and the open-weight gap seen in 2024 has disappeared. Python exceeds JavaScript for every model. The set of names hallucinated by all five models shrinks from the 127 reported earlier to 104; 26 earlier entries were correct import statements whose module name differs from the package name. Using the accidental six-month public exposure of our own list as a natural experiment, we find no detectable effect of publication: about 1% of hallucinated names were registered within six months, at the same rate for published and unpublished names, and the 95% interval excludes increases above 2.5 percentage points. Still, at least 23 package names that the models suggest had previously hosted malware later removed by the registries, and one hallucinated name was registered with malware after our runs.

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "The Range Shrinks, the Threat Remains".

Elias: Here's the gist of the paper. Spracklen et al.

Nadia: showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm at rates ranging from 5.2% on commercial models to 21.7% on open-source models, creating an attack surface for slopsquatting-the registration of malicious packages under hallucinated names<ref:2605.17062 Replication and Scope The paper replicates the methodology of Spracklen et al.

Elias: on five frontier 2026 LLMs released between October 2025 and March 2026 to measure package hallucination rates<ref:2605.17062#pg2> The authors generated 199,845 generations across these models, validating every emitted package against PyPI and npm master lists supplied with the original artifact<ref:2605.17062#pg2> This replication serves as a direct cross-temporal comparison to the 2024 baseline study<ref:2605.17062#pg2> Rate Compression and Persistence The inter-model hallucination range has compressed substantially, moving from a 5.2%–21.7% spread to 4.62%–6.10% across the five evaluated models<ref:2605.17062#pg2> This narrowing is attributed to the fact that the open-source–commercial gap has effectively closed at the frontier, and training-data curation on package references appears near-saturated<ref:2605.17062#pg2> Despite this compression, a 4–7% rate remains adversary-economically viable for slopsquatting because package upload costs are zero<ref:2605.17062#pg2> Universal Attack Surface Identification A central finding is the existence of a universal-hallucination set containing 127 package names (109 PyPI, 18 npm) that all five evaluated models invent<ref:2605.17062#pg2> Following coordinated disclosure, this set was narrowed to 53 names (41 PyPI, 12 npm) that remain registrable after registry-side defenses<ref:2605.17062#pg2> This model-agnostic attack surface means a single registered name can target users of all five evaluated frontier providers simultaneously<ref:2605.17062#pg2> Asymmetries and Convergent Patterns The study observes several asymmetries, including a Python-over-JavaScript hallucination asymmetry where Python rates exceed JavaScript rates by between +2.73 pp (Haiku 4.5) and +4.13 pp (GPT-5.4-mini)<ref:2605.17062#pg3> Furthermore, the DeepSeek V3.2 / GPT-5.4-mini Jaccard peak of J = 0.343 suggests shared training-data origins or convergent generation patterns<ref:2605.17062#pg2> Within the Anthropic family, Claude Haiku 4.5 hallucinates measurably less than Claude Sonnet 4.6, contradicting typical smaller-model findings<ref:2605.17062#pg3> Limitations and Future Directions The study notes several limitations, including the evaluation of only five frontier models and point-in-time measurements taken between April 22, 2026 and April 28, 2026<ref:2605.17062#pg2> The denominator asymmetry for GPT-5.4-mini is due to running at reasoning effort=minimal, which produced a refusal rate of 32.14%<ref:2605.17062#pg2> Future work should explore agentic configurations where retrieval mechanisms can eliminate package hallucinations entirely<ref:2605.17062#pg2> Ethics and Open Science To minimize harm, the authors did not register any hallucinated names on any public registry but disclosed the full enumerated list to PyPI Security and Socket.dev for coordination<ref:2605.17062#pg2> The replication code, raw outputs, and analysis scripts are publicly available under a verified-researcher access policy<ref:2605.17062#pg2> The residual registrable attack surface is 53 names (41 PyPI, 12 npm), which is a point-in-time estimate of the registrable subset<ref:2605.17062#pg2> The range has shrunk from a 5.2%–21.7% spread to 4.62%–6.10%, but the threat persists at rates well below 1% for adversary viability<ref:2605.17062#pg2> The existence of a model-agnostic universal-hallucination set means that cross-cohort intersections should be a standard reported quantity in future work<ref:2605.17062#pg2> A single registration can target users of all five evaluated frontier providers simultaneously<ref:2605.17062#pg2> Acknowledgments The authors thank the authors of Spracklen et al.

Nadia: for releasing their prompt datasets, master lists, and validation code as a public artifact<ref:2605.17062#pg3> They also thank Mike Fiedler of PyPI Security and the Socket.dev security team for their coordinated-disclosure review of the universal-hallucination set<ref:2605.17062#pg3> The author funded this study from personal commercial accounts, with the exception of a one-time Google Cloud free credit grant for Gemini 2.5 Pro inference<ref:2605.17062#pg3> No institutional, governmental, or industry funding was received<ref:2605.17062#pg3> The author is an independent researcher and reports no conflicts of interest with any of the studied model providers<ref:2605.

Title and authors: Nadia: So we're looking at a paper called "The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the two thousand twenty-six Frontier-Model Cohort," and it looks like they are digging into how much these code generating models actually mess up package names <ref:2605.17062#pg1,The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on>.

Elias: Yeah, it's about taking what Spracklen et al. did before and testing it on five newer models released in late two thousand twenty-five through early two thousand twenty-six measuring the actual hallucination rates for Python and JavaScript packages against PyPI and npm lists <ref:2605.17062#pg1>.

Priya: Before we get into the numbers, I want to know what this study is actually trying to show us about the risk of this slopsquatting. Is it just a statistical curiosity or does it point to something practical for people building software?

Nadia: It’s pointing toward a real attack surface that can be exploited cheaply. They are showing that even though the rates have gotten smaller, there's still a persistent threat because the cost to an attacker is really just the cost of using the LLM API spend to find those names.

Elias: Exactly. They replicated one hundred ninety-nine thousand eight hundred forty-five code samples across these models and found this thing called a universal hallucination set of one hundred twenty-seven package names that all five models invent identically <ref:2605.17062#pg1>.

Priya: A universal set is important because it means you don't have to worry about which specific model you are using; if you target those names, they should fail everywhere. But how many of those still actually matter?

Nadia: Well, they narrowed that set down to fifty-three names after registry defenses were factored in, meaning forty-one on PyPI and twelve on npm remain registrable right now. That’s the actual window for an attacker.

Elias: And it's interesting because they also found some weird family differences; for instance, one model within the Claude family hallucinates less than another one in the same family, which goes against what you might expect from a smaller model generally hallucinating more.

Priya: That suggests that we can’t just treat all models as equal when it comes to privacy and safety testing; we have to look at how different architectures behave differently.

Title and authors: Nadia: Right. And the paper actually suggests some ways to get better, like identifying specific patterns in the names that are common hallucination artifacts, which helps narrow down what’s actually dangerous for developers.

Elias: They also pointed out an asymmetry where Python packages are hallucinated more often than JavaScript ones, with differences ranging from two point seven to four point one percentage points depending on the model used.

Priya: So, even with these specific findings, what does this mean for a developer who is just trying to write some standard code and wants to be safe? Does this change how we use AI assistants?

Nadia: It means developers need to be more careful about trusting package names generated by the AI because that name could lead them directly into a malicious package installation if they trust the registry blindly.

Elias: The whole attack chain they laid out involves six stages, starting with identifying these names, registering them maliciously, and then tricking a developer into installing it through their package manager after the LLM generates code referencing the fake name.

Priya: I wonder if we should be focusing more on the detection part—how to filter out these hallucinated names before they even make it into a developer's environment?

Nadia: That’s where some of those improvements come in; they are suggesting ways to build better noise filters that specifically look for things like prefixes or path aliases that might be common artifacts, so you can reduce false positives.

Elias: And they also talked about making the evaluation process more dynamic, meaning instead of just using static lists from a single point in time, you should integrate feedback loops from registries to keep that list of registrable names current.

Priya: That sounds like a big step toward making the defense mechanism reactive instead of just based on what we knew last week. It moves us closer to real-time risk assessment for the supply chain.

Nadia: It does, because right now, if you get a new package registered that was hallucinated yesterday but isn't yet on the list, your current defense system misses it entirely.

Title and authors: Elias: So looking at the conclusion of "The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the two thousand twenty-six Frontier-Model Cohort," they summarize that while the spread has shrunk from fifty point two to six point ten percent, we still have a threat because of those persistent names <ref:2605.17062#pg1,The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on>.

Priya: And they emphasize that even with these smaller rates, the risk remains because there’s always that one name in the universal set an attacker can use to hit multiple providers at once.

Nadia: That model-agnostic attack surface is a big deal for security teams because it means your defense strategy needs to be broad enough to cover all providers simultaneously.

Elias: And they confirm that even with these findings, the threat is still not high enough for an attacker to make money on it because the per package upload cost is zero, meaning the only real cost is just how much you spend on using the LLM API to find those hallucinations.

Priya: So it’s a low-cost, high-reach attack vector that relies entirely on human trust in the package manager interface. That feels like a very grounded way to look at it.

Nadia: Yeah, and the paper itself is open, so security researchers can actually use their replication code to test these five models themselves and see if those findings hold up in their own environments.

Elias: And they also set up a coordinated disclosure process with PyPI Security and Socket.dev to make sure that when they find this universal set of names, the fix happens quickly across the major registries.

Priya: It seems like the biggest implication is that for any developer using these tools, treating generated package names with some level of skepticism about their existence in a real repository is now a necessary part of secure coding practice.

Nadia: Absolutely. We need to shift from assuming generated code is safe to actively validating every external reference point it makes, especially when it comes to dependencies.

Elias: That’s the core message of "The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the two thousand twenty-six Frontier-Model Cohort <ref:2605.17062#pg1,The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on>." We’ll be looking at how this affects our next topic soon.

The paper's summary: Nadia: So, we just talked about how code generators can accidentally invent package names that don't actually exist on PyPI or npm, creating a risk for slopsquatting attacks because there are zero costs to uploading a fake package.

Elias: Right, and this paper took that idea and tested it across five different frontier models released in two thousand twenty-six to see if the hallucination rates were still high or if they’d shrunk.

Nadia: Well, the summary shows that while we went from a wide range of five point two percent to about six point ten percent, the actual threat level hasn't dropped enough for it to be considered low risk anymore.

Elias: That’s because they found this universal set of one hundred twenty-seven package names that every single model invented, regardless of whether it was open source or commercial.

Nadia: It means if you target those specific names, you’re hitting users across all five major providers at the same time.

Elias: But the paper also found some interesting weird things, like how one family of models hallucinates less than another in the same group, which suggests we can't just treat all of them as identical.

Nadia: And they pointed out that Python packages get hallucinated more often than JavaScript ones, with differences between two point seven and four point one percent depending on the model used.

Elias: That asymmetry is a key finding because it tells us how the underlying training data or architecture of these models favors certain languages over others in terms of naming conventions.

Nadia: So, for a developer, this means they can’t just assume the AI is generating safe code references across all ecosystems; they have to be aware that Python names might be slightly more prone to being invented than JavaScript ones.

Elias: And they did get practical by narrowing that universal set down to fifty-three names that are still registrable even after some registry defenses are in place.

Nadia: That’s the window, right? The actual risk is now focused on those fifty-three specific names, which is a manageable list for a security team to monitor and block.

Elias: Exactly. But they also noted that the method itself has limitations because it only looked at five models at one specific time in April two thousand twenty-six.

Nadia: That’s important, because it means this is just a snapshot of what’s happening right now, and we need more testing to make sure these patterns don't change when new models come out.

Elias: And they did suggest some future directions where you could use agentic configurations with better retrieval mechanisms to try and eliminate these package hallucinations entirely at the source.

Nadia: That sounds like a big goal—trying to build an AI workflow where it can’t even hallucinate that kind of dependency reference in the first place.

Elias: It is, and that leads us right into how we can actually build those filters and tuning methods you mentioned earlier, which might help catch these errors before they ever reach a developer's machine.

The paper's improvements: Tom: So we’re looking at how these authors are proposing ways to actually fix these hallucination problems they found in this study on package names.

Nadia: They aren't just stopping at finding the problem, though; they’re suggesting real improvements for detection and filtering before a developer even sees that fake name.

Elias: I mean, what they suggest is creating a robust noise filter that specifically hunts for certain syntactic artifacts like Dartpackage prefixes or Vue Webpack path aliases.

Nadia: That’s smart because it moves beyond just checking if the name exists in the registry; it checks the structure of the name itself to see if it looks like an artifact of an AI hallucination.

Elias: It helps them reduce those false positives, which is a big deal when you’re trying to build a system that needs high precision for security filtering.

Nadia: Right, because if you get too many false alarms, nobody trusts the filter anymore and they just ignore it.

Elias: And they also talked about making the evaluation process dynamic instead of relying on static lists from a single point in time.

Nadia: That means integrating real-time updates from registries so the list of registrable attack surface names stays current, instead of using an old snapshot.

Elias: Precisely. They want to build a feedback loop where registry changes automatically refine what the risk list actually looks like, making the defense much more reactive.

Nadia: So we’re moving toward a system that isn't just checking against yesterday's data, but one that lives in real-time with the package registries themselves.

Elias: And they also discussed this idea of model-specific tuning based on family inversions; for instance, understanding why one model in a family hallucinates less than another.

Nadia: That’s about leveraging the unique strengths and weaknesses of different AI architectures to design better prompts or safety guardrails for specific use cases.

Elias: It’s about knowing when to trust a smaller model versus when you need the capability of a larger one, based on how they behave in these kinds of generation tasks.

Nadia: So instead of treating all models the same, we can design tailored safety protocols that account for those internal family behaviors.

Elias: It’s about making sure our defense strategy isn't just a blanket rule but something that respects the underlying math and structure of what we’re actually running.

Nadia: That takes us right into how we actually implement these things, because having a good idea is one thing, but building the filter or the tuning layer is another whole problem.

Conclusion: Nadia: So we’re wrapping up this look at "The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the two thousand twenty-six Frontier-Model Cohort." This study confirmed that even though the hallucination rates have shrunk from five point two to about six point ten percent, the persistent names still create a risk for slopsquatting attacks.

Elias: Right, and that universal set of one hundred twenty-seven invented names means any single registration can target users across all five major providers simultaneously.

Nadia: It shows that even with these smaller rates, the threat remains because there’s always that one name in the universal set an attacker can use to hit multiple providers at once.

Priya: From a privacy standpoint, it’s important to know that while the overall rate is low, we still have a specific target list of fifty-three names that registry defenses might miss right now.

Elias: That's the caveat; the risk isn't zero because there are still those registrable names in that set.

Nadia: The implication for us is clear: any developer using AI assistants needs to be more careful about trusting package names and validating every external reference point it makes when dealing with dependencies.

Priya: So, for me, the big picture is that we need better ways to filter out these artifacts before they ever get into a developer's environment at scale.

Elias: Yeah, and the authors did suggest using model-specific tuning based on how different AI families behave to design better safety guardrails for those specific architectures.

Nadia: It’s about making our defense strategy less generic and more tailored to the actual models we are deploying.

Elias: So, while this paper lays out a clear attack surface, the authors also pointed out that they only evaluated five frontier models at one point in time in April two thousand twenty-six.

Priya: That means we need future work to see if these patterns hold up as more and more models come online and change how they generate these names.

Nadia: Exactly, because the evaluation isn't a permanent state; it’s a point-in-time measurement of what's happening right now.

Elias: We’re looking at "The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the two thousand twenty-six Frontier-Model Cohort" as a case study in how we track these emerging supply chain risks.

Priya: I think moving toward real-time feedback loops with registries is going to be key for keeping that risk list current for everyone involved.

Nadia: Right, because the next big question is how we actually build those filters and tuning layers to make that kind of real-time defense happen in practice.

More episodes

← Home