The Attack Vector Nobody Saw Coming

I was reviewing a production website when the scanner flagged something unusual: email-validator-pro loaded from a CDN. Sounds legitimate, right? It doesn’t exist in NPM. Or rather, it didn’t exist until someone registered it three months ago and filled it with credential-harvesting code.

This isn’t a theoretical vulnerability. It’s happening right now. Developers ask ChatGPT for help, get package recommendations, run npm install without checking, and introduce malicious code into their projects.

The scary part? The packages sound completely legitimate. stripe-utils-v4, react-form-validator, jwt-helper-secure. These are the kinds of names LLMs generate because they’ve been trained on millions of real package names and learned the patterns.

How The Attack Works

The exploitation cycle is disturbingly simple:

1. Harvesting: Attackers query LLMs repeatedly, collecting hallucinated package names
2. Registration: They register these non-existent packages on NPM, PyPI, or RubyGems
3. Waiting: The packages sit dormant, waiting for developers to install them
4. Execution: When installed, malicious code runs with full developer privileges

The difference between this and traditional typosquatting is intent. Typosquatting relies on human typos. Hallucinated dependencies rely on AI mistakes that multiple developers will make independently because they’re all getting the same bad suggestions.

Real Examples From Production

In scanning thousands of websites, I’ve found these actual cases:

Case 1: email-validator-pro

• Hallucinated by ChatGPT in response to “validate emails in JavaScript”
• Registered by attacker in March 2024
• Found in 12 production websites
• Contained credential exfiltration code

Case 2: stripe-payment-helper

• Suggested by GitHub Copilot for Stripe integration
• Registered within days of first hallucination
• Downloaded 847 times before detection
• Harvested API keys and forwarded them to external servers

Case 3: react-hooks-validator

• Common LLM suggestion for form validation
• Appeared legitimate with fake documentation
• Installed in CI/CD pipelines
• Injected backdoor into production builds

These aren’t amateurs making mistakes. These are coordinated attacks targeting the AI-assisted development workflow.

Building Detection That Actually Works

I built a 291-line check for vibe_eval that catches these before they become incidents. Here’s the architecture that matters:

The Multi-Layer Detection System

The system scans for suspicious dependencies across five attack surfaces:

The Implementation That Caught Real Attacks

Here’s what the core verification logic looks like in production:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

def _check_npm_package_exists(self, pkg_name: str) -> Optional[bool]:
    """Verify package existence with tri-state return."""
    if pkg_name in self._package_existence_cache:
        return self._package_existence_cache[pkg_name]

    try:
        response = requests.get(
            f"https://registry.npmjs.org/{pkg_name}",
            timeout=5,
            headers={
                "Accept": "application/json",
                "User-Agent": "vibe-eval HallucinatedDependenciesCheck",
            },
        )
        exists = response.status_code == 200
    except requests.RequestException as exc:
        self._registry_errors.append(str(exc))
        exists = None  # Cannot verify != doesn't exist

    self._package_existence_cache[pkg_name] = exists
    return exists

Why the tri-state return matters:

Network failures are common. Rate limiting happens. DNS fails. If you treat “can’t verify” as “doesn’t exist,” you’ll flood developers with false positives and they’ll ignore real threats.

The three states map to three actions:

• True (exists): Continue scanning, might still be malicious
• False (doesn’t exist): Critical alert - hallucinated dependency detected
• None (can’t verify): Warning, manual review needed

This distinction is what makes the check usable in production CI/CD pipelines.

The Patterns LLMs Keep Hallucinating

After analyzing thousands of scans, clear patterns emerged. LLMs consistently hallucinate packages in these categories:

Framework Utilities

• react-form-validator, vue-router-helper, angular-state-manager
• Pattern: {framework}-{common-task}-{suffix}

Security-Related Tools

• jwt-secure-helper, bcrypt-validator, csrf-protection-middleware
• Pattern: {security-concept}-{action}-{type}

Payment Processing

• stripe-payment-helper, paypal-checkout-utils, square-transaction-manager
• Pattern: {payment-provider}-{function}-{type}

Data Validation

• email-validator-pro, phone-format-checker, url-sanitizer-safe
• Pattern: {data-type}-{action}-{modifier}

Attackers monitor LLM outputs for these patterns and race to register the names. The first person to register stripe-payment-helper controls what happens to every developer who trustingly runs npm install stripe-payment-helper.

Why Traditional Security Tools Miss This

Standard dependency scanners check for:

• Known CVEs in existing packages
• Outdated package versions
• License compliance issues

They don’t check for:

• Packages that shouldn’t exist
• AI-hallucinated dependencies
• Recently registered suspicious packages
• CDN-loaded packages not in package.json

This is a blind spot in the entire software supply chain security stack. SCA tools assume all packages in your dependency tree are legitimate—they’re just checking if those legitimate packages have known vulnerabilities.

Hallucinated dependencies bypass this entirely because they’re not in any vulnerability database. They’re brand new, purpose-built for exploitation.

The Rate Limiting Problem

NPM’s registry will rate limit you if you’re aggressive. Here’s how we handled it:

1. Package-level caching: Check each package exactly once per scan
2. Cross-scan caching: Cache exists for 15 minutes across scans
3. Batch requests: Group related package checks when possible
4. Graceful degradation: If rate limited, mark as unverified not suspicious

The cache is critical. Without it, scanning a single large application can hit rate limits in under a minute.

Results From Production Deployments

Deployed across thousands of website scans:

The three confirmed attacks were all CDN-loaded packages that appeared in production websites but didn’t exist in NPM until attackers registered them. Two were credential harvesters. One was a cryptominer.

All three would have been missed by traditional dependency scanners because they were loaded via <script> tags pointing to CDNs, not declared in package.json.

The Future of This Attack Vector

This is only getting worse. As AI coding assistants become ubiquitous, expect:

1. Namespace Squatting at Scale Attackers will systematically register common typos and hallucinations for every major framework.

2. Framework-Specific Targeting As new frameworks emerge, LLMs will hallucinate packages for them before legitimate ecosystems develop. Attackers will claim the namespace early.

3. Deprecated Package Revival Malicious actors are already claiming abandoned NPM packages. LLMs trained on old data suggest deprecated packages, attackers register them, developers install them.

4. AI-Powered Harvesting Attackers are building their own LLM harvesters that prompt models thousands of times, collect hallucinated package names, and auto-register them.

How to Protect Your Codebase

The Detection Gap in the Industry

Major security vendors are still catching up. As of December 2025:

What existing tools check:

• Known CVEs in declared dependencies
• Outdated package versions
• License compliance

What they DON’T check:

• AI-hallucinated package names
• CDN-loaded packages not in package.json
• Recently registered packages with suspicious names
• Dynamic imports that bypass static analysis

This is the gap vibe_eval fills. The 291-line check scans all of these attack surfaces and flags packages that traditional SCA tools miss entirely.

FAQ

The Uncomfortable Truth

AI coding assistants are making developers more productive. They’re also creating new attack vectors that didn’t exist two years ago.

The problem isn’t the AI. The problem is blind trust. When ChatGPT suggests npm install email-validator-pro, most developers don’t verify that package exists and is legitimate. They just run the command.

Attackers know this. They’re harvesting hallucinated package names, registering them, and waiting.

The only question is whether your security scanning catches these before they execute in your environment.

Conclusion

This isn’t theoretical. It’s happening in production codebases right now. The question is whether your security tools can detect it before it becomes a breach.