Data Poisoning
: An attack against machine learning systems where an adversary manipulates training data to influence the model’s behavior in a desired way. For code generation models, data poisoning means injecting vulnerable code patterns into training data (public repositories, documentation) so the model learns to reproduce these insecure patterns in its output.
Why It Matters for AI-Coded Apps
If training data contains deliberately planted insecure code patterns, every developer using the poisoned model receives vulnerable code suggestions. This is a supply chain attack at the AI model level – instead of poisoning one package, the attacker poisons the model that generates code for millions of developers.
Real-World Example
A researcher demonstrates that by contributing thousands of commits to popular open-source projects with subtle backdoors (e.g., using a weak random number generator), they can influence code generation models to suggest the same weak patterns. The poisoned patterns appear in AI-generated code across all projects using that model.
How to Detect and Prevent It
Use AI models from reputable providers with documented training data curation practices. Do not blindly trust AI-generated security-critical code (cryptography, authentication, authorization). Apply the same security review to AI-generated code as you would to untrusted third-party code. Run SAST and DAST tools on all generated code.
Frequently Asked Questions
Can I tell if an AI model has been poisoned?
It is extremely difficult to detect data poisoning in large models. The poisoned patterns are subtle and appear in statistically normal-looking code. Vigilance through security review, automated scanning, and testing is the practical defense. Model providers invest in training data curation to prevent poisoning.
How is data poisoning different from prompt injection?
Data poisoning targets the model’s training data, permanently altering its behavior for all users. Prompt injection targets a specific application session by manipulating input. Data poisoning is a supply chain attack on the model; prompt injection is a runtime attack on the application.
Who is at risk from data poisoning?
Every developer using AI code generation tools is potentially affected. The risk is highest for security-critical code (authentication, cryptography, access control) where a subtle insecure pattern could go undetected. Using multiple AI tools and manual review reduces the risk of any single poisoned model affecting your code.
Scan your app for security issues automatically
Vibe Eval checks for 200+ vulnerabilities in AI-generated code.
Try Vibe Eval