An AI model is not just code with weights attached. It’s a data pipeline, a runtime, a set of API endpoints, and a supply chain of pretrained checkpoints you probably didn’t build yourself. Securing it means locking down all four: the training data, the model artifact itself, the infrastructure it runs on, and the inputs and outputs flowing through it at inference time. Skip any one layer and attackers have a way in, as the ShadowRay, DeepSeek, and Sleepy Pickle incidents below all show.
What you’re actually protecting when you “secure a model”
Ask five security teams what “AI model security” means and you’ll get five different answers, because the term covers four distinct layers that get attacked in completely different ways:
- Training data and the pipeline that feeds it, where poisoning and unauthorized access to labeled data happen
- The model artifact itself, vulnerable to theft, extraction, and tampering with weights or serialized files
- The infrastructure it runs on: the servers, orchestration frameworks, and APIs hosting inference
- The inputs and outputs at runtime, where prompt injection, jailbreaks, and data leakage through responses show up
Most teams start with layer 4 because it’s the one vendors sell tools for. That’s backwards. Layer 3, the infrastructure, is where the easiest and most damaging breaches have actually happened.
Why attackers go after the model, not just the app around it
Traditional application security assumes the thing you’re defending is static: a binary, a database schema, a fixed set of endpoints. A deployed model is none of that. It changes behavior based on input, it gets retrained on data you may not fully control, and it often ships inside frameworks nobody on the security team has audited.
Three incidents from the last two years show what that gap actually costs.
In March 2024, Oligo Security disclosed ShadowRay, a flaw in Ray, the open-source framework that companies including Uber, Pinterest, and Netflix use to orchestrate distributed AI training and inference. The bug, tracked as CVE-2023-48022, let attackers run arbitrary code through Ray’s Jobs API with no authentication required. Because the vendor disputed it was a vulnerability, it never got a patch, and thousands of exposed Ray servers were compromised in the wild before anyone outside the affected companies noticed. By late 2025, a follow-on campaign Oligo calls ShadowRay 2.0 had grown the exposed footprint to roughly 230,000 environments, hijacked into a self-propagating cryptomining botnet.
In January 2025, Wiz Research found a DeepSeek database exposed to the open internet: two open ports, no authentication, and full read access to a ClickHouse instance holding more than a million log lines. Plaintext chat histories, API secret keys, and backend operational details were all sitting there for anyone who found the ports. Nothing about the model itself was flawed. The exposure was a database configuration mistake sitting one layer below it.
And in June 2024, Trail of Bits demonstrated Sleepy Pickle, an attack that doesn’t touch your servers at all. It corrupts the model artifact before you ever load it, by injecting a malicious opcode sequence into the Pickle file most PyTorch models ship as. The payload runs the moment the file is deserialized, silently altering the model’s weights or behavior. As Trail of Bits put it, “no trace of the exploit is left on the disk,” no server compromise required, no obvious signal that anything happened.
Three different layers, three different failure modes, none of them prompt injection. That’s the pattern worth internalizing: the model itself is rarely the weakest link. The pipeline and infrastructure around it usually are.
Start with the data and the pipeline, not the model file
Most attacks on AI systems don’t start with the model. They start with the data flowing into it, because that’s the point with the least scrutiny and the widest blast radius once it’s compromised.
- Know where every training source came from. If you can’t trace a dataset back to its origin, you can’t tell whether it was poisoned before you got it. Maintain a data provenance record the same way you’d maintain a software bill of materials.
- Control who can write to training data, not just who can read it. Read access gets audited. Write access to a labeling pipeline or a fine-tuning dataset often doesn’t, and that’s exactly where a poisoning attack gets inserted.
- Treat retraining pipelines as production infrastructure, with the same change control and access logging you’d apply to a deployment pipeline, not as a data science sandbox.
- Encrypt data at rest and in transit through every stage of the pipeline, not just in the final storage bucket. Intermediate preprocessing steps are frequently the least protected part of the chain.
None of this stops an attacker who’s already inside your network. It does mean that when something goes wrong, you can trace exactly which data touched the model and when, which is the difference between a contained incident and a full retrain from scratch.
Harden the model before it ships, not after
Once training data is under control, the model artifact itself needs its own defenses, distinct from the infrastructure question below.
Test it like an attacker would, not like a QA engineer would. Standard model evaluation checks accuracy against a held-out test set. Red teaming checks what happens when someone deliberately tries to break it: adversarial examples crafted to flip a classification, extraction queries designed to reconstruct training data, or inputs built to trigger a jailbreak. MITRE ATLAS catalogs these techniques the way MITRE ATT&CK catalogs conventional intrusion tactics, with 84 documented techniques as of its late-2025 release. Run your model against the tactics relevant to your deployment before an attacker does it for you.
Verify the artifact before you load it, not just its source. Sleepy Pickle worked because nothing in a typical deployment pipeline checks a model file’s integrity after it’s downloaded. A cryptographic hash comparison against a known-good checkpoint catches tampering that a source-trust check alone won’t. Where the framework supports it, use a safer serialization format than raw Pickle in the first place.
Decide what “theft” and “extraction” mean for your specific model, then defend against those, not a generic threat. A public-facing recommendation model leaking its weights is a different problem than a fine-tuned model trained on regulated customer data leaking through repeated inference queries designed to reconstruct that data. Rate limiting and output perturbation address the second case. They do nothing for the first.
Lock down access, encryption, and the runtime
This is the layer every competing checklist covers, and for good reason: it’s foundational, even if it’s not where the more interesting recent breaches happened.
- Apply least-privilege access to the model itself, not just to the data feeding it. Whoever can pull the model artifact from your registry can potentially extract, modify, or redeploy it elsewhere.
- Encrypt the model at rest. A stolen unencrypted checkpoint is a stolen model, full stop, no further exploitation required.
- Put authentication in front of every orchestration and serving API, including internal ones. ShadowRay existed because Ray’s Jobs API had none by default, and nobody assumed an internal tool needed it.
- Rate-limit and monitor inference endpoints for the query patterns that precede extraction or inversion attacks: unusually systematic input sweeps, repeated near-duplicate queries, or volume spikes from a single source.
- Isolate training and inference infrastructure from each other where your architecture allows it, so a compromise of one doesn’t hand an attacker a path to the other.
Which framework should you actually follow?
Every vendor guide name-drops the same four or five frameworks without saying which one to actually start with. They’re not interchangeable, and picking the wrong starting point wastes a quarter.
| Framework | What it actually covers | Best starting point if you’re… |
|---|---|---|
| OWASP Top 10 for LLM Applications | Concrete application-layer risks: prompt injection, insecure output handling, training data poisoning, supply chain vulnerabilities | Shipping an LLM-powered app and need a prioritized risk checklist now |
| NIST AI RMF | Organization-wide governance across four functions: Govern, Map, Measure, Manage | Building AI risk management as a program, not a one-time audit |
| MITRE ATLAS | A tactics-and-techniques knowledge base of real-world attacks against AI systems, modeled on ATT&CK | Running red team exercises or need a common attacker taxonomy for your SOC |
| Google Secure AI Framework (SAIF) | Six-element framework focused on securing the AI development lifecycle at engineering-team scale | You’re an engineering org that already thinks in terms of a secure SDLC and wants an AI-specific extension of it |
| ISO/IEC 42001 | A certifiable AI management system standard, similar in structure to ISO 27001 for information security | You need a formal, auditable certification for customers or regulators |
If you’re doing this for the first time and have to pick one: start with OWASP’s LLM Top 10 for the concrete risk list, then layer NIST AI RMF on top once you need governance structure across multiple models and teams. Add MITRE ATLAS when you start red teaming. Treat ISO 42001 as a certification decision, not a starting security control.
Monitor for the failure, and have a plan for when it happens anyway
None of the above makes a model unbreachable. It reduces the odds and shrinks the blast radius. You still need to know when something got through.
- Log inference requests and responses, not just infrastructure metrics. A model behaving normally by every server-health metric can still be leaking data or producing manipulated outputs.
- Set alerts on drift, not just downtime. A sudden shift in output distribution, confidence scores, or query patterns is often the first visible sign of either a poisoning attack maturing or an extraction attempt in progress.
- Write the incident response plan before you need it, and make sure it answers AI-specific questions a generic IR plan won’t: can you roll back to a prior model version quickly? Do you know which training data a compromised model touched? Can you revoke a leaked model artifact’s access to production data?
- Practice the DeepSeek scenario specifically. An exposed database or open port is a boring, common misconfiguration, not an exotic AI attack, and it’s exactly the kind of finding that shows up in a routine cloud security scan if anyone runs one against your AI infrastructure the way they would against any other production system.
A 90-day checklist, sized to your team
Not every team has a dedicated AI security function. Here’s what to prioritize depending on where you’re starting from.
If you’re a small team shipping your first AI feature (weeks 1-4):
- Put authentication on every model-serving and orchestration API, no exceptions for “internal-only” tools
- Encrypt model artifacts at rest in your registry
- Add rate limiting to any public-facing inference endpoint
If you have models in production already and no formal review process (weeks 4-8):
- Run a cloud configuration scan specifically against AI infrastructure, not just your general cloud footprint
- Establish data provenance tracking for anything currently in a training pipeline
- Pick one framework from the table above and map your current controls against it to find the gaps
If you’re scaling to multiple models and teams (weeks 8-12):
- Stand up a model registry with integrity verification before deployment
- Build AI-specific incident response runbooks, tested against a tabletop exercise
- Start red teaming against MITRE ATLAS techniques relevant to your deployment pattern
Common questions about securing AI models
How can you tell if a model has been compromised?
Look for behavior that diverges from its validation baseline: confidence scores drifting on inputs that used to score consistently, outputs that don’t match the training distribution, or inference logs showing query patterns consistent with extraction (systematic sweeps, high-volume near-duplicate requests). A cryptographic integrity check against the original checkpoint hash catches artifact-level tampering like Sleepy Pickle before the model is even loaded.
How often should you re-run a security review?
Whenever the model is retrained, whenever the serving infrastructure changes, and on a fixed quarterly cadence regardless. Retraining on new data resets your provenance guarantees. An infrastructure change can silently reopen a port or drop an authentication check, exactly what happened with Ray’s Jobs API.
Where should you store an AI model securely?
In a model registry with access controls and integrity verification built in, not a general-purpose file share or an S3 bucket with default permissions. Encrypt the artifact at rest, log every pull, and verify its hash against a known-good value before it’s loaded into a serving environment.
Do these risks apply if you’re only using a third-party model through an API, not hosting one yourself?
Partially. You’re no longer responsible for training data poisoning or artifact theft, since the vendor holds that risk. You’re still fully responsible for runtime inputs and outputs: what you send to the API, what comes back, and whether your own logging exposes sensitive data in either direction. Vet the vendor’s own posture against the frameworks above before assuming their security is your security.
If you’re building this out for the first time, don’t try to implement every control in this guide simultaneously. Start with the infrastructure basics in the 90-day checklist, since that’s where the ShadowRay and DeepSeek incidents both landed, and build outward from there as your model footprint grows.