What is AI security? Definition, threats, and defenses explained

Updated September 2026 with IBM’s 2026 breach data, the EU Digital Omnibus deadlines, and OWASP’s 2026 LLM Top 10.

Most AI breaches don’t start with an exotic model attack. They start with an inference endpoint nobody authenticated, or an agent holding far more permissions than its job needed.

This guide covers what securing an AI system actually involves: the layers you’re protecting, the threats that hit each one, the frameworks that apply, and where to start if you’re responsible for models in production.

Jump to: Definition · AI security vs. related terms · Why it matters now · What you’re protecting · Across the lifecycle · Threats · LLM and RAG applications · Frameworks · Getting started · AI agents · FAQ

What is AI security?

AI security is the set of processes, controls, and policies that protect AI systems, including their training data, models, pipelines, and the infrastructure that serves them, from attack, theft, misuse, and failure. It applies to every stage of a model’s life, from data collection through training, deployment, and retirement.

In a security context, “AI” means artificial intelligence: machine learning models, large language models (LLMs), and the agents built on top of them. What changes is the attack surface. A traditional application fails when its code has a bug. An AI system can fail with flawless code, because its behavior comes from data and weights an attacker can influence.

That’s the practical reason AI security became its own discipline instead of a line item under application security. You can ship bug-free code and still deploy a model an attacker controls.

How is AI security different from AI safety and AI for cybersecurity?

Three terms get used interchangeably, and they answer different questions.

Term	The question it answers	Example
AI security	Is the AI system protected from someone trying to attack, steal, or misuse it?	An attacker poisons a fine-tuning dataset
AI safety	Does the system behave as intended when nobody is attacking it?	A model gives harmful advice on its own
AI for cybersecurity	Can AI help defend the rest of the organization?	A model triages SOC alerts

This guide is about the first row.

The first two overlap constantly. A model that’s easy to manipulate with adversarial inputs (a security failure) can produce unsafe outputs as a direct result (a safety failure), which is why mature AI governance programs staff both functions instead of folding one into the other.

The third row loops back to the first. A detection model in your SOC is itself an AI system, with training data, weights, and API keys an attacker would love to reach. If that’s the side of the topic you came for, our explainer on how generative AI affects security covers it.

Why does AI security matter in 2026?

Adoption and breach costs have been climbing for years. What’s new in 2026 is that attacks on AI systems themselves are showing up in breach data, and the regulatory deadlines are fixed.

Adoption outran security. Stanford’s 2026 AI Index reports organizational AI adoption at 88%. Models now sit inside loan approvals, hiring screens, medical imaging, and customer support, and every deployment adds endpoints, data flows, and credentials someone has to protect.

AI systems are now breach targets. IBM’s 2026 Cost of a Data Breach Report, based on 602 organizations breached between March 2025 and February 2026, found that more than 20% of organizations reported a breach targeting their AI models or applications. The global average breach cost hit a record $4.99 million, up 12% from the year before.

Regulation now has dates attached. The EU AI Act is in force, with fines of up to €35 million or 7% of global annual turnover for the most serious violations. The Digital Omnibus amendment, published in July 2026 as Regulation (EU) 2026/1744, moved the compliance deadline for stand-alone high-risk systems from August 2, 2026 to December 2, 2027. The obligations were deferred, and they still apply.

What exactly are you protecting in an AI system?

Securing AI is a stack. A failure at any layer undermines everything above it.

Layer	What it protects	Key controls
Data	Training and fine-tuning data, plus documents retrieved at inference	Provenance tracking, access controls, encryption at rest and in transit
Model	Weights, parameters, and behavior	Adversarial testing, rate limiting, API authentication, differential privacy
Infrastructure and pipeline	Training clusters, model registries, inference APIs, dependencies	Dependency scanning, environment isolation, endpoint hardening
Governance	Accountability and regulatory alignment	Risk assessment, documentation, bias testing, audit trails

Most practical defenses map to the CIA triad, applied to models and data instead of servers. Confidentiality keeps training data, weights, and prompts visible only to authorized people and systems. Integrity means nothing in the pipeline gets tampered with, by an attacker or by a misconfiguration nobody caught. Availability means inference endpoints keep answering under load or under attack.

Two controls carry most of the practical weight:

• Role-based access control (RBAC) scopes who can read training data, push to a model registry, or call a production endpoint.
• Differential privacy adds calibrated noise during training so no single record can be reconstructed from what the model learned. It’s a standard countermeasure to model inversion, and it costs accuracy, so the balance needs tuning per use case.

Neither control works alone. RBAC without differential privacy still lets a compromised but authorized account run inversion attacks. Differential privacy without RBAC leaves the raw training data open to anyone with access.

Encryption belongs on the same list, and it’s often missing: in IBM’s 2026 study, only 37% of breached organizations encrypted sensitive data both at rest and in transit.

For the data layer in depth, see our guide to AI data security.

Where do AI systems get attacked across the lifecycle?

Threats don’t hit “AI” in the abstract. They hit a specific system at a specific stage, and each stage needs different controls.

Stage	What can go wrong	Controls that matter most
Data collection and labeling	Poisoned or unvetted data enters the training set	Source allowlists, provenance records, anomaly detection on new data
Training and fine-tuning	Tampered base models or libraries, exposed training clusters	Verified model sources, dependency scanning, isolated training environments
Model storage and distribution	Weights stolen or swapped in the registry	Registry access controls, signing, audit logs
Deployment and inference	Prompt injection, adversarial inputs, extraction via the API	Authentication, rate limiting, input and output filtering, monitoring
Retirement	Old versions left reachable and unpatched	Formal deprecation, revoked credentials, endpoint removal

The stage teams skip is the last one. A deprecated model left running next to its replacement is still reachable, still an attack surface, and nobody’s watching it.

The inference stage gets the most attention because it’s where attackers can reach a model without any internal access, which is the territory AI runtime security covers. It’s also where the fastest-growing class of threats, the ones specific to LLM applications, lives.

What are the biggest AI security threats?

Ten threats cover most of the territory. Each one hits a different layer and needs its own defense.

Threat	What it does	Primary defense
Prompt injection	Crafted input overrides an LLM’s instructions	Separating instructions from data, least privilege, output filtering
Data poisoning	Corrupts training data to change model behavior	Data provenance, anomaly detection
Sensitive information disclosure	The model leaks training data, secrets, or confidential context	Data minimization, output filtering, access controls
Excessive agency	An LLM or agent takes actions beyond what its task requires	Scoped permissions, human approval for high-impact actions
Adversarial attacks	Subtly altered inputs flip a model’s output	Adversarial training, stress testing against perturbed inputs
Model extraction	Repeated queries reconstruct a proprietary model	Rate limiting, query-pattern monitoring
Model inversion	Model outputs are used to reconstruct training data	Differential privacy
Supply chain attacks	A tampered model, dataset, or library enters the pipeline	Component verification, AI bills of materials (AIBOMs)
Shadow AI	Unsanctioned AI tools leak data and add unmanaged attack surface	Discovery, policy enforcement, approved alternatives
Misconfiguration	Exposed endpoints, public storage buckets, default credentials	Configuration audits, RBAC, endpoint hardening

Four of these deserve more than a table row.

Prompt injection is the signature attack on LLMs, and it holds the top spot on OWASP’s 2026 LLM list. It works because a model treats developer instructions and user-supplied text as one stream of input. The indirect version is worse: a web page, email, or document the model retrieves can carry instructions of its own. OWASP’s own guidance puts it bluntly: “it is unclear if there are fool-proof methods of prevention for prompt injection.” The realistic plan is to assume the model can be tricked and limit what a tricked model can reach.

Data poisoning is one of the hardest threats to catch after the fact. A fraud model that learned to ignore one transaction pattern looks healthy on every dashboard until someone exploits the gap. You can’t patch poison out of a trained model. You retrain, which is why provenance (knowing what went into the model and who touched it) matters more than any downstream fix.

Supply chain attacks exploit how much of a model you didn’t build. Base models from public hubs, third-party datasets, and training libraries all arrive with implicit trust. Model files deserve particular suspicion: pickle, the default serialization format for PyTorch weights, can execute arbitrary code when a file is loaded. Formats like safetensors, which store only tensor data, exist as the safer alternative. Treat an unverified model download the way you’d treat an unsigned binary.

Misconfiguration is the least glamorous threat and one of the most common. In IBM’s 2026 study, the leading causes of breaches that targeted AI were compromised APIs, applications, or plug-ins (27%) and cloud misconfigurations affecting AI workloads (27%). Neither needs a novel exploit. Both need a gap nobody closed.

For every threat in detail, with examples and mitigations, see our breakdown of AI security risks.

How do you secure LLM applications and RAG pipelines?

Secure the retrieval and output boundaries as carefully as the model itself, because that’s where these applications break. Most teams aren’t training models from scratch. They’re wrapping a hosted or open-weight LLM in an application and connecting it to company data through retrieval-augmented generation (RAG). That architecture has its own weak points.

• Retrieval inherits your permission problems. If the vector store holds every document in the company and the retriever ignores who’s asking, the model will happily summarize a board memo for an intern. Enforce the user’s existing access rights at retrieval time, not after generation.
• Retrieved content is untrusted input. Any document, web page, or ticket the model reads can carry an injected instruction. Keep retrieved text clearly separated from system instructions, and never let retrieved content alone trigger a privileged action.
• Model output is untrusted too. OWASP lists output handling as its own risk for a reason. Output that flows into a browser, a shell, a SQL query, or another API needs the same validation and encoding you’d apply to user input.
• Context is broader than the prompt. OWASP’s 2026 list renamed and broadened its system prompt leakage category into “hidden context exposure,” covering context users never see, such as system instructions, tool schemas, and retrieved data. Assume anything in the context window can be extracted by a determined user, and keep secrets out of it.

Our guides to LLM security and securing AI applications go deeper on each control.

Which frameworks govern AI security?

Six frameworks do most of the work, from voluntary guidance to binding law.

Framework	Type	What it’s for
NIST AI RMF	Voluntary US guidance	Organizing AI risk into four functions: govern, map, measure, manage
EU AI Act	Binding EU regulation	Risk tiers, conformity assessments, transparency, human oversight
OWASP Top 10 for LLM Applications (2026)	Community threat list	The ten most critical LLM application risks, with mitigations
OWASP Top 10 for Agentic Applications	Community threat list	Risks specific to autonomous agents with tools, memory, and credentials
ISO/IEC 42001	Certifiable management system standard	An auditable AI management program that sits alongside ISO/IEC 27001
MITRE ATLAS	Adversary knowledge base	Real-world adversarial ML tactics, structured like MITRE ATT&CK

NIST AI RMF. NIST describes it as “intended for voluntary use,” and it’s where we’d suggest most US programs start, because its four functions give every other control on this page a place to live.

EU AI Act. It reaches any organization that places an AI system on the EU market or whose system’s output is used in the EU. After the Digital Omnibus, stand-alone high-risk systems (hiring tools, credit scoring, biometric identification, and similar Annex III uses) must comply by December 2, 2027. High-risk AI embedded in regulated products such as medical devices has until August 2, 2028. The Article 50 transparency obligations still apply from August 2, 2026, with a short transition period for some systems already on the market.

OWASP. The 2026 edition of the LLM Top 10, published in August 2026, keeps prompt injection first and sensitive information disclosure second. Excessive agency jumped to third, and system prompt leakage was folded into the broader hidden context exposure category. The separate Top 10 for Agentic Applications, released in December 2025, covers risks such as agent goal hijacking and tool misuse. If you’re building agents, read both.

ISO/IEC 42001 is the one to pick when you need a certification a board or a customer will recognize. Google’s Secure AI Framework (SAIF), a conceptual framework built on six core elements, is worth reading alongside it, and our overview of AI governance frameworks compares the governance-side options in more depth.

How do you start an AI security program?

Start with visibility, then work outward. The order matters more than the tooling.

1. Inventory every AI asset. Models, datasets, pipelines, vector stores, API keys, and the AI tools employees already use without approval. AI security posture management (AI-SPM) tools automate the discovery. Every later step depends on this map.
2. Threat-model each system. Decide which of the threats above actually apply, and at which lifecycle stage. A fine-tuned internal classifier and a customer-facing agent with tool access have very different risk profiles.
3. Lock down access. Apply least-privilege access controls to data stores, model registries, and inference endpoints. Agents get the minimum permissions their task needs, and every tool call gets logged.
4. Test like an attacker. AI red teaming needs two skill sets that rarely live in one person: someone who understands networks and APIs, and someone who understands model behavior. MITRE ATLAS is a solid playbook to build around.
5. Monitor and plan the response. Track performance, input distribution, and output patterns in production. A latency spike could be a denial-of-service attempt or a config problem, and an odd output pattern could be manipulation or plain drift. Decide in advance who can pull a model offline, and what you roll back to, before you need to.
6. Retire old models on purpose. Trace every deployed model to a training run and dataset snapshot (good MLOps tooling records this already), gate promotion to production behind security review, and revoke access to deprecated versions.

For the full control-by-control checklist, see AI security best practices.

What changes when AI agents enter the picture?

Agents turn a text risk into an action risk. A prompt-injected chatbot writes a bad answer. A prompt-injected agent can send the email, run the query, or push the code.

That’s the reason excessive agency climbed OWASP’s 2026 list, and the reason the Model Context Protocol (MCP), an open standard for connecting AI applications to tools and data, has become a security topic of its own. CVE-2025-6514, a flaw in the mcp-remote client, carried a critical 9.6 CVSS score and allowed remote code execution. Researchers have since found command-injection bugs in other MCP servers too, including ones for Git and the Serverless Framework.

The guardrails are familiar, just applied more strictly: permissions scoped per tool, instructions kept apart from untrusted content, human approval for irreversible actions, and monitoring of what agents actually do. Our agentic AI security guide covers each of these, and the agentic AI security checklist turns them into pre-deployment gates.

Frequently asked questions

What does AI mean in security?

In security, AI stands for artificial intelligence. AI security specifically means protecting AI systems (models, training data, pipelines, and the endpoints that serve them) from attack and misuse. Using AI tools to defend other systems is a related but separate field, usually called AI for cybersecurity.

Which are two security risks of generative AI?

Prompt injection and sensitive information disclosure, the top two entries on OWASP’s 2026 LLM Top 10. Prompt injection lets crafted input override a model’s instructions. Sensitive information disclosure is a model revealing training data, secrets, or confidential context it has access to.

What is the biggest security risk with using AI?

For LLM applications, prompt injection ranks first on OWASP’s 2026 list, and no complete technical fix exists yet. Data poisoning is among the hardest to detect, since a poisoned model can behave normally for months before the flaw gets exploited.

Which AI systems count as high-risk under the EU AI Act?

AI used in areas such as critical infrastructure, employment and worker management, education, access to essential services like credit, law enforcement, migration and border control, and the administration of justice. Stand-alone high-risk systems must comply by December 2, 2027, and they face conformity assessments, human oversight requirements, and detailed technical documentation.

What is shadow AI and how is it addressed?

Shadow AI is the use of AI tools or models without the security team’s knowledge or approval, like pasting proprietary code into a public chatbot. The response starts with finding what’s already in use, then blocking high-risk services, offering approved alternatives, and teaching employees what data never goes into a public tool.

What is AI security posture management?

AI security posture management (AI-SPM) continuously inventories an organization’s AI assets, including models, datasets, pipelines, and AI-enabled applications, and flags misconfigurations, shadow deployments, and compliance gaps against frameworks like the NIST AI RMF.

Are AI applications safe to use?

They’re as safe as the controls around them. An AI application with protected data, authenticated APIs, scoped permissions, and active monitoring is reasonably safe, and the same application without those controls isn’t. The implementation and the oversight determine the outcome.

Next steps

Start small. Pick one production AI system, trace it through the lifecycle table above, and write down who can reach it at each stage. Fix the highest-risk gap before you expand.

For implementation detail, NIST’s AI Risk Management Framework resources walk through each function, and the OWASP GenAI Security Project publishes free mitigation guidance for every risk on both of its Top 10 lists. If you operate in the EU, our EU AI Act compliance checklist maps the obligations to engineering tasks.