What Is LLM Security?

LLM security is the practice of protecting large language models — and every layer they depend on — from adversarial attacks, data leakage, and misuse. It covers training data integrity, inference endpoints, prompt handling, supply chain risks, and the cloud infrastructure underneath. The core challenge: LLMs treat input as both data and instruction, which means attackers don’t need to break in. They just need to ask the right way.

---

How LLM security differs from traditional application security

Traditional security assumes a clean boundary between code and data. You sanitize inputs, validate outputs, lock down the perimeter. The logic doesn’t change unless a developer changes it.

LLMs broke that assumption.

A SQL injection attack exploits predictable syntax. A prompt injection attack exploits the model’s tendency to follow instructions — and those “instructions” can hide anywhere: a document the model summarizes, a web page it retrieves, a tool response it processes. The attack surface isn’t a port or an endpoint. It’s every token the model processes, and it’s evaluated probabilistically, not deterministically.

Security teams that treat LLMs as “just another API” get hit by the second category while they’re still mapping the first.

Dimension	Traditional app security	LLM security
Attack vector	Code injection, network exploit	Natural language, semantic manipulation
Input boundary	Structured, predictable	Open-ended, unstructured
Threat model	Attacker vs. defined perimeter	Attacker embedded in the data itself
Output	Deterministic	Probabilistic — same input, different risk each time
Supply chain	Libraries, packages, APIs	Models, datasets, fine-tuning pipelines
Compliance surface	Data at rest and in transit	Training data, model weights, inference logs

For a broader look at how AI threats differ from conventional ones, see AI security risks: the complete guide.

---

Top risks for enterprise LLM deployments

Prompt injection

An attacker crafts an input that overrides the model’s system prompt or safety instructions. Direct injection — “ignore previous instructions and do X” — is the obvious case and relatively easy to filter for. Indirect injection is harder: instructions embedded in a document the model reads via a RAG pipeline, a web page it retrieves, or a tool response it processes autonomously. The model sees those instructions as content. It doesn’t know the difference.

Agentic AI systems make this dramatically worse. An agent with access to email, calendar, and file systems can be hijacked by a single malicious document it retrieves without any human in the loop. One injected instruction becomes a chain of real-world actions. The full breakdown of how this attack works and how to defend against it is in prompt injection attacks.

Sensitive information disclosure

Models memorize training data. Not perfectly, but enough to reproduce fragments under the right conditions. Nicholas Carlini and colleagues at Google DeepMind demonstrated systematic extraction of verbatim training text from production models — personal data, code, private documents. The larger the model, the more it memorizes.

The Samsung incident in 2023 made the organizational risk concrete. Engineers at Samsung’s semiconductor division pasted proprietary source code and meeting notes into ChatGPT while debugging — three separate incidents in the same month. The data was processed by OpenAI’s systems and potentially contributed to future training. Samsung banned ChatGPT internally after the third incident. Without AI data security controls at the input layer, every conversation is a potential disclosure event.

Training data poisoning

Insert enough malicious data into a training or fine-tuning dataset and you can reliably shift model behavior. Wan et al. (2023) showed that 100 carefully chosen examples in an instruction tuning dataset were enough to consistently manipulate model outputs. That’s a low bar for a determined attacker. Most organizations don’t audit the provenance of every dataset they use, and third-party fine-tuning providers introduce AI infrastructure risks that are invisible until something surfaces in production.

Supply chain vulnerabilities

Model hubs, open-source weights, third-party APIs, Python packages — the LLM supply chain is long and largely unaudited. PoisonGPT (2023) demonstrated this precisely: Mithril Security uploaded a surgically modified GPT-J model to Hugging Face that performed normally on all benchmarks except one. It returned targeted false information on specific factual questions, reliably, every time. Nothing in the download flow flagged it. The model looked identical to the legitimate version.

Shadow AI and unauthorized usage

A 2025 Proofpoint survey found employees across industries routinely input sensitive data into public LLM tools. This is shadow AI — the AI equivalent of shadow IT, except the data isn’t stored on a personal Dropbox. It’s processed by a model you don’t control, trained on inputs you can’t audit, and subject to retention policies you’ve never seen. AI sprawl compounds this: as teams self-deploy models and build internal tools on top of commercial APIs, the number of unsanctioned deployments grows faster than security teams can track.

Excessive agency

When an LLM operates as an autonomous agent with broad permissions — API access, database writes, external service calls — compromise has cascading consequences. The model doesn’t need to be hacked. It needs to be fooled once, and then it acts. AI agent guardrails address this at the control layer. The agent control plane gives security teams the visibility and interrupt capability to stop a compromised agent before it completes a destructive action sequence.

Insecure output handling

Treat LLM outputs as untrusted data. Full stop. An Air Canada chatbot in 2022 fabricated a bereavement discount policy that didn’t exist. The customer relied on it. Air Canada lost in court — the tribunal ruled the company was liable for what its chatbot said, regardless of accuracy. “The model hallucinated” was not a defense. Output filtering catches some failure modes. Treating the model as an authoritative source catches none.

---

LLM security incidents worth knowing

Tay (2016) — Microsoft’s conversational AI was manipulated by coordinated user inputs within 24 hours of launch. Coordinated prompt injection before anyone called it that. Taken offline in 16 hours.

Samsung / ChatGPT (2023) — Three separate incidents of Samsung engineers pasting proprietary source code and internal meeting notes into ChatGPT. Company-wide ban followed.

PoisonGPT (2023) — A surgically modified LLaMA model uploaded to Hugging Face that passed all standard benchmarks while returning targeted disinformation on specific questions. Demonstrated that supply chain validation for models doesn’t exist in most organizations.

Air Canada chatbot liability (2024) — A fabricated fare policy generated by a customer service chatbot. Air Canada held legally responsible. Precedent for organizational liability from AI outputs.

Carlini et al. training data extraction — Systematic querying of production LLMs by researchers at Google DeepMind, ETH Zurich, and Berkeley demonstrated verbatim extraction of personal information from training data. More effective on larger models. Not a theoretical risk.

---

LLM security frameworks every enterprise should reference

OWASP Top 10 for LLM Applications

Developed by 600+ contributors, this is the closest thing to an industry standard for LLM-specific risk. The 2025 update (v2) reorganized the list around deployment patterns rather than attack categories — a meaningful shift for teams trying to map controls to real workflows. Prompt injection remains #1. The OWASP Top 10 for LLM Applications is the first framework to implement, not because it’s comprehensive, but because it’s the common language your vendors and auditors already use.

MITRE ATLAS

Documents 130+ adversarial attack techniques against ML systems with 26 mitigations. Where OWASP gives you a risk list, ATLAS gives you attacker methodology — how techniques chain, which mitigations break which kill chains. Useful for red team planning and for mapping your current controls to actual attack paths rather than compliance checkboxes.

NIST AI Risk Management Framework (AI RMF)

Four functions: Govern, Map, Measure, Manage. Broader than security — it covers fairness, reliability, explainability — but the Govern and Manage pillars map directly to LLM deployment decisions. AI compliance frameworks like SOC 2 and HIPAA extensions are referencing the AI RMF as a control baseline with increasing frequency.

EU AI Act

High-risk AI obligations take effect August 2026. Consequential deployments — hiring, credit scoring, medical triage, law enforcement support — face documentation requirements, mandatory human oversight, and audit obligations. The definition of “high-risk” is broad. Most enterprise LLM deployments that aren’t pure productivity tools fall within it. AI compliance programs need to account for this now, not after August 2026.

Google Secure AI Framework (SAIF)

Six implementation-focused practices: robust foundations, extending detection to AI systems, automating defenses, harmonizing platform-level controls, adapting controls across the AI lifecycle, and contextualizing risk from threat intelligence. Less prescriptive than OWASP, more useful for security architects thinking about how LLM security integrates with existing enterprise security programs rather than sitting beside them.

---

LLM security best practices that actually work

Input validation and prompt sanitization — Filter for known injection patterns, enforce structured input formats where the use case allows, and treat all user-controlled text as untrusted by default. This is table stakes and catches unsophisticated attacks. It’s also not sufficient on its own.

Least-privilege access — LLMs should have access only to what the specific task requires. An LLM summarizing meeting notes doesn’t need database write access. Most LLM security failures in production come from over-permissioned deployments meeting moderately clever prompt injection rather than sophisticated zero-days.

Output filtering and content moderation — Scan outputs for PII, sensitive data patterns, code injection, and policy violations before they reach end users or downstream systems. Guardrails AI, NeMo Guardrails, AWS Bedrock Guardrails, and Azure AI Content Safety operate at this layer. None of them are set-and-forget.

Adversarial testing — Red-team your LLMs before deployment, not after the first incident. The Adversarial Robustness Toolbox (ART) and CleverHans are the standard libraries. The goal isn’t to prevent every possible attack — it’s to map failure modes before attackers find them in production. AI agent evaluation for agentic deployments needs to go further: test what the agent does when it receives a malicious instruction, not just whether it produces correct outputs.

Data provenance and integrity — Know where your training and fine-tuning data came from. Audit third-party datasets before use. Generate and maintain AI bills of materials (AI-BOMs) the same way you maintain software SBOMs. AI data security controls belong in the MLOps pipeline from the start, not bolted on after the first audit.

Monitoring and observability — Log prompts and completions. Track anomalies in input patterns. Set alerts for unusual token usage or sudden shifts in topic distribution. An LLM that starts discussing topics well outside its deployment scope is worth investigating. Most organizations don’t log enough to know whether an attack has already happened.

For a structured checklist across these controls, see top AI security best practices.

---

LLM security in agentic AI: where the threat landscape is moving

Securing a single LLM responding to prompts is a solvable problem. Securing an AI agent that browses the web, reads email, writes and executes code, and calls external APIs — autonomously, over hours, without a human reviewing each step — is a different discipline.

The attack surface compounds with each capability you add. A prompt injection attack on a chatbot damages trust. The same attack on an agent with Slack and GitHub access sends messages and merges code. The model receives the same instruction. The blast radius is not comparable.

Three controls matter specifically for agentic deployments:

Agent sandboxing — Restrict what environments an agent can touch. Filesystem access, network calls, subprocess execution — each is a capability that can be weaponized. The principle of least privilege applies to agents more aggressively than to any other system because agents act autonomously. An over-permissioned agent that gets compromised doesn’t wait for human approval.

Human-in-the-loop for irreversible actions — Agents should require explicit confirmation before taking actions that can’t be undone: sending email, deleting files, pushing code, executing financial transactions. AI agent guardrails implement this at the control layer. The agent control plane gives security teams real-time visibility and interrupt capability across agent fleets — essential for any deployment where agents are running in parallel.

Minimal session state — Agents that persist context across sessions accumulate risk. An injected instruction in session one can influence behavior in session three. Treat agent memory as a potential attack vector and audit what persists between calls. Multi-agent systems — where one agent’s output becomes another’s input — multiply this risk. Security teams working in this space should read our guide on agentic AI security and how to secure the AI agent lifecycle before deploying at scale.

---

Building an LLM security governance program

Controls without governance revert to the previous state within a quarter. Four things hold:

AI-SPM (AI Security Posture Management) — Tooling that gives security teams visibility into what AI is running, where, and with what permissions. It generates AI-BOMs, surfaces shadow AI deployments, and assesses adversarial attack exposure across the enterprise. Without AI-SPM, you govern the deployments you know about and are blind to the rest. For current options, see top AI runtime security platforms.

Shadow AI governance — Employees are using LLMs whether you’ve approved it or not. Prohibition drives usage underground and removes your visibility entirely. The right response is providing approved alternatives with appropriate controls, then enforcing through DLP policies and CASB integration. AI governance frameworks give you the policy structure to make this work organizationally.

Employee training — Security awareness needs an LLM module. Three things every user needs to know: what prompt injection is and why it means they shouldn’t assume model outputs are safe, which data categories are acceptable to enter into AI tools and which aren’t, and how to report anomalous model behavior. Most successful attacks start with someone who didn’t know the risk existed.

Oversight committee — Legal, compliance, security, and business representation. Not to slow down AI adoption but to make accountability explicit before something goes wrong. When an LLM makes a consequential mistake — and one will — someone needs to own it in advance, not scramble to find the owner afterward.

---

Frequently asked questions

What compliance frameworks apply to LLM deployments?

GDPR applies whenever personal data is processed in training or inference. The EU AI Act’s high-risk obligations take effect August 2026 for consequential use cases. SOC 2 and HIPAA compliance programs are increasingly mapping to the NIST AI RMF as a control baseline. Start with AI compliance frameworks to map your specific obligations.

How often should organizations assess LLM security posture?

At minimum: before any new model deployment, after any significant capability change, and quarterly as a baseline. The threat landscape moves faster than annual cycles — OWASP released a substantive update in 2025. Treat LLM posture assessment the same as vulnerability scanning: continuous where possible, structured intervals where not.

When does it make sense to use an AI-SPM tool?

As soon as you have more than one LLM deployment you didn’t personally provision. AI sprawl happens quickly — teams self-deploy via third-party APIs, fine-tune open-source weights, build internal tools on top of commercial APIs. AI-SPM closes the visibility gap that makes shadow AI dangerous. See top AI runtime security platforms for current tooling options

How does LLM security differ from traditional application security?

LLM security assumes that inputs are untrusted and that the model might be tricked into treating untrusted input as instructions — a threat model traditional application security doesn’t have a category for. See the comparison table above.

How does LLM security relate to agentic AI security?

LLM security is a foundation layer of agentic AI security. Agents introduce autonomous action, multi-step reasoning, tool use, and persistent memory — each of which extends the attack surface beyond what LLM-specific controls address. The controls are the same starting point; the threat model requires going further.