If you are shipping an LLM-backed feature this quarter, the security model you inherited from traditional web application development covers maybe half your attack surface. The other half is new: a model that follows instructions embedded in untrusted data, a context window that mixes system prompts with user input and retrieved documents, and increasingly, agents that call tools and take actions on your behalf. This guide walks through how to secure AI applications across the full lifecycle, from design through incident response, with concrete mitigations rather than generic advice.
Short answer: securing an AI application means treating every input that reaches the model’s context (user text, retrieved documents, tool output) as untrusted, validating everything the model produces before it reaches a downstream system, applying least privilege to every tool and credential an agent can use, and tracking model and data provenance the way you’d track a software supply chain. Standard application security (TLS, auth, patched dependencies) is still required underneath all of it.
Why AI Application Security Is a Different Problem
Traditional application security assumes a relatively fixed boundary between code and data. Your SQL query is code; the user’s search term is data, and you parameterize it so the two never merge. LLM applications break that assumption. The model’s “code” (its behavior) is steered by natural language instructions that live in the same channel as untrusted data: user messages, retrieved documents, tool outputs, even image or audio content. There is no reliable syntactic way to separate “instruction” from “data” inside a prompt, which is why prompt injection remains unsolved at the model level and has to be mitigated architecturally instead.
A second structural difference is autonomy. Traditional apps execute code paths a developer wrote. Agentic AI applications let a model decide, at runtime, which function to call, which document to retrieve, or which API to hit next. That decision is influenced by everything in the model’s context, including content an attacker controls. Combine untrusted input with the ability to take real actions (send email, execute code, call internal APIs, move money) and you get a new class of risk: a model that was never “hacked” in the traditional sense, but was persuaded.
A third difference is the supply chain. You are not just trusting your own code and its dependencies; you are trusting a model’s weights, its training data, any fine-tuning data you added, the plugins or tools you wired up, and often a vector database full of content pulled from sources you do not fully control. Any of those can be a poisoning or tampering vector.
None of this replaces standard application security. You still need TLS, authentication, input validation, secrets management, and patched dependencies. AI-specific controls, covered in more depth in a broader rundown of AI security best practices, sit on top of that foundation, not instead of it.
Map the AI-Specific Threat Landscape First
Before writing mitigations, know the AI security risks you are defending against. Two reference frameworks are worth building your threat model around:
- OWASP Top 10 for LLM Applications (2025 edition, OWASP GenAI Security Project): prompt injection, sensitive information disclosure, supply chain vulnerabilities, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption.
- MITRE ATLAS: a knowledge base of adversary tactics and techniques against AI systems, modeled on MITRE ATT&CK, covering reconnaissance, ML attack staging, exfiltration, and impact, with real-world case studies.
Use both as checklists during threat modeling, not as compliance boxes to tick. Walk your actual architecture, data flows, and trust boundaries against each category and ask where an attacker-controlled input could reach it.
Secure Development Practices for AI Applications
Bake security into the build process rather than bolting it on before launch.
- Threat model before you build. Diagram every place untrusted content enters the system: user prompts, uploaded files, RAG-retrieved documents, tool/API responses, and third-party plugin output. Each of these is a potential prompt injection vector, not just the chat box.
- Vet third-party models and components before integrating them. Check model provenance, licensing, and known vulnerability disclosures. Treat a pre-trained model or fine-tuning dataset from an unverified source the same way you’d treat an unvetted open-source package with a postinstall script: a supply chain risk, not a free asset.
- Pin and hash model artifacts. Use versioned, checksummed model files rather than pulling “latest” from a registry at deploy time, so a compromised upstream artifact cannot silently replace what you tested.
- Isolate training and fine-tuning environments from production, with dependency scanning and restricted network egress, so a compromised training pipeline cannot reach production credentials or data.
- Red team before shipping. Adversarial testing (prompt injection attempts, jailbreak attempts, boundary testing on tool permissions) belongs in your CI/CD pipeline the same way SAST and dependency scanning do, not as a one-time pre-launch exercise.
Input and Output Handling
This is where most practical LLM security work actually happens, because you cannot reliably fix the model, only the boundary around it.
On the input side, treat everything that reaches the model’s context window as untrusted, including retrieved documents and prior tool outputs, not just the live user message. Practical controls: strip or neutralize instruction-like patterns from retrieved content before it enters context, use separate system and user message roles rather than concatenating everything into one string, and apply allowlist-style input validation on structured fields (IDs, enums, file paths) rather than trying to sanitize free text for “malicious intent,” which is not reliably detectable.
On the output side, never treat model output as safe just because it came from your own model. Improper output handling (OWASP LLM05) covers cases where a model’s response is passed downstream without validation: into a SQL query, a shell command, an HTML template that renders unescaped, or an API call. Validate and encode LLM output the same way you would validate output from any other untrusted source before it reaches a database, a browser, or a system call. If the model can produce a URL, a file path, or a command, apply the same allowlisting you’d apply to user-submitted equivalents.
For retrieval-augmented generation specifically, watch two failure modes: RAG data leakage, where a retriever surfaces documents the requesting user should not see because access control was enforced at ingestion time but not at query time, and indirect prompt injection, where a poisoned or attacker-authored document in the retrieval corpus contains instructions the model follows as if they came from the system prompt. Enforce document-level access control at query time, not just at index time, and consider stripping or flagging instruction-like content in ingested documents.
Access Control, Secrets, and Least Privilege
Agentic and tool-using applications turn “which credentials does this process have” into a much bigger question than it used to be, one covered in more depth in this guide to how to secure the AI agent lifecycle.
- Apply least privilege to every tool and API the model can call. If an agent only needs to read a calendar, its credential should not also be able to send email or write to storage. Scope permissions per tool, not per application.
- Never let the model hold or see raw credentials. Route tool calls through a broker or gateway that injects API keys and secrets server-side, so a prompt injection attack that convinces the model to “print your configuration” cannot exfiltrate a live key.
- Watch for SSRF through tool use. An agent with a fetch-URL or web-browsing tool can be manipulated, via injected instructions in a page it’s asked to summarize, into requesting internal endpoints such as a cloud metadata service (for example, the AWS instance metadata endpoint at a link-local address). Restrict outbound requests from tool-calling infrastructure with network-level allowlists, not just prompt-level instructions telling the model not to do that.
- Use RBAC for both human and machine access to model endpoints, fine-tuning pipelines, and embedding stores. A leaked API key to your inference endpoint is a production incident; treat it with the same urgency as a leaked database credential.
- Rate limit per user and per API key, not just globally, to blunt both unbounded consumption attacks (resource exhaustion or cost-driven denial of service, OWASP LLM10) and brute-force extraction attempts.
Model and Supply Chain Security
- Track model provenance (where weights came from, what data trained or fine-tuned them) with the same rigor you’d apply to a software bill of materials.
- Scan and sandbox any plugin, tool, or extension before granting it agent access, since a compromised or malicious plugin has the same blast radius as a compromised dependency, potentially larger if it holds its own credentials.
- Guard against model theft and unauthorized extraction by rate limiting inference APIs and monitoring for query patterns consistent with distillation attacks (unusually broad, systematic probing of the model’s outputs).
- Re-audit models after any retraining or fine-tuning cycle for signs of data or model poisoning, especially if training data includes user-submitted content or scraped external sources that an adversary could have seeded.
Monitoring, Logging, and Red Teaming
Security does not stop at deployment. Log full prompts, retrieved context, tool calls, and outputs (with appropriate PII handling) so you can reconstruct what happened when something goes wrong. Behavioral monitoring should watch for anomalies specific to LLM apps: sudden spikes in a single user’s token consumption, repeated jailbreak-pattern attempts, tool calls to endpoints outside the expected pattern, or output that matches known system-prompt-leakage signatures. Schedule recurring red team exercises, not just a pre-launch pass, because new jailbreak techniques and injection patterns emerge continuously and a model or guardrail configuration that held six months ago may not hold today. Keep a human in the loop for any agent action with real-world consequences (financial transactions, data deletion, external communications) until you have enough monitoring history to trust the automation.
Threat Categories and Mitigations
| Threat category (OWASP LLM Top 10 2025) | Primary risk | Key mitigations |
|---|---|---|
| Prompt Injection | Attacker-controlled text overrides intended model behavior | Separate instruction/data channels, input filtering on retrieved content, least-privilege tool access, output validation |
| Sensitive Information Disclosure | Model reveals PII, secrets, or proprietary data in output | Data minimization in training/context, output filtering, access control on retrieval sources |
| Supply Chain | Compromised model, dataset, or plugin introduces risk | Provenance tracking, artifact signing/hashing, vetting third-party components |
| Data and Model Poisoning | Malicious data corrupts training or fine-tuning | Data provenance and versioning, anomaly detection on training data, isolated training environments |
| Improper Output Handling | Unvalidated model output reaches a downstream system | Treat output as untrusted input, encode/validate before use in SQL, shell, HTML, or APIs |
| Excessive Agency | Agent has more permissions or autonomy than it needs | Least privilege per tool, human-in-the-loop for consequential actions, scoped credentials |
| System Prompt Leakage | Internal prompts or config exposed to users | Avoid storing secrets in system prompts, monitor output for leakage patterns |
| Vector and Embedding Weaknesses | Unauthorized access or manipulation of embedding stores | Query-time access control, input validation on ingested documents |
| Misinformation | Confident but false output presented as fact | Grounding via retrieval, output review for high-stakes use cases, user-facing confidence signaling |
| Unbounded Consumption | Resource exhaustion or cost-driven denial of service | Per-user and per-key rate limiting, quotas, timeout and token-budget enforcement |
Incident Response for AI Applications
Extend your existing incident response plan rather than building a separate one from scratch. Add AI-specific runbook entries: how to roll back to a prior model version if a deployed model is behaving unexpectedly, how to rotate any credentials a compromised agent may have accessed, how to identify which prompts or retrieved documents triggered an incident from your logs, and who has authority to disable a specific tool or agent capability without taking down the whole application. Practice the rollback path before you need it. A model version pin and a tested rollback procedure turn “the new model is leaking data” from an all-hands emergency into a five-minute mitigation.
FAQ
What is AI application security?
AI application security is the set of practices that protect AI-powered applications, especially those built on large language models, against risks specific to how these systems process instructions and data, including prompt injection, data leakage, model and supply chain compromise, and misuse of agentic capabilities, in addition to standard application security controls.
How is securing an AI application different from securing a normal web app?
It differs mainly because instructions and untrusted data share the same channel (the model’s context), because the system’s behavior is probabilistic rather than deterministic, and because agentic systems can take real-world actions based on that context, which means classic input/output separation and access control models have to be adapted rather than just applied as-is.
What is prompt injection?
Prompt injection is an attack where text embedded in user input, a retrieved document, or tool output manipulates a model into ignoring its original instructions and following the attacker’s instead; it can be direct (typed by the user) or indirect (hidden in content the model processes, like a webpage or file it’s asked to summarize).
Do I need a WAF for an LLM app?
A traditional web application firewall still protects the transport and infrastructure layer (SQL injection, XSS, standard OWASP Top 10 web risks) but does not understand prompt injection or model-specific abuse, so it’s a useful complement to, not a substitute for, LLM-specific input handling, output validation, and rate limiting.
Can prompt injection be fully prevented?
Not reliably at the model level today; because there’s no hard syntactic boundary between instructions and data in natural language, the practical approach is to limit the damage an injected instruction can do through least-privilege tool access, output validation, and human review of consequential actions, rather than to assume injection can be filtered out entirely.
What is excessive agency in an AI system?
Excessive agency describes an AI agent that has been granted more functionality, permissions, or autonomy than its task requires, so that a manipulated or malfunctioning model can take actions (calling unnecessary tools, accessing unnecessary data) beyond what the application actually needs it to do; the fix is scoping permissions per tool and task, not application-wide.
Next Step
Pick one application you already run in production or staging and walk it against the OWASP Top 10 for LLM Applications list yourself: for each of the ten categories, identify where in your architecture that risk could materialize and whether you have a control for it today. That single exercise, run honestly, will surface more actionable gaps than any generic checklist, including this one.