Key takeaways
- Sensitive information disclosure is when an LLM reveals private or regulated data in its output. OWASP ranks it LLM02:2025.
- There are four main causes: memorized training data, leaked system prompts, prompt injection as delivery, and people pasting information in.
- Layer the fixes: redact inputs, gate retrieval by permission, scan outputs, write policy.
LLM sensitive information disclosure is when a large language model reveals private, confidential, or regulated data through its output, whether that data came from its training set, a connected document store, or the conversation itself.
OWASP ranks sensitive information disclosure as LLM02 in its 2025 Top 10 for LLM Applications, right behind prompt injection. It’s a category of failure that shows up in four or five different ways, and most teams only notice it after something has already leaked.
How LLMs end up leaking sensitive information
There’s no single mechanism here. Four causes show up again and again across real incidents.
1. Training data memorization. Large models sometimes memorize exact sequences from their training data, especially text that appeared many times or was unusually distinctive. A November 2023 paper from Google DeepMind and academic collaborators, “Scalable Extraction of Training Data from (Production) Language Models,” showed this.
The researchers built a “divergence attack” that made ChatGPT stop behaving like a chatbot and instead emit raw training data, at a rate 150 times higher than normal operation. Some of what came out was personally identifiable information. The paper’s core finding: alignment and reinforcement learning from human feedback (RLHF) reduce this behavior, but don’t eliminate it.
2. System prompt leakage. Every LLM application runs on a system prompt, the hidden instructions that define its behavior, tone, and constraints. Ask the right way (or the wrong way, depending on your role) and some models will reprint that prompt verbatim, along with any credentials, internal logic, or business rules someone embedded in it.
3. Prompt injection as the delivery mechanism. Sensitive information disclosure is the payoff; prompt injection is often the exploit that gets you there. An attacker crafts input, direct or hidden inside a document the model retrieves, that overrides the model’s instructions and coaxes it into repeating something it shouldn’t. OWASP lists prompt injection as LLM01, one slot above sensitive information disclosure, and the two are frequently chained together.
4. Human input, not model failure. Sometimes there’s no attack at all. Someone just pastes something they shouldn’t into a chat box.
Cyberhaven’s 2023 analysis of 1.6 million corporate users found that 4.7% of employees had pasted sensitive data into ChatGPT at least once, and that sensitive material made up 11% of everything employees pasted into the tool.
In one case Cyberhaven documented, an executive dropped a company’s entire 2023 strategy document into ChatGPT and asked for a slide deck. In another, a doctor entered a patient’s name and diagnosis to draft an insurance letter. Neither model did anything wrong.
This is shadow AI in its purest form: no attack, no malicious insider, just a tool used outside any policy that accounted for it. The data was already gone the moment it was typed.
What kind of data is actually at risk
“Sensitive information” covers more ground than most people assume when they first hear the term, and it overlaps heavily with the broader set of AI privacy risks organizations are already tracking.
| Data type | Example | How it typically leaks |
|---|---|---|
| PII | Names, emails, phone numbers | Training data memorization, careless user input |
| Credentials and API keys | Hardcoded secrets in code | Code-completion suggestions, system prompt leakage |
| Protected health information | Diagnoses, patient names | Employees pasting records into chat tools |
| Source code and IP | Proprietary algorithms, unreleased features | Employees using LLMs as debugging assistants |
| Business strategy | Internal decks, pricing, M&A plans | Employees using LLMs for document generation |
Credentials deserve their own callout, because the exposure path is different from the rest.
A 2024 study from researchers at the Chinese University of Hong Kong and Sun Yat-sen University, published at ACM FSE 2024, tested GitHub Copilot with 900 prompts built from real GitHub code that had secrets redacted out. Copilot filled the secrets back in 2,702 times out of 8,127 attempts, a 33.2% hit rate, and 200 of those were live, working credentials pulled from actual repositories.
Separately, GitGuardian found that 6.4% of Copilot-enabled repositories leaked at least one secret, 40% higher than the 4.6% baseline across public repos generally. The model was just doing what it was trained to do: complete the pattern.
Real incidents that show what this looks like
Three cases, three different failure modes.
Samsung banned ChatGPT company-wide in May 2023, less than three weeks after allowing it, following three separate incidents in which engineers pasted confidential material into the tool:
- One employee entered faulty source code from a semiconductor database program to ask for a fix.
- Another entered code meant to identify defective equipment, hoping for an optimization suggestion.
- A third converted an internal meeting recording into a document and fed it in to get meeting minutes. All three were normal work habits applied to a tool nobody had drawn a boundary around yet.
DeepSeek’s exposure in January 2025 looked nothing like that. Wiz Research found a publicly accessible ClickHouse database tied to DeepSeek’s infrastructure, reachable through two non-standard ports with no authentication required. Inside was over a million log lines going back to January 6, including plaintext chat history, API keys, and backend operational details. Anyone who found the open ports could run arbitrary SQL queries straight from a browser. This was a database left open to the internet, the same category of mistake that predates LLMs entirely, just with LLM-scale conversation logs sitting inside it.
The GPT-3.5 memorization research sits between those two. Nobody misconfigured anything and no employee pasted anything in. The vulnerability was baked into the model itself, and it took a specific, non-obvious attack technique to surface it.
Three different root causes. Same OWASP category. That’s the part teams tend to miss when they treat sensitive information disclosure as one problem with one fix.
Where this sits in the OWASP top 10
LLM02:2025 Sensitive Information Disclosure is one entry in the broader OWASP Top 10 for LLM applications, and it rarely operates alone. It sits next to LLM01 (prompt injection), which is frequently the delivery mechanism, and LLM07 (system prompt leakage), which is one specific flavor of disclosure.
If you’re building out an LLM security program, treating these as one connected risk surface rather than three separate checklist items is the difference between catching an attack chain and catching one link of it.
How to prevent sensitive information disclosure
No single control closes this off. Layering matters more than any individual technique.
- Redact and sanitize before data reaches the model. Strip PII, credentials, and other regulated data from prompts and RAG source documents before they’re ever processed, not after. This is standard AI data security practice applied to a new pipeline.
- Apply role-based access control to retrieval. If a RAG pipeline can pull from a document store, the model should only retrieve what the requesting user is already authorized to see. Access control belongs in front of the model, not as an afterthought in its output.
- Filter output, not just input. Scan responses for patterns that look like credentials, PII, or other regulated data before they reach the user, the same way a DLP tool would watch outbound traffic.
- Use differential privacy or federated learning during training, where the goal is to prevent memorization from happening in the first place rather than catching it after deployment. Differential privacy adds calibrated noise so no single training example can be reconstructed; federated learning keeps raw data on local devices instead of centralizing it.
- Write policy for humans, not just models. Samsung’s incidents were pure user behavior. A written policy on what can and can’t go into a third-party AI tool, paired with actual DLP monitoring at the browser or network level, would have caught it before ChatGPT ever saw the data.
- Treat system prompts as sensitive by default. Don’t hardcode credentials, internal URLs, or business logic into a system prompt and assume it stays hidden. Assume it will eventually be extracted, and design accordingly.
FAQ
Is sensitive information disclosure the same thing as prompt injection?
No. Prompt injection (OWASP LLM01) is an attack technique, a way of manipulating model input to override its instructions. Sensitive information disclosure (LLM02) is the outcome. Prompt injection is frequently the method used to trigger a disclosure, but disclosure can also happen without any injection at all, as the Samsung and DeepSeek cases show.
Can training data really be extracted from a deployed, production model?
Yes. The 2023 Nasr et al. paper demonstrated a working extraction attack against ChatGPT specifically, not just research models, and recovered training data including PII at a rate 150 times higher than the model’s normal output. Alignment techniques reduced but did not eliminate the behavior.
Does this only affect chatbots like ChatGPT?
No. It affects any system built on an LLM, including code assistants (GitHub Copilot’s secret-leaking behavior), RAG-based internal search tools, and customer-facing agents. Anywhere a model generates text from something it was trained on or retrieved, the same failure mode applies.