Generative AI has split cybersecurity into two camps: people trying to use it to defend systems faster than before, and people using it to attack those same systems in ways that didn’t exist three years ago.
Both camps are winning. That’s what makes this moment different.
For most of cybersecurity’s history, attackers had to be skilled to be dangerous. Writing convincing phishing emails took time. Building custom malware required real coding knowledge. Deepfakes needed significant compute. Generative AI changed all of that. It lowered the floor for attackers while raising the ceiling for defenders, and the net effect depends almost entirely on which side moves faster.
What makes generative AI different from other AI
Generative AI creates original content by learning patterns from large datasets, then producing new output on demand. Unlike traditional AI models that classify or predict from existing data, generative models create: text, code, images, audio, video. That distinction matters for security because it turns AI from a detection tool into a creation tool, available to anyone with a browser.
Traditional AI excels at finding patterns in known data. A spam filter learns what spam looks like and flags it. A fraud detection model learns what fraudulent transactions look like and blocks them. Generative AI goes further: it can write the phishing email, generate the malware, fabricate an executive’s voice, and craft a social engineering script, automatically and at scale, with no specialized knowledge required from the person running it.
How generative AI is changing the attack landscape
Phishing at scale with large language models
Phishing used to scale badly. Writing a convincing email takes time. Personalizing it to a specific target takes more. Getting the corporate tone right, avoiding grammar mistakes that trigger spam filters, choosing the right pretext. These were real constraints that limited both the volume and quality of phishing campaigns.
Large language models removed all of them at once.
An attacker can now feed an LLM a target’s LinkedIn profile, company blog, and a few public emails, then get back a phishing message that mirrors the organization’s internal communication style precisely. At scale. In seconds. According to SoSafe’s 2024 research, 87% of organizations encountered AI-driven cyberattacks that year, and phishing is the most common entry point.
What’s harder to communicate is how much better these emails are now. Early AI-generated phishing had tells. Research from Harvard Business Review shows AI-generated phishing fools recipients at rates equivalent to human-written messages. The effort cost dropped; the success rate didn’t.
AI-generated malware and polymorphic code
Code-generation models do the same thing for malware that large language models do for phishing. Attackers who couldn’t write malware before can now describe what they want in plain language and get functional code back. Attackers who could already write it use AI to generate polymorphic variants: code that rewrites itself each time it executes, making signature-based antivirus nearly useless against it.
A CSET evaluation found that nearly half of code snippets generated by major AI models, including GPT-4, contained at least one security-relevant flaw. Attackers exploit this in two directions: generating intentionally malicious code, and injecting vulnerable AI-generated code into targets’ own software pipelines through supply chain attacks.
Deepfake attacks and CEO fraud
Business email compromise has always relied on impersonation. Generative AI made impersonation good enough to fool people who should know better.
In early 2024, a finance employee at the multinational engineering firm Arup transferred $25 million after a video call with what appeared to be the company’s CFO and other senior leaders. Everyone on that call was a deepfake. The video looked real. The voices matched. The instructions seemed credible. The employee did what was asked.
Synthetic video, synthetic voice, synthetic authority. This type of attack is no longer exclusive to nation-state actors. Mid-tier threat actors can now run it with off-the-shelf tools. FTSE 100 companies have deployed AI systems specifically to detect synthetic voice in executive calls, which tells you something about how common this has become.
Prompt injection and data leakage
When an organization embeds an AI assistant into its workflows, that assistant becomes a new attack surface.
Prompt injection attacks work by embedding malicious instructions into content the AI processes. A customer service chatbot reads an attacker-controlled document. That document contains hidden instructions. The chatbot follows them, leaking internal data or taking actions outside its intended scope. No one notices because the chatbot appears to be behaving normally.
Indirect prompt injection is particularly hard to defend against because the malicious input doesn’t come from the user. It comes from external content the AI system is trusted to read: email summaries, document analyzers, browsing agents, code review tools. Any AI that reads untrusted external input and acts on its contents is a potential injection target, and most organizations don’t have input validation in place for these systems yet.
Shadow AI as an internal risk
Shadow AI is what happens when employees use AI tools that IT and security teams haven’t approved.
Someone finds a free writing tool and pastes internal documents into it. A developer uses an AI coding assistant that sends code snippets to a third-party server. A sales team member runs customer data through an AI summarizer before a call. None of these people intend to cause a breach. Most have no idea they’re doing anything risky.
But unmanaged AI tool use adds an average of $670,000 to breach costs, according to IBM’s 2025 Cost of a Data Breach Report. The risk isn’t only data leaving the organization. It’s data entering AI systems with unknown retention policies, unclear security controls, and no audit trail. You can’t secure what you can’t see, and shadow AI is an invisible attack surface expanding inside most organizations right now.
Data poisoning and adversarial attacks
Data poisoning is less visible than phishing or malware, and potentially more damaging over time.
An attacker corrupts the training data for an AI model before or during training. The model learns from that corrupted data. Its behavior changes in ways the attacker intended: misclassifying certain inputs, ignoring specific patterns, producing outputs that look legitimate but aren’t. Research published on arXiv (2025) shows data poisoning can reduce model classification accuracy by up to 27% in image recognition tasks and 22% in fraud detection models. For either application, those gaps represent holes that malicious inputs pass through undetected.
Adversarial attacks work at inference time rather than training time. They craft inputs specifically designed to fool a deployed model: images with subtle pixel-level alterations that cause computer vision systems to misidentify objects, text with deliberate patterns that bypass content moderation filters. These attacks don’t break a model. They exploit how it generalizes from its training data to new inputs.
How generative AI is strengthening cyber defenses
The same capabilities that help attackers also help defenders. The key difference is that defenders can deploy AI systematically across an entire organization, while most attackers are still using it opportunistically, one campaign at a time.
Faster threat detection and anomaly analysis
Security teams generate enormous volumes of log data, network telemetry, and behavioral signals that no human team can review comprehensively. AI-powered detection systems establish behavioral baselines for users, devices, and applications, then flag deviations that might indicate compromise.
The operational impact is measurable. Organizations using extensive AI in security operations identify and contain breaches nearly 100 days faster than those that don’t, according to IBM’s 2024 Cost of a Data Breach Report. One hundred days of attacker access time removed from an incident. At the scale of a typical breach, that’s the difference between a contained incident and a catastrophic data loss.
Automated incident response
Alert fatigue is a real and documented problem. Security operations center analysts receive hundreds of alerts per shift. Triaging them manually, generating investigation playbooks, coordinating across tools: all of this consumes most of the working day before any actual investigation starts.
Generative AI handles the mechanical parts: summarizing alert context, correlating events across systems, drafting investigation plans, automating routine remediation steps. Analysts focus on judgment calls instead of documentation. Organizations implementing AI-assisted response workflows report significant reductions in manual alert triage time and false positive rates, with leading security vendors including Splunk and Darktrace publishing figures in the range of 50% fewer manual triage hours and 70% fewer false positive alerts.
AI-powered phishing simulation and training
The same LLMs that attackers use to write convincing phishing emails can be used by security teams to test whether employees would fall for them.
Generic phishing simulations are increasingly easy to spot. They use obvious templates, outdated pretexts, and formats that security-aware employees recognize from previous rounds. AI-generated simulations can mirror actual techniques active in the wild: the same personalization, the same corporate language mimicry, the same timing. Training that reflects real current threats builds real resilience. Training that reflects last year’s generic templates builds false confidence.
Identifying vulnerabilities in AI-generated code
As AI coding assistants become standard in software development, the code they produce needs to be treated as untrusted input until validated. Security teams are now running automated security scanning against AI-generated code before it reaches production.
The failure mode here is automation bias. Developers trust AI-generated output at higher rates than handwritten code, according to the CSET evaluation, which means security flaws that do exist are less likely to be caught in review. Catching these vulnerabilities requires deliberately scanning AI-generated code with the same rigor applied to any external dependency.
Can generative AI make cyberattacks worse?
Yes, in four specific ways.
- First, generative AI lowers the skill floor. Attacks that previously required real technical expertise, including writing functional malware, creating believable synthetic media, and crafting personalized social engineering, are now accessible to actors with no technical background.
- Second, it increases volume. When generating a phishing campaign takes minutes instead of days, attackers can run more of them, target more organizations, and adapt faster when one approach stops working.
- Third, it improves quality on the dimensions that matter for attack success. AI-generated phishing reaches human-level persuasiveness. Deepfakes are getting harder to detect as generation models improve. Polymorphic malware evades signature detection more reliably.
- Fourth, generative AI creates entirely new attack categories. Prompt injection, data poisoning, adversarial inputs, and model theft didn’t exist as practical threats before AI systems became part of enterprise infrastructure. These aren’t variations on old attacks. They’re structurally new.
The counterweight is that AI helps defenders catch attacks faster and at greater scale. But the current situation is that attackers adopted these tools quickly and many defenders are still catching up on what that means for their security posture.
How are companies responding to AI-based threats?
The organizations taking this seriously are working on four fronts at once.
AI usage policies establish what employees can do with AI tools, which tools are approved, and how sensitive data must be handled before interacting with any AI system. Without a clear policy, every employee makes their own judgment about what’s acceptable, and those judgments vary widely.
AI red teaming (running adversarial tests against AI systems before and after deployment) is becoming standard practice rather than an optional exercise. The White House Executive Order on AI mandated red teaming for powerful AI models before public release, and enterprises are building internal programs that apply the same logic to their own AI deployments.
Regulatory compliance is forcing security decisions at organizations that might otherwise deprioritize them. The NIST AI Risk Management Framework provides voluntary guidance for managing AI risks across the system lifecycle. The EU AI Act creates legal requirements for high-risk AI applications in Europe. Organizations operating across jurisdictions face genuinely fragmented requirements, with different frameworks defining risk, documentation, and oversight differently.
Updated employee training covers prompt injection risks, how to recognize synthetic media, and why using unsanctioned AI tools creates organizational exposure. Security awareness training that doesn’t address AI threats is missing a significant portion of the current threat surface.
What are the benefits of using AI for cybersecurity?
The business case is quantified clearly in IBM’s 2024 Cost of a Data Breach Report: organizations using extensive AI in security operations experience 45.6% lower breach costs on average, saving roughly $2.22 million per incident.
That number reflects faster detection, faster containment, and less time attackers spend moving through systems undetected. It also reflects lower investigation costs, because AI-generated event correlation and alert summaries replace hours of manual log review.
The AI cybersecurity market reflects this value. It’s projected to grow from $22.4 billion in 2023 to $60.6 billion by 2028, according to MarketsandMarkets, driven by organizations that have done the math and decided AI security tooling pays for itself.
What AI doesn’t replace is security judgment. The 100-days-faster detection finding comes from organizations using AI to augment human analysts, not substitute for them. Decisions about what a breach means, how to communicate it, and what it reveals about organizational risk remain human responsibilities. AI handles the volume problem. Humans handle the complexity.
To get a clearer picture of how AI security tools stack up in practice, the comparison of top AI security solutions breaks down the leading platforms by capability and use case.
Frequently asked questions
Can generative AI make cyberattacks worse?
Yes. It lowers the skill requirements for executing sophisticated attacks, increases the volume attackers can run in a given period, improves the quality of phishing emails and synthetic media, and creates entirely new attack categories like prompt injection and data poisoning that didn’t exist before AI systems became part of enterprise infrastructure.
What is a prompt injection attack?
A prompt injection attack embeds malicious instructions into content that an AI system processes. The AI follows those instructions rather than its intended purpose, potentially leaking data or taking unauthorized actions. Indirect prompt injections are particularly difficult to defend against because the malicious input comes from external content the AI is trusted to read and act on.
What is shadow AI?
Shadow AI refers to AI tools that employees use without IT or security approval. The risk is sensitive organizational data entering AI systems with unknown data retention policies and no visibility for security teams. IBM’s research shows unmanaged AI tool usage adds an average of $670,000 to breach costs. Organizations need AI governance policies that cover which tools employees can use and how data flows through them.
How do companies protect against AI-generated phishing?
The most effective approaches combine AI-powered email filtering, which catches patterns that rule-based filters miss, with employee training using AI-generated simulations that reflect real current attack techniques, and clear internal reporting procedures so suspicious messages are flagged quickly rather than acted on.
What security frameworks address generative AI risks?
The NIST AI Risk Management Framework provides voluntary guidance for managing AI risks across the full system lifecycle. The EU AI Act creates regulatory requirements for high-risk AI applications in Europe. OWASP’s Top 10 for LLM Applications covers the most critical security risks specific to large language model deployments.