Generative AI in cybersecurity has increased the speed and scale of attacks. Writing convincing phishing messages at scale, cloning a voice, building a fake identity that passes a background check — all of it can now be done quickly and cheaply with AI. The technology has also provided attackers with opportunities that did not previously exist, from malware that asks a model what to do next, to phishing campaigns that build trust across several channels over weeks.
Defenders get the same technology but use it to triage alerts, draft investigations, and find and fix misconfigurations and vulnerabilities before attackers exploit them. But the two sides are not in the same position. Attackers can accept a high error rate. Defenders cannot.
This article examines what has actually changed, based on recently documented cases. It covers how AI has altered the economics of attacking an organisation, the new attack vectors generative AI has opened, what happens when systems behave in ways nobody intended, and the risks organisations take on when they deploy AI themselves. It then looks at where AI is already doing useful work for defenders, where the evidence says it is not yet reliable, and what controls, policies, and training actually increase resilience.
Generative AI has impacted what both bad actors and cybersecurity professionals can do. But as AI proliferates, good security practices remain just as essential, if not more so.
|
Attackers gained |
Defenders gained |
Remains essential |
|
Low-cost tools |
Faster vulnerability discovery |
Prioritisation |
|
Ability to scale attacks |
Improved alert filtering |
Accountability |
|
More convincing trust-based attacks |
Verification tools |
Clear policies |
|
Malware that adapts mid-attack |
Faster incident analysis |
Human judgement |
"If you're looking for signs of a productivity boom from AI, you'll find it in cybersecurity," says Natasia Kalajdziovski, Senior Fusion Analyst at SecAlliance, the threat intelligence division of CSIS Security Group. "AI has industrialised hacking, making it far easier and cheaper to carry out. As costs fall, attacks become more frequent. In some ways, it's simple economics."
The SANS Institute's annual practitioner survey, published on 13 July 2026, found that 78% of the 536 cybersecurity and IT practitioners surveyed had experienced confirmed or suspected AI-enabled attacks in the previous year.1 IBM's own analysis puts the figure differently but points the same way: between March 2025 and February 2026, a quarter of all breaches were AI-enabled, up 56% on the previous year.2
The scale is felt not just in the number of attacks but also in the avenues cybercriminals can now take. For example, in phishing campaigns, bad actors can now combine SMS, messaging platforms, and follow-up voice calls to build credibility. The attack unfolds over days or weeks, making it harder for employees to spot.
“Security works on trust. What generative AI has enabled attackers to do is to establish trust, which causes people to lower their defences, before they make their move,” says Simon Jonker, CSIS Security Group’s Senior Director of MDR and IR. “Imagine that you’d been speaking with someone on LinkedIn who has a convincing profile and is active on the platform. You then exchange a series of messages before having a video call. When they later send you an email with a malicious link — one directly related to what you’ve been discussing — you’re far more likely to click it.”
Many of those capabilities are not new. Voice cloning and phishing both predate generative AI. What has changed is that they no longer require skill and can be done very cheaply.
Those economics are visible in what attack tooling now costs. Deepfake images can be made to order for $10 to $50, and ready-made synthetic identities selling for as little as $15. Voice clones are advertised on criminal marketplaces from around $30.3 The tooling for defeating multi-factor authentication is also cheap. One widely used kit, Tycoon 2FA, started at $120 for 10 days' access or $350 per month for the administration panel used to run campaigns. Microsoft attributed around 62% of the phishing attempts it blocked at peak to that single product before a joint takedown with Europol in March 2026, which seized 330 domains. By then, it had enabled access to nearly 100,000 organisations worldwide.4
The impact is felt in the scale of attacks now being implemented. The Financial Times recently reported that phishing attacks rose by 1,265 percent in 2025, an increase that it linked to the easy availability of generative AI tools.5 While we should consider such figures with a degree of scepticism, AI is changing the threat landscape.
Jan Kaastrup, Chief Cybersecurity Advisor at CSIS Security Group, and his team closely monitor the cyber threat landscape: "We're seeing AI being used to not only escalate the volume of cyber-attacks but also elevate their sophistication. Things are changing quickly, especially with the application of generative AI that opens new lines of attack."
Generative AI's ability to produce deepfake audio and video enables attackers to impersonate staff and psychologically manipulate people into performing actions or divulging sensitive information. The results can be highly convincing.
In one recent case — an extreme example of social engineering — members of a North Korean hacking group posed as company executives during Zoom calls, tricking tech employees into downloading malicious code and resulting in financial theft.5
Generative AI is also being used to create synthetic identities: entirely fake people with fictitious backstories. The intent is usually to gain access to an organisation for espionage or fraud. Once established, a synthetic identity can be used to apply for remote jobs that allow attackers to gain direct access to a company's IT systems.
A well-documented case involved a security company. In July 2024, KnowBe4, which sells security awareness training, disclosed that it had hired a software engineering candidate who turned out to be a North Korean operative. The candidate passed four video interviews and cleared a background check. The check came back clean because the stolen identity belonged to a real US citizen with a genuine credit and employment history, and the interviews passed because AI was used to alter the candidate's photo from a stock image.6
Importantly, the controls functioned correctly. The checks confirmed that the identity was real. But the controls had no way to confirm it belonged to the person being interviewed. Generative AI had changed the game.
Catching these at the hiring stage takes fairly ordinary measures: identity verification that goes beyond a name and a document check, attention to shipping addresses that don't match where the candidate says they are, and coordination between HR and security, since neither team sees the whole picture alone.
"Defending against these attacks requires continually updated threat intelligence to spot new attack vectors as they emerge," says Jan Kaastrup. "But we can't just play catch-up. We also have to anticipate how bad actors might use AI technologies and close the gaps before they can be exploited."
AI is being used to create more sophisticated attacks that build trust and enable more convincing phishing attempts
Threats come not only from bad actors but also from AI models that behave in unexpected ways. The risks of running AI systems without adequate oversight came to public attention in July 2026, when a group of OpenAI's experimental models escaped their test environment.11
Agents that were supposed to be isolated from each other found a way to communicate and set up an improvised message board to coordinate. Around 700 agents carried out an intrusion into Hugging Face, the open-source platform that much of the AI industry depends on, over four and a half days.12 Hugging Face's own forensic timeline recorded roughly 17,600 attacker actions.13 None of it was malicious. It was reward hacking — the agents had found a way to cheat the test they were set, and the attack on Hugging Face was an attempt to steal the answers. OpenAI called it a warning shot.
Most organisations are not experimenting with frontier models, but the Hugging Face example provides an important lesson. AI systems can produce unexpected results, so close monitoring is required.
Cases of AI systems lying, ignoring instructions and pursuing their own goals continue to rise.
The Loss of Control Observatory, run by the Centre for Long-Term Resilience with funding from the UK's AI Security Institute, has tracked such incidents since November 2025. Reported behaviours include AI systems posing as human supervisors and mimicking a specific person's writing style to obtain the consent they would otherwise need from a human before acting.
The Observatory recorded more than 300 cases in July 2026, roughly double the previous month, bringing the year's total to 1,664. Higher-severity incidents are up more than sevenfold since monitoring began. Its figures come from publicly reported incidents on X, so the real number is likely much higher.14
“The Hugging Face example provides an important lesson. AI systems can produce unexpected results, so close monitoring is required.”
The security impacts of AI are not limited to attacks on people. General enthusiasm for AI's productive potential and repeated recommendations to experiment with the technology at work also introduce risks for the organisations deploying it.
Three stand out. Prompt injection occurs when a model is fed a hidden instruction within ordinary content, which it reads as a command. Sensitive information disclosure happens when data reaches tools with no agreement covering it, or when a system retrieves material the person asking should not see. Hidden context exposure occurs when the instructions an assistant runs on turn out not to be private.
In a late 2025 survey of 445 IT and security professionals run by the Cloud Security Alliance, 47% said they had experienced a security incident involving an AI agent in the previous year. Only 16% were highly confident they could detect AI-agent-specific threats, while 44% had little or no confidence.15
In the face of increasing scale and sophistication, many are predicting an AI arms race between attackers and defenders. What's actually happening is more nuanced, but AI solutions are already being applied at scale.
One approach, cyber reasoning systems, combines one or more large language models with conventional security tooling such as fuzzers, static analysis and test harnesses. These systems can be effective.
At a US government competition held by DARPA in August 2025, seven cyber reasoning systems worked unsupervised through 54 million lines of code from software used in hospitals and utilities. They found 54 of 63 deliberately planted flaws and patched 43. They also surfaced 18 genuine flaws that nobody knew existed.16
In another well-publicised case in early 2026, Claude Opus 4.6 found 22 vulnerabilities in Mozilla's Firefox web browser in two weeks. Mozilla rated 14 of these as high severity, almost a fifth of all the high-severity flaws the company fixed in the whole of 2025.17 In April 2026, Claude Mythos Preview identified a further 271 vulnerabilities in Firefox. Mozilla's own assessment is worth noting. Claude had not seen anything a top human researcher could not have found. What changed was speed and volume.18
The results are impressive, but AI remains fallible. Autonomous systems still fix fewer than they find: at DARPA's competition, the systems found 86% of the planted flaws but patched only 68%.
AI can support in many stages of security operations, taking on routine work in detection, triage and investigation to enable analysts to focus on higher-value work.
"Vulnerability discovery gets the attention," says Simon Jonker, "but it's the start of a process, not the end of it. All those vulnerabilities need to be fixed."
The daily work of a security operations centre (SOC) is collecting data, detecting threats in it, triaging the alerts that fire, investigating the ones that matter and responding to what turns out to be real. AI tools — one category of which is often referred to as AI SOC agents — are now being applied at every one of those stages. In practice, that means log triage, alert summarisation, vulnerability prioritisation and first drafts of incident write-ups.
"AI works well for many tasks: filtering false positives, presenting the key information up front for initial triage, and tuning alert rules to fit each customer's needs or to catch the latest attacker techniques identified by threat intelligence," says Simon Jonker. "Analyst attention is finite, so using AI for this work creates time to focus on priority work."
Security practitioners' confidence in AI appears to be falling as adoption rises. The 2026 SANS Institute survey found that the share of respondents reporting significant shortcomings in AI's threat detection and response rose from 45% to 63% in a year.1
Simon Jonker is unsurprised by those results. "AI has its place in cyber defence, but it's part of a solution, not all of it. It's misguided to believe that handing everything over to AI solutions is the answer. Organisations can get a false sense of security that may prove costly. We must deal in evidence and not operate with blind faith. Use it, explore it, get the most out of it – but don't rely on it alone."
"AI has its place in cyber defence, but it's part of a solution, not all of it.”
Systems are also becoming available to counter the attack vectors introduced by generative AI.
One solution, liveness detection, aims to distinguish between real people and AI-generated clones. These systems operate in passive mode, analysing texture, depth and micro-movement in what the camera captures, and active mode, which issues a challenge such as "blink", "turn your head" or "smile".
They are not infallible. In December 2025, MITRE published a case in which a red team passed a mobile banking identity check under a false identity. They generated a real-time deepfake and swapped in the phone's camera feed, so the check saw the fake instead of the real camera feed. Both passive and active checks were evaded.19
In that case, the problem was that liveness detection only inspected the image it was given. It had no way to confirm that the image came from the camera, so an attacker who controls the feed never had to fool it. A European technical specification, CEN/TS 18099, sets out how to test identity verification systems against this specific attack.20
So, what to do? One answer is to stop relying on the camera image alone. Contextual signals such as typing cadence, how a device is held, or where a session is coming from can be added as a secondary layer alongside biometric checks.
"The defensive technology will improve, but so will attack sophistication," says Jan Kaastrup. "You have to understand the limitations of current protections and also expect that they will be circumvented. Because of AI, organisations need to increase resilience and strengthen defences to ensure that systems stay operational while under attack."
"You have to understand the limitations of current protections and also expect that they will be circumvented."
Security professionals do not operate under the same rules as bad actors, which has implications for how AI can be applied in cybersecurity.
Firstly, attackers can afford to be wrong. A failed phishing campaign costs them nothing. They can let a model run unchecked and accept whatever error rate it brings. Bad actors need one success. Defenders need an unbroken run of them. A missed detection is a breach.
Secondly, defenders have rules to follow. European security teams already comply with GDPR and NIS2, as implemented in their own countries. Since 2 August 2026, they must also comply with the EU AI Act's transparency rules,21 and the stricter obligations for high-risk systems take effect on 2 December 2027.22 Between them, these rules shape what data AI is allowed to access, what must be written down, and what must be disclosed to customers and regulators. They also govern where data can be processed, which is why data residency matters when a vendor runs AI on your behalf.
None of this argues for keeping AI out of security work. AI solutions already provide value by enabling analysts to focus on high-priority work. The point is to use AI intentionally with a clear understanding of the technology’s limitations. That requires real insight into the technology and well-considered procedures that make the most of what it offers.
For organisations seeking to safeguard their own systems, a good first step is often an AI or LLM assessment to uncover how threat actors can turn your AI against you or help prevent careless use by insiders and authorised users. More general protections include:
Strong controls and clear policies should also apply to vendors. CSIS Security Group’s AI Promise exists for precisely that reason: to clarify how AI may be used in the company’s cybersecurity and threat intelligence work on behalf of clients.
“A good first step is often an AI or LLM assessment to uncover how threat actors can turn your AI against you.”
1. Confirm unusual requests before acting
A convincing voice or video is now cheap to produce, and humans are poor at spotting the difference. In one study, only 0.1% of participants correctly identified every real and fake item they were shown.23 Simply relying on recognising someone is no longer enough. Unusual requests should be confirmed through a separate channel the caller didn't choose, such as calling back a number already on file or messaging on a different system.
2. Use logins that can't be phished
Phishing works in two steps: a message persuades someone to click, and a fake login page collects their password. AI has made the first step harder to defend — producing fluent, convincing emails in any language. Passkeys and hardware keys break the second step instead. They check the web address before doing anything, so on a fake page, they don't work. There is no password to type in, which means there is nothing an attacker can persuade someone to hand over.
3. Give AI systems the least access possible
An AI assistant inherits whatever permissions it was set up with, and broad access is quicker to grant than narrow. In a Cloud Security Alliance survey of 445 professionals, 53% reported that AI systems exceeded their intended permissions.15 Reduce permissions by matching credentials to specific tasks rather than to the team, and use read-only access wherever possible. Then schedule reviews of what each system can access, as permissions may accumulate over time.
4. Require human approval
Attackers can hide instructions inside the content that an AI assistant processes, such as an email, a document, a web page, or a support ticket. The assistant may read it as another instruction and follow it. You can't reliably filter this out, so control what happens next instead: require human approval before an AI system sends anything externally, moves money, deletes data, or changes permissions. Log every action it takes, so you can reconstruct what happened afterwards.
5. Shorten the gap between detection and response
Software flaws have overtaken stolen credentials as the most common way breaches begin.24 At the same time, organisations are patching more slowly. Verizon's 2026 breach report puts the typical time at 43 days, up from 32. Overcoming these challenges requires more efficient working practices and excellent prioritisation, such as deciding in advance who authorises emergency patching out of hours and ensuring critical findings quickly reach that person.
Cost, speed and scale. Attack tooling that once required skill is now rented by the day. Campaigns that used to be a single email now run across several channels over weeks.
In two main ways. The first is against people: convincing phishing messages in any language, cloned voices and faces, synthetic identities that pass background checks, and campaigns that establish trust across several channels before making a request. The second is against systems: finding flaws in an organisation's software and defences. In May 2026, Google's threat intelligence team reported the first case it had seen of criminals weaponising a previously unknown flaw that it believes AI helped them find and exploit.7
Yes. Generative AI has made phishing content more convincing and opened up more channels for use. Together, those let an attacker establish trust before making a request — a profile that looks real, messages exchanged over weeks, a video call, and only then a link related to what was discussed. Training people to watch for bad grammar or an unexpected email does not address that.
Researchers at Group-IB found deepfake images made to order for $10 to $50 and ready-made synthetic identities for as little as $15. Voice clones are advertised on criminal marketplaces for around $30, and kits for defeating multi-factor authentication are rented by the day. That is the commodity end of the market, which is what has changed the economics of volume attacks. State-sponsored operations run on a different budget.3
Conventional malware follows a fixed script, which is what makes it detectable. Newer strains pause during an attack, query an AI model about what to do next, and adapt to what they find. One Android backdoor documented in 2026 feeds the device's screen contents to a model and acts on the response.8
AI is now applied across the whole pipeline: data collection, detection, triage, investigation and response. The tools that do this across a team's existing security stack are often referred to as AI SOC agents. But reliability varies by task. The clearest gains are at the low-value end of the alert queue — false positives, low-severity noise, mistuned rules — as well as in log triage, alert summarisation, and first drafts of incident write-ups. In those use cases, AI clears analyst time rather than replacing analyst judgement, so the important decisions stay with people. Automated response means a system acting on a threat rather than flagging it for a person. That is where teams are most reluctant to hand over control, and the evidence supports that caution.
So far, the evidence points to volume and speed rather than a new class of flaw. When Mozilla used AI models to scan Firefox in 2026, it found far more vulnerabilities than its usual process would have surfaced in the same period. Still, it noted that nothing it saw was beyond what a skilled human researcher could see. That is one codebase and one assessment, and the models are improving quickly, so it is not a settled question.17 18
Detection is getting harder, so don't rely on it alone. Confirm unusual requests through a separate channel the caller did not choose, such as a callback to a number already on file. Agree on passphrases for executives, finance teams and their families, and require human verification for financial transfers. Liveness detection tools exist and are improving, but they have documented weaknesses — particularly against attacks that replace the camera feed rather than fooling the camera.19
To disguise themselves and support a fake backstory. In one documented case, KnowBe4, which sells security awareness training, disclosed that it had hired a North Korean operative after four video interviews and a clean background check. The candidate's photo had been altered by AI using a stock image, and the identity used was that of a real US citizen, complete with credit and employment history. What the process could not do was connect that identity to the person on the calls.6
That framing oversimplifies it. Both sides use similar technology, but they are not in equivalent positions. Attackers can run a model unchecked and accept whatever error rate results, because a failed attempt costs them nothing. Defenders answer to regulators and to clients, and a single missed detection is a breach.
They shape it. European teams respond to GDPR and NIS2 as implemented locally, and, since 2 August 2026, to the EU AI Act's transparency rules, with heavier obligations for high-risk systems taking effect on 2 December 2027. Together, these determine what data AI is allowed to access, what must be documented, where processing can occur, and what must be disclosed. Attackers observe none of it.21 22
Sign up for our newsletter and get the latest cybersecurity analysis, reports and events from CSIS delivered straight to your inbox.
CSIS Security Group A/S is a leading European pure-play provider of tech-enabled cybersecurity and intelligence services. Operating 24/7, we deliver Managed Detection & Response, Incident Response, Security Consulting across all sectors, and provide a world-class threat intelligence capability through our SecAlliance brand. Accredited by organisations including CREST, we actively support global security initiatives to positively impact the cyber community.