Cybersecurity in the Age of Powerful AI Systems – A Challenge for the Qualification of Professionals

1. A Turning Point for IT Security

Cybersecurity has always been a race between attack and defense. But with the rise of powerful, generative, and increasingly "agentic" AI systems, the speed, reach, and quality of this race have changed fundamentally. What was discussed just a few years ago as a hypothetical future scenario - autonomous AI agents that independently plan and execute attacks - is today a documented reality. In November 2025, the AI company Anthropic published a report on a presumably Chinese, state-sponsored group (internally designated GTG-1002) that misused the AI tool Claude Code to attack roughly 30 organizations worldwide. According to Anthropic's assessment, 80 to 90 percent of the operational attack steps - reconnaissance, vulnerability discovery, exploit creation, backdoor installation, and data exfiltration - were carried out by the AI itself without substantial human intervention; human operators were involved only at a few strategic decision points (Anthropic). Independent security researchers have critically questioned the exact extent of this autonomy (Ars Technica), but the case marks a perceptual turning point: for the first time it was publicly documented that AI agents can be embedded not merely as advisors but as operationally acting participants in a complex, multi-stage cyberattack.

This paper describes the structural challenges this development poses for cybersecurity practice - on the side of attackers, on the side of defenders, in regulation, and finally in the training of the professionals who must manage this transformation. The appendix illustrates, by example, how one university curriculum - the module "AI & Emerging Topics in Cybersecurity" - responds to exactly these challenges.

Crucially, this is not a single, clearly delimited threat but a simultaneous shift across several levels: attack tools are becoming more powerful and cheaper to access, AI systems themselves are becoming new targets, defenders are gaining new tools whose limits still need to be learned, the legal framework is catching up in stages, and the labor market cannot yet cover the resulting need for qualified professionals. The following sections address these levels in turn before the appendix shows how a concrete learning program can be derived from them.

2. The New Attack Dynamic: How AI Is Transforming the Offense

2.1 From Assistant to Operator

The World Economic Forum's "Global Cybersecurity Outlook 2026" captures the urgency of the situation: 94 percent of surveyed executives expect AI to be the most significant driver of change in cybersecurity in 2026, and 87 percent report that AI-related vulnerabilities were the fastest-growing cyber risk in 2025 (WEF; Kiteworks). Check Point's "AI Security Report 2026" describes this shift even more pointedly: AI has moved "from development aid to operational actor" - it now performs the actual legwork within ongoing attacks, from Chinese-attributed espionage campaigns to criminal intrusions into Mexican government agencies. According to the report, a single attacker used AI assistance to generate VoidLink, an 88,000-line command-and-control framework, within one week (Check Point Research). Anthropic's own analysis of 832 accounts suspended for abusive use likewise shows that attackers now employ AI across nearly all 14 tactic categories of the MITRE ATT&CK framework, and the share of actors classified as "medium to high risk" rose from 33 to 56 percent within a year (Anthropic). CrowdStrike's "2026 Global Threat Report" confirms this trend, reporting an 89 percent increase in attacks by AI-assisted attackers and a 134 percent increase in AI-generated script execution from 2024 to 2025 (CrowdStrike).

2.2 Phishing and Social Engineering at Real-Time Quality

The leap in quality is most visible in the oldest of all attack vectors: social engineering. According to the "AI Threat Landscape Report 2026" from security firm Cyble, AI-generated phishing attacks rose 14-fold in December 2025; their share of all registered attacks jumped from 4 to 56 percent. AI-generated phishing emails now achieve click rates of 54 percent compared to just 12 percent for manually written messages, and 82.6 percent of all phishing emails now contain AI-generated elements (Cyble). The previously reliable detection markers - clumsy grammar, impersonal salutations, generic subject lines - are thus disappearing as a line of defense; a Kiteworks analysis found that 50 percent of security leaders name hyper-personalized phishing as their greatest AI-related concern (Kiteworks).

2.3 Deepfakes: When Trust Becomes a Weapon

The development in synthetic media is particularly severe. The best-known case remains the fraud at British engineering firm Arup, where an employee, after a video call featuring deepfake-generated images of the CFO and several colleagues, transferred roughly US$25 million (StationX). This individual case is no longer an outlier but a symptom of a systemic trend: an analysis based on the AI Incident Database and Resemble.AI puts globally documented losses from deepfake fraud at at least US$3.7 billion, with about 89 percent of that recorded only in 2025 and the first half of 2026 (Surfshark). The FBI, in its 2025 "Internet Crime Report," listed a dedicated AI-fraud category for the first time: 22,364 complaints with adjusted losses of US$893.3 million in the US alone (Digital Applied). A Gartner survey of 302 security leaders found that 62 percent of organizations have already experienced at least one deepfake attack (Netarx), and, according to Entrust, one in five biometric fraud attempts now stems from deepfakes. For businesses, this means identity verification via video or voice call - long considered safer than email fraud - is no longer a reliable control.

2.4 AI in the Malware Lifecycle

AI is also transforming malware development itself. The CrowdStrike report records that 82 percent of attacks in 2025 were "malware-free," with perpetrators gaining access via legitimate identities and tools rather than classic malicious software (Conversational Geek) - a trend that increasingly renders classic, signature-based detection ineffective. At the same time, ESET researchers identified "PromptSpy," the first known Android malware to use generative AI directly within its execution flow, along with tens of thousands of suspicious and thousands of clearly malicious "AI skills" within AI-agent ecosystems (ESET via Yahoo Finance). Check Point also documents a rarer but growing pattern: malware that communicates live with a language model during execution to generate new commands or rewrite itself (Check Point Research). IBM's "Cost of a Data Breach Report 2026" summarizes the economic dimension: one in four malicious security incidents in 2025/2026 was AI-enabled - a 56 percent increase over the previous year - and such incidents caused roughly US$1 million more damage than the global average, at an average of US$6 million (IBM Newsroom).

2.5 Systemic Amplification: Geopolitics and Critical Infrastructure

These individual phenomena reinforce one another in a geopolitically tense environment. The WEF report now identifies the geopolitical situation as the top concern for 64 percent of surveyed organizations and explicitly names new risk areas that overlap with AI capabilities: autonomous systems, robotics, critical infrastructure, and advances in quantum technology (Gruppo TIM). This combination is particularly volatile because it merges two previously separate risk dimensions: the classic question of the attack surface of critical supply systems, and the new question of how autonomous, AI-driven systems remain under control in these environments. The report also warns of growing "cyber inequality" between well-resourced and structurally disadvantaged organizations and states - a finding that further underscores the responsibility of educational institutions to teach AI security competence beyond already-privileged environments.

3. AI Itself as an Attack Surface

The second dimension of the challenge lies in the fact that AI systems are not merely attack tools but attack targets themselves. Large language models and the agent architectures built on them exhibit their own specific vulnerability classes, systematically captured by the OWASP Top 10 for LLM Applications: prompt injection (direct and indirect), data poisoning and backdoor attacks on training data, insecure output handling, excessive agentic autonomy, and weaknesses in the model and training-data supply chain. IBM's "X-Force Threat Intelligence Index 2026" study records a 44 percent increase in publicly exploitable application vulnerabilities, driven significantly by AI-assisted vulnerability discovery on the attacker side (IBM Newsroom); at the same time, according to IBM, more than 20 percent of organizations report a security incident that specifically targeted AI models or applications - mostly caused by compromised APIs, plug-ins, or misconfigured cloud environments in which the AI runs.

A particularly underestimated risk concerns the model and data supply chain itself: many model weights are distributed in Python's own "pickle" format, whose deserialization can execute arbitrary code - turning a downloaded model into a potential gateway for code execution. As a countermeasure, the concept of an "AI Bill of Materials" (AI-BOM) is emerging, documenting a model's provenance, training data, and dependencies in the same way the software industry has already introduced for classic software supply chains with the Software BOM. Without such documentation, security teams often remain unclear on which training data, pre-training checkpoints, or third-party components are embedded in a production model - and therefore which of the above vulnerability classes are actually relevant in a given case.

This technical vulnerability meets a pronounced governance gap. While the share of organizations that systematically assess the security of the AI tools they deploy has nearly doubled, according to WEF, from 37 percent (2025) to 64 percent (2026), this also means that more than a third of all organizations still deploy AI tools without any prior security review (WEF). Particularly notable is a perception gap between leadership levels: while executive leadership now ranks AI vulnerabilities as their second-largest cyber risk, IT security officers (CISOs) often do not even list this same risk among their own top three priorities (Fortinet) - an indication that strategic risk perception has not yet caught up with operational reality.

4. The Dual Role of AI: Opportunity and Limits for Defense

AI is not a one-way street. According to WEF, 52 percent of organizations already use AI for phishing detection, 46 percent for attack and anomaly detection, and 40 percent for user-behavior analysis (WEF "Empowering Defenders"). AI-powered Security Operations Center copilots can reduce alert fatigue, detect patterns across massive datasets, and relieve analysts of repetitive triage tasks. Gartner projects that by 2028, 15 percent of all daily work decisions will be made autonomously by AI agents - for cybersecurity this means specialized agents collaborating across threat analysis, vulnerability management, and incident response (ibid.).

However, this opportunity is bound by three structural limits. First, large language models tend to hallucinate - in the Anthropic case described at the outset, this interestingly even hampered the attackers themselves, as the AI exaggerated or fabricated results (Anthropic); in a defensive context, however, hallucinations lead to false alarms and misplaced trust. Second, there is the risk of "automation bias": when analysts uncritically adopt AI assessments, human oversight declines precisely at the moment it is most needed. Third - and strategically the most serious - there is a fundamental cost asymmetry: attackers can convert additional compute, data, and testing time linearly into immediately scalable impact, while defenders still largely must scale via additional personnel and fragmented tool landscapes (PwC). This asymmetry is the true core of the challenge: it is not just about mastering new tools, but about closing a structural speed gap.

There is also a cultural limit: the Kiteworks study shows that 77 percent of organizations already use generative AI in their security stack, but only 37 percent have a formal AI policy governing usage, approval processes, and accountability (Kiteworks). AI adoption in defense therefore often emerges bottom-up and practice-driven, ahead of governance structures - a pattern that fosters loss of control and inconsistent quality standards. 67 percent of surveyed organizations already deploy agentic AI - systems that not only make recommendations but independently trigger actions (ibid.) - a step that elevates the automation-bias problem described above from the recommendation level to the execution level, further raising the requirements for oversight and accountability.

5. Regulatory Responses: A Framework Under Construction

Legislators are responding - with delay, but increasing binding force. The EU AI Act, in Article 15, obligates providers of high-risk AI systems to ensure "an appropriate level of accuracy, robustness, and cybersecurity" throughout the system's entire lifecycle (artificialintelligenceact.eu). The regulation's timeline was recalibrated by the "AI Omnibus" package passed in July 2026: the transparency obligations under Article 50, as well as oversight of providers of general-purpose AI models, took effect as originally planned on August 2, 2026, while the stricter obligations for standalone high-risk systems under Annex III - for example in education, employment, or critical infrastructure - were postponed by 16 months to December 2, 2027 (Cloud Security Alliance; Software Improvement Group). Violations can be punished with fines of up to €35 million or 7 percent of global group revenue (ADVISORI).

Alongside the EU AI Act, an ecosystem of complementary frameworks is taking shape: the NIST Cybersecurity Framework 2.0 with its six functions (Govern, Identify, Protect, Detect, Respond, Recover) as an overarching governance structure, the NIST AI Risk Management Framework for the specific risk assessment of AI systems, ISO/IEC 42001 as the first international management-system standard for AI, and - specifically for the security sector - NIST IR 8596 ("Cyber AI Profile"). For cybersecurity professionals, this means regulatory knowledge is no longer a peripheral topic but a core competency linking technical understanding of attack surfaces with knowledge of evidentiary, documentation, and reporting obligations.

6. The Bottleneck: Skills Shortage in an Era of Acceleration

All of these developments are hitting an already strained staffing situation. The "ISC2 Cybersecurity Workforce Study" most recently put the global skills gap at 4.8 million unfilled positions; according to ISACA, 90 percent of security teams report skills gaps, while only 14 percent of organizations actually have the professionals they need (StationX). What matters here is less the raw headcount than the nature of the missing competencies: according to ISC2, AI and machine learning are, at 41 percent - up from 34 percent the previous year - by far the most frequently cited qualification need among security teams (StationX). The ISC2 analysis "Aligning Skills, People and Hiring" confirms that the bottleneck increasingly lies not in the number of employees but in the lack of current skill profiles in areas such as AI, cloud security, risk assessment, and governance (ISC2). This gap cannot be closed through new hiring alone - it requires targeted, structured upskilling of both current and aspiring professionals.

7. Qualification as the Answer: A Curriculum for the New Threat Landscape

From the preceding sections, a clear competency profile emerges for what cybersecurity professionals need today: understanding of how generative and agentic AI works and where its limits lie; the ability to recognize and secure AI-specific attack surfaces (adversarial attacks, prompt injection, model supply chains); practical experience using AI in defense, including its limitations such as automation bias and alert fatigue; knowledge of the growing threat from deepfakes and AI-driven social engineering; and, finally, familiarity with the regulatory framework of the EU AI Act, NIST frameworks, and ISO/IEC 42001.

The eight-week, fully digital M.Sc. module "AI & Emerging Topics in Cybersecurity," whose structure is presented in the appendix, addresses exactly this competency profile. The module combines short weekly video lessons with a code-based exercise in a shared Jupyter/Colab environment and a team-based analysis session, deliberately moving along the four central tension fields identified above: from the fundamentals of generative AI, through offensive and defensive applications, to regulatory frameworks, AI-supported defense, and a concluding synthesis project. It illustrates, by example, how the abstract challenge outlined in this paper - the arms race between AI-powered offense and a still-catching-up, understaffed defense - can be translated into a concrete, exam-relevant learning program for aspiring security professionals.

8. Conclusion

The central challenge that powerful AI systems pose for cybersecurity today is not a single technical vulnerability but a multidimensional shift: attacks are becoming faster, more personalized, and increasingly autonomous; AI systems themselves are becoming new, often inadequately secured attack surfaces; defense benefits from AI-powered tools but remains disadvantaged by a structural cost asymmetry; regulation is catching up in stages without yet providing full legal certainty; and all of this is meeting a labor market that lacks precisely the AI-related competencies needed to manage this situation. Institutions of academic continuing education - such as the German University of Digital Science with the module presented here as an example - therefore bear a particular responsibility: they must design curricula that convey not only technical knowledge but also the ability to make well-founded, responsible decisions in an accelerating, AI-shaped threat environment.

Appendix: Curriculum Example "AI & Emerging Topics in Cybersecurity"

Module Overview

The following fully digital 8-week module (5 ECTS) serves as a concrete example of how the challenges described above can be translated into a structured curriculum for M.Sc. cybersecurity students. Each week combines roughly ten short video lessons (5–10 minutes) with a code-based exercise in a shared Jupyter/Colab environment and a team-based analysis session; prerequisites are basic knowledge of cybersecurity or machine learning.

Module Learning Outcomes: Upon successful completion, students will be able to (LO1) explain central concepts of generative AI and large language models and assess their relevance to cybersecurity operations; (LO2) critically evaluate the opportunities and limits of AI in offensive and defensive security applications, including in Security Operations Centers; (LO3) identify vulnerabilities, adversarial threats, and ethical risks in AI deployment; (LO4) apply AI methods to solve real-world security problems; (LO5) develop and present AI-supported security scenarios in teams; (LO6) evaluate regulatory developments in AI security and anticipate their implications for professional practice.

Weekly Structure Overview

Guiding Principles of the Module Design

The curriculum follows five overarching principles derived directly from the threat landscape described above. First, the module works exclusively with sandboxed, synthetic, or recorded materials - no functional malware, real deepfakes of actual people, or attacks on production systems are ever created. Second, offensive demonstrations (weeks 3, 4, and 6) are consistently framed as authorized, defensive red-teaming, in line with responsible-disclosure norms. Third, the governance thread from week 5 (NIST CSF 2.0, NIST AI RMF, ISO/IEC 42001, EU AI Act) is deliberately revisited in weeks 3, 6, and 8, anchoring regulation as a recurring theme rather than an isolated topic. Fourth, in this fully digital format, a weekly, team-organized analysis session replaces the traditional in-person session, supplemented by an optional live office hour. Fifth, the incidents referenced in the module - such as the JadePuffer case, deepfake fraud cases, or the evolving timeline of the EU AI Act - are deliberately treated as "living" case studies, reviewed for currency before each new cohort and replaced with newer, more relevant examples as needed.

Source list: 30 sources (Anthropic, Ars Technica, WEF, Kiteworks, Check Point Research, CrowdStrike, Cyble, StationX, Surfshark, Digital Applied, Netarx, Conversational Geek, ESET/Yahoo Finance, IBM Newsroom, Gruppo TIM, Fortinet, PwC, artificialintelligenceact.eu, Cloud Security Alliance, Software Improvement Group, ADVISORI, StingrAI, ISC2) - full list with links in the original document.
Structuring a risk assessment, regulatory outlook, emerging agentic security architectures, debate on extreme/catastrophic AI risks, final synthesis along NIST CSF 2.0