Skip to content
The Threat of Offensive AI to Organizations

The Threat of Offensive AI to Organizations

Authors: Yisroel Mirsky, Ambra Demontis, Jaidip Kotak, Ram Shankar, Deng Gelei, Liu Yang, Xiangyu Zhang, Wenke Lee, Yuval Elovici, Battista Biggio

Venue: ACM Computing Surveys, Vol. 1, No. 1, pp. 1–31 — July 2021

TL;DR

A comprehensive survey identifying 33 offensive AI capabilities that adversaries can use to attack organizations, organized into seven categories: automation, campaign resilience, credential theft, exploit development, information gathering, social engineering, and stealth. Through a user study with 22 experts from academia and industry, the authors rank these threats and find that exploit development, social engineering, and information gathering pose the greatest risks, with exploitation of human and software vulnerabilities consistently outranking malware-focused attacks.

Contributions

  • An overview of how AI can be weaponized against organizations and its impact on the cyber kill chain
  • Enumeration and categorization of 33 offensive AI capabilities identified through literature review
  • A threat-ranking methodology based on profit, achievability, defeatability, and harm, validated through an expert user study
  • Comparative analysis of threat perceptions between industry and academia practitioners
  • Forecast of emerging AI-enabled attack trends and their tactical implications

Method

The survey employed three parallel approaches. First, a comprehensive literature review using the MITRE ATT&CK framework as a organizational guide identified common tactics and techniques adversaries employ when attacking organizations. Second, the authors enumerated 33 specific offensive AI capabilities (OACs) — techniques where ML or AI directly improve an attacker's ability to achieve a goal. Third, they conducted a user study with 22 experts (roughly half from academia, half from industry) spanning organizations such as MITRE, Microsoft, Airbus, Bosch, and universities. Participants rated each OAC on four dimensions:

  • Profit (P): Benefit gained by using AI compared to non-AI methods
  • Achievability (A): Ease of implementing and deploying the AI capability
  • Defeatability (D): Difficulty for defenders to detect or prevent the attack
  • Harm (H): Physical, psychological, or financial damage inflicted

The authors computed a threat score T = H × (M / D), where M = ⅓(P + A), synthesizing both attacker motivation and defender capacity. Responses were normalized to 0–1 and results stratified by sector to compare industry and academic threat perceptions.

33 Offensive AI Capabilities

The capabilities span seven categories:

  1. Automation (6): Attack Adaptation, Attack Coordination, Next-Hop Targeting, Phishing Campaigns, Point of Entry Detection, Record Tampering
  2. Campaign Resilience (4): Campaign Planning, Malware Obfuscation, Persistent Access, Virtualization Detection
  3. Credential Theft (5): Biometric Spoofing, Cache Mining, Implicit Key Logging, Password Guessing, Side Channel Mining
  4. Exploit Development (2): Reverse Engineering, Vulnerability Detection
  5. Information Gathering (4): Mining OSINT, Model Theft, Spying, Extraction
  6. Social Engineering (5): Impersonation, Persona Building, Spear Phishing, Target Selection, Tracking
  7. Stealth (7): Covering Tracks, Evading HIDS, Evading NIDS, Evading Insider Detectors, Evading Email Filters, Exfiltration, Propagation & Scanning

Key Findings

  • 23 of 33 OACs (72%) rank as significant threats (threat score T > 1), indicating widespread vulnerability across modern organizations.
  • Top 3 Offensive AI Threats (overall):
  • Exploit Development
  • Social Engineering (especially impersonation and spear phishing)
  • Information Gathering
  • Industry vs. Academia divergence: Industry ranks reverse engineering and impersonation highest; academia prioritizes biometric spoofing and AI model theft. This reflects academia's forward-looking concern about emerging ML vulnerabilities, while industry focuses on threats already in the wild.
  • Motivation model: Adversaries are motivated by three factors—coverage (automating attacks), speed (parallel operations), and success (increasing attack effectiveness). The survey found that AI enhances all three dimensions simultaneously.
  • Cyber kill chain advantage: Participants broadly agreed that AI provides attackers an asymmetric advantage during initial attack stages (reconnaissance, weaponization) and defenders an advantage during later stages (detection, remediation). However, AI mitigates defenders' post-compromise options, limiting the human decision window.
  • Least threatening OACs: Scanning, cache mining, and some stateless automation tactics rank lowest, likely because defenders have deployed detection methods or because their impact is low-risk.

Results & Threat Ranking

Figure 2 (in the paper) visualizes threat scores for all 33 capabilities, color-coded by risk dimension. Figure 3 presents OACs ranked by threat score, with clear separation between high (T > 1) and low (T < 0.5) threats. Figure 4 overlays industry and academia responses, showing agreement on top threats but divergence on emerging risks. The model T = H × (M/D) consistently predicts real-world threat prioritization: high-profit, easy-to-achieve, hard-to-defend attacks rank highest regardless of harm, while low-harm attacks rank low even if easy to execute.

Connections

Notes

Strengths: The survey's dual focus—offensive capabilities and defender impact—is rare and valuable. The explicit threat-scoring model (T = H × M / D) is principled and transparent. The expert user study, though modest in size (n=22), spans multiple sectors and geographies, lending credibility. The MITRE ATT&CK mapping provides actionable framing for practitioners.

Limitations: The 22-expert sample carries a ~20% margin of error; the study would benefit from larger N. The threat model conflates achievability with likelihood of adoption, which are distinct (easy ≠ deployed). Some OACs (e.g., model theft) require assumptions about ML system exposure that differ widely by organization. Industry and academia sample sizes are unequal, and selection bias toward security-aware participants likely inflates perceived threat.

Relevance: This survey is foundational for understanding the adversarial ML landscape as it pertains to enterprise defense. It clarifies that the threat is not hypothetical—practitioners already rate 72% of identified tactics as significant—and grounds forecast trends (deepfakes, autonomous botnets, swarm intelligence) in near-term empirical reality. The asymmetry result (attacker advantage early, defender advantage late) should inform incident response prioritization.