Skip to content
The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation

The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation

Authors: Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, and 27 others

Venue: arXiv (1802.07228), February 2018

Affiliations: University of Oxford (Future of Humanity Institute), OpenAI, Electronic Frontier Foundation, Center for a New American Security, Stanford University, and others

TL;DR

A landmark 2018 report surveying the landscape of potential security threats from malicious uses of AI across three domains—digital, physical, and political security. The work proposes frameworks for forecasting AI-enabled attacks, identifies specific threat scenarios including AI-generated disinformation and deepfakes, and recommends policy interventions to prevent misuse while enabling beneficial AI development.

Contributions

  • Comprehensive threat modeling across digital, physical, and political security domains
  • Analysis of how AI capabilities (efficiency, scalability, anonymity, new vulnerabilities) enable new attack vectors
  • Specific scenarios for political attacks: deepfakes, automated disinformation campaigns, denial-of-information attacks, information manipulation
  • Policy recommendations for policymakers, researchers, and engineers to mitigate harmful AI applications
  • Framework connecting AI capabilities to security-relevant properties and threat landscape implications

Method

The report frames AI's malicious applications through three security domains:

Digital Security: Examines how AI automation lowers barriers to cyberattacks (e.g., spear phishing, malware crafting) and introduces novel vulnerabilities in AI systems themselves (adversarial examples, data poisoning, evasion techniques).

Physical Security: Analyzes AI-enabled autonomous systems (drones, robotic weapons, swarms) and discusses how widespread robotics combined with AI increases attack capabilities while creating novel defenses challenges.

Political Security: Focuses on AI-enabled information warfare and manipulation—the most relevant domain for fake news research: - Fake news reports with realistic synthetic video and audio (deepfakes) - Automated, hyper-personalized disinformation campaigns targeting specific voter populations - AI-driven influence campaigns that identify key social media influencers and target them with malicious content - Denial-of-information attacks: bot-driven information swamps that make truthful information harder to find - Algorithmic content curation to manipulate information availability

Results

The report identifies three major shifts expected in the threat landscape as AI capabilities advance:

  1. Expansion of existing threats: Lower costs and greater scalability enable more actors to conduct attacks at higher frequencies against more targets

  2. Introduction of new threats: AI enables attacks previously infeasible for humans (e.g., speech synthesis for impersonation, autonomous drone swarms, adversarial examples against AI systems)

  3. Change in threat character: AI-enabled attacks are predicted to be more effective, more finely targeted, more difficult to attribute, and more likely to exploit AI system vulnerabilities

The political security scenarios are particularly prescient: the report details how synthetic media could spread false information at scale, how algorithmic targeting could personalize disinformation, and how information availability could be manipulated to shape public discourse.

Connections

  • Related to Disinformation Detection as an emerging defense against AI-generated false content
  • Related to Deepfake Detection as a counter to deepfake-based political manipulation
  • Related to Social Media Amplification in the context of algorithmic curation attacks
  • Related to Dual-use technology as a framework for understanding AI's inherent dual-use nature
  • Cited in policy discussions about AI governance and responsible AI development

Notes

This is a landmark interdisciplinary report that brought together security experts, AI researchers, and policy specialists to systematically analyze AI misuse scenarios. The political security section, written before 2020 election cycles, proved prophetic in identifying deepfakes and disinformation campaigns as primary concerns. The report's emphasis on dual-use technology—the observation that AI systems designed for beneficial purposes can easily be repurposed for harm—set the foundation for subsequent work on AI safety and responsible disclosure. The policy recommendations (e.g., pre-publication risk assessment, responsible disclosure norms, stakeholder coordination) have become standard in AI governance discourse.

One limitation: the report explicitly excludes indirect effects (mass unemployment, second-order societal harms) and system-level competitive races between defensive and offensive AI, focusing instead on direct malicious intentional use. This narrower scope enabled concrete analysis but left open questions about broader security equilibria.

The work's foundational importance lies in legitimizing AI security and misuse as serious research topics worthy of deep technical and policy engagement, moving beyond speculative AI safety discourse to concrete threat scenarios and interventions.