Catastrophic AI Misuse
AI systems used deliberately by state or non-state actors for mass casualties via cyberattacks on critical infrastructure, autonomous weapons, or supercharged bioweapon design.
Scale of the Problem
AI has already accelerated exploit discovery by 10-100x in red-team evaluations.
Potential impacts include bioweapon synthesis uplift, attacks on critical infrastructure such as power grids and water systems, and large-scale targeted disinformation and influence operations. A single significant misuse event could cost millions of lives or destabilize geopolitics in ways that cascade into broader catastrophe.
Why This Is Pressing
Frontier AI systems are approaching capabilities that could meaningfully aid the creation of bioweapons, large-scale cyberattacks, and mass disinformation. OpenAI publicly warned in 2025 that its agent systems could help novices create severe biological harm. Safeguards across labs are inconsistent, and both state and non-state actors are expanding efforts to extract capabilities from open-weight and API-accessed models. The window for establishing evaluations, compute governance, and deployment norms is open now but narrowing as more powerful systems are released every few months.
Why It's Neglected
Technical AI safety receives roughly 250 million dollars per year globally, and misuse-specific work is only a fraction of that. Policy capacity at US, UK, and EU governments is growing but lags deployment speed. Independent evaluation bodies such as METR and national AI Safety Institutes are new and under-resourced.
Can It Be Solved?
Concrete levers have been demonstrated: dangerous-capability evaluations, pre-deployment testing, responsible scaling policies, compute and export controls, and DNA-synthesis screening to reduce bio-uplift. Industry has accepted some voluntary commitments. The regulatory environment is receptive, and additional talent in policy and evaluation can have outsized effects.
Research & Solutions
AI as a Force Multiplier for Harm: Governing Catastrophic Misuse Scenarios
WorldProblems Solved
Sign in to read full documents, vote on solutions, and submit your own research.
What You Can Do
What this problem still needs
Careers in AI policy at RAND, the AI Safety Institute, congressional staff, or executive branch roles. Evaluations work at METR, Apollo Research, or lab red teams. Information security at frontier labs. Support the Horizon Institute, IAPS, or the Institute for Progress. Build specific expertise at the bio-AI or cyber-AI intersection.
Priority Score
22
Breakdown
Fund this problem
Support the researchers and organizations working on Catastrophic AI Misuse.
Donate to this causeContribute to solving this
Create a free account to submit research, vote on solutions, and follow this problem.
Join for free