Rank #4 ITN Score: 22/30

Catastrophic AI Misuse

AI systems used deliberately by state or non-state actors for mass casualties via cyberattacks on critical infrastructure, autonomous weapons, or supercharged bioweapon design.

Submit New Draft

1 approved

Sort

AI as a Force Multiplier for Harm: Governing Catastrophic Misuse Scenariosv1Accepted

WorldProblems Consortium

Artificial intelligence does not need to be misaligned to cause catastrophe — it only needs to be misused. State and non-state actors can already leverage AI for cyberattacks on critical infrastructure, accelerated weapons development, and mass disinformation at unprecedented scale. This report maps the near-term misuse threat landscape, evaluates governance gaps, and proposes a research and policy agenda to reduce catastrophic risk from deliberate AI exploitation.

+0 votes

Followers

Approved Docs

Evidence & References

80,000 Hours Problem Profile

https://80000hours.org/problem-profiles/catastrophic-ai-misuse/

Problem Profile

Why Pressing

Frontier AI systems are approaching capabilities that could meaningfully aid the creation of bioweapons, large-scale cyberattacks, and mass disinformation. OpenAI publicly warned in 2025 that its agent systems could help novices create severe biological harm. Safeguards across labs are inconsistent, and both state and non-state actors are expanding efforts to extract capabilities from open-weight and API-accessed models. The window for establishing evaluations, compute governance, and deployment norms is open now but narrowing as more powerful systems are released every few months.

Scale

Potential impacts include bioweapon synthesis uplift, attacks on critical infrastructure such as power grids and water systems, and large-scale targeted disinformation and influence operations. A single significant misuse event could cost millions of lives or destabilize geopolitics in ways that cascade into broader catastrophe.

Neglectedness

Technical AI safety receives roughly 250 million dollars per year globally, and misuse-specific work is only a fraction of that. Policy capacity at US, UK, and EU governments is growing but lags deployment speed. Independent evaluation bodies such as METR and national AI Safety Institutes are new and under-resourced.

Tractability

Concrete levers have been demonstrated: dangerous-capability evaluations, pre-deployment testing, responsible scaling policies, compute and export controls, and DNA-synthesis screening to reduce bio-uplift. Industry has accepted some voluntary commitments. The regulatory environment is receptive, and additional talent in policy and evaluation can have outsized effects.