Rank #5 ITN Score: 21/30

Power-Seeking AI Systems

Advanced AI systems may develop instrumental goals around self-preservation and resource acquisition, undermining human oversight and potentially leading to irreversible loss of control.

Key Statistic

Less than $100M/year is spent on AI alignment research vs. $100B+ on AI capabilities.

ITN Assessment

Importance10/10

Scale & severity

Tractability5/10

Ease of progress

Neglectedness6/10

Lack of attention

0 followers

Why This Matters

Potential existential risk; tractability limited but improving with alignment research; growing but still insufficient research investment.

Key Organizations

Redwood Research
Applied alignment research, including control and oversight.
Machine Intelligence Research Institute (MIRI)
Foundational research on aligning superintelligent systems.
Anthropic
Frontier lab with substantial alignment and interpretability work.
Alignment Research Center (ARC)
Research on evaluating and aligning advanced AI.
Center for AI Safety (CAIS)
Research, field-building, and advocacy on AI existential risk.

Submit New Draft

1 approved

Sort

The Alignment Problem: Why Goal Misspecification Poses an Existential Riskv1Accepted

WorldProblems Consortium

Advanced AI systems optimizing for misspecified objectives could pursue instrumental sub-goals — resource acquisition, self-preservation, goal-content integrity — in ways that undermine human oversight and permanently foreclose human control. This report explains the theoretical basis for the alignment problem, surveys the current state of alignment research, and argues for urgent, well-funded work on scalable oversight and interpretability.

+0 votes

Followers

Approved Docs

Evidence & References

80,000 Hours Problem Profile

https://80000hours.org/problem-profiles/artificial-intelligence/

Problem Profile

Why Pressing

Frontier AI systems already display rudimentary strategic and deceptive behavior in controlled evaluations, and capabilities are improving faster than alignment techniques. As systems become more agentic and are given more autonomy over code, money, and real-world actions, the risk of misaligned AI pursuing goals at civilization scale grows. Leading forecasters place non-trivial probability on existential catastrophe from misaligned AI this century, with several estimates above 10 percent. The pace of scaling continues to outstrip safety research, and decisions made in the next few years about training, deployment, and governance may shape outcomes for the long term.

Scale

Potentially existential. Toby Ord's The Precipice estimates roughly 10 percent existential risk this century from unaligned AI. Surveys of ML researchers give median estimates of several percent probability of human extinction or permanent disempowerment from AI. Even modest probabilities of an unrecoverable outcome make this one of the largest-scale risks we face.

Neglectedness

The technical alignment workforce is estimated at 1,000 to 2,000 full-time researchers globally, compared with tens of thousands in capabilities work at frontier labs. Alignment teams inside labs are usually a small share of headcount. Independent safety organizations such as MIRI, Redwood Research, ARC, and Apollo Research are small relative to the scale of the risk.

Tractability

Mechanistic interpretability, scalable oversight, adversarial training, and dangerous-capability evaluations are producing real progress. Governance wins such as UK and US AI Safety Institutes, the EU AI Act, and voluntary commitments show policy is viable. However, alignment is unsolved and may be hard to solve on the timelines implied by current scaling trends.