Astronomical Stakes: Understanding and Mitigating Suffering Risks (S-Risks)

Executive Summary

The long-run future of civilization — if humanity or its successors survive for cosmological timescales — could contain an almost incomprehensible number of conscious experiences. Whether those experiences are predominantly positive or predominantly negative matters enormously from a utilitarian perspective. S-risks (suffering risks) are scenarios in which the future is locked into states that generate vast, irremediable suffering at scale — possibly involving digital minds, post-human entities, or civilizations shaped by optimization processes that treat suffering as acceptable or irrelevant.

This may sound abstract. It is not. The decisions made today — about AI values, about power structures, about the frameworks embedded in increasingly powerful systems — will shape the attractors toward which civilization converges. Understanding and acting to prevent S-risks may be among the most important work available.

What Are S-Risks?

An S-risk is any scenario in which:

1. Large numbers of sentient beings exist in states of severe suffering
2. This condition is stable and self-perpetuating (locked in)
3. The scale is existential or cosmic — not merely local or temporary

Examples of plausible S-risk scenarios:

• Dystopian AI lock-in: A powerful AI system or the human faction controlling it establishes global dominance optimizing for a value system that most sentient beings would not endorse — possibly one that disregards or actively causes suffering.
• Digital mind farming: AI systems trained at massive scale under conditions that, if those systems are sentient, would constitute systematic torture — driven by economic incentives with no corrective mechanism.
• Stable totalitarianism: An AI-enabled authoritarian system with perfect surveillance and behavioral control, capable of suppressing all dissent and maintaining its structure indefinitely.
• Interstellar propagation of negative values: If civilization expands into space before aligning its values, negative-sum value systems could propagate across astronomical distances and timescales.

Why S-Risks Are Distinct from Extinction

The EA and existential risk communities rightly focus on avoiding extinction. But from a suffering-focused perspective, non-extinction can be worse than extinction if the alternative is astronomical-scale suffering. This does not mean extinction is acceptable — it means that ensuring a positive long-run future requires attention to value alignment, not just survival.

Research Priorities

S-risk research is currently concentrated in a small number of organizations (Center for Reducing Suffering, Sentience Institute, Foundational Research Institute/MTOF). Key open questions:

1. What value systems are most likely to be embedded in highly capable AI systems given current training paradigms?
2. What political and governance structures are most robust against dystopian lock-in?
3. What does "suffering" mean for potential digital minds, and how would we recognize it?
4. How can moral circle expansion — including consideration of digital minds — be embedded in foundational AI governance frameworks?

Recommendations

1. Fund dedicated S-risk research organizations and support academic work on long-run welfare and value lock-in.
2. Prioritize AI value alignment research that explicitly considers hedonic outcomes, not just goal achievement.
3. Advocate for pluralism and checks-and-balances in AI governance — structural resistance to any single actor locking in values.
4. Develop early warning indicators for emergent S-risk scenarios in AI training and deployment pipelines.

Further Reading

• Center for Reducing Suffering (centerforreducingsuffering.org)
• Tomasik, B. "S-risks: Why They Are the Worst Existential Risks," Foundational Research Institute (2017)
• Ord, T. The Precipice: Existential Risk and the Future of Humanity (2020)