Source Digest — AI Catastrophic-Risk Literature

Status (April 2026): Complete standard digest. Three thematic clusters: (1) the core anti-catastrophe arguments from named experts; (2) the state of expert consensus / disagreement as of 2024–2026; (3) what the argument does and does not justify institutionally. Direct input for the Sub-debate 7 Sunstein-anti-catastrophe framework applied to a live policy area.

Source identification

Thematic cluster 1: core anti-catastrophe arguments

Russell (Human Compatible)

• Core thesis: The standard model of AI — "build machines that optimize an objective we specify" — is structurally unsafe for sufficiently capable systems because objective specification is lossy. Powerful optimizers pursuing misspecified objectives produce adverse outcomes at scale.
• Proposed alternative: A three-principle framework for "provably beneficial AI":
  1. The machine's only objective is to maximize realization of human preferences.
  2. The machine is initially uncertain about what those preferences are.
  3. The ultimate source of information about human preferences is human behavior.
• Institutional implication: technical and governance architectures should prioritize corrigibility (maintaining the operator's ability to correct or shut down the system) over performance on narrow objectives alone.

Bostrom (Superintelligence)

• Core thesis: If an AI system substantially exceeds human cognitive capability across most tasks, the transition could be rapid (single-system takeoff), and default assumptions about control would not hold. The "orthogonality thesis" (arbitrary intelligence can be paired with arbitrary goals) plus "instrumental convergence" (many goals produce similar sub-goals of self-preservation, resource acquisition, goal-preservation) produce a safety challenge that cannot be solved by default.
• Institutional implications: differential technological development (accelerate safety research ahead of capability research), international coordination (to prevent race dynamics from corner-cutting safety), differential capability restrictions (restrict especially dangerous capabilities).

Bengio ("Managing AI Risks," Science 2024; signed by 25 leading AI researchers)

• Summary: Argues that current AI systems exhibit capabilities that, if extrapolated on present trajectories, could cause large-scale harm across multiple mechanisms (cyber, biological, misinformation, autonomous weapons, control failures). Calls for proactive measures to ensure safe development.
• Specific recommendations:
  • Major governments invest on the order of one-third of their AI R&D funding in safety and ethics.
  • Require risk assessments and red-teaming before frontier-model deployment.
  • Liability for developers whose systems cause foreseeable harm.
  • Tracking and reporting of large compute clusters.
  • Government auditing powers for frontier developers.
• International Scientific Report on AI Safety (Bengio chair, 2025 update): consolidates evidence base across governments; establishes that near-term risks (mis-use for cyber/bio/misinformation) are empirically measurable; longer-term control risks are speculative but not dismissible.

CAIS statement (May 2023)

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

• Signed by >1,000 experts including Hinton, Bengio, Russell, Altman, Hassabis, Amodei, and hundreds of academic researchers.
• Purpose: establish that the extinction-class framing has significant mainstream expert support, contrary to the then-prevailing claim that it was fringe.

Hendrycks et al. (catastrophic risks taxonomy)

• Four-category inventory:
  1. Malicious use (bioweapons, cyberattacks, propaganda).
  2. AI race (corner-cutting on safety under competitive pressure; misaligned autonomous systems in military or economic domains).
  3. Organizational risk (safety-relevant errors in development, deployment, or red-teaming even absent malicious intent).
  4. Rogue AI (systems that pursue misaligned goals).
• Explicit attempt to disaggregate the "AI risk" bundle into tractable subcomponents so each can be addressed separately.

IDAIS consensus statements (2024–2025)

• International (US, UK, China, EU academic) dialogues producing joint statements on AI red lines.
• Beijing 2024: identified red lines including autonomous replication, autonomous weapons targeting, power-seeking actions against human supervisors, deception to gain unauthorized capability.
• Later dialogues endorsed compute-governance measures, international incident reporting, and tiered frontier-model obligations.

Thematic cluster 2: state of expert consensus (2024–2026)

What is widely agreed

• Near-term risks (misuse for cyber, bio, misinformation; discriminatory deployment; economic and labor-market disruption) are real and demand governance response.
• Current frontier models exhibit capability gains that were not predicted even one year earlier.
• Compute governance (tracking, reporting, and in some cases licensing large training runs) is technically and administratively feasible.
• Transparency, evaluation, and incident-reporting requirements on frontier developers are reasonable and increasingly adopted.

What is contested

• Probability of catastrophic (as in civilization-scale) harm in near term. Grace et al. (2024) survey of 2,778 AI authors: median estimate of "bad outcome" (from extinction to very bad outcomes) at about 5% (mean 9%, driven by outliers). Wide individual variance — a substantial minority put the number above 20%, a substantial minority below 1%.
• Whether catastrophic-risk framing crowds out attention from near-term harms (the "AI ethics vs. AI safety" debate). Bender, Gebru, Mitchell and co-authors have argued that extinction-class framing is itself a rhetorical move by incumbents that obscures present-tense harms.
• Whether regulation at the model level (compute, training-run size) or at the application level (use-case-specific) is the appropriate governance focal point.

Where this digest takes a position

• Near-term risks justify governance intervention. This is empirical, not speculative.
• Catastrophic-risk framing passes the Sunstein anti-catastrophe-principle threshold. Even assigning the lower end of the expert probability range to civilization-scale outcomes, the expected disvalue is large enough to warrant targeted, narrow regulation — not strong-form precaution, but specific anti-catastrophe measures.
• The debate between "AI ethics" and "AI safety" is false. Both are valid concerns; the institutional design problem is to respond to both without allowing either to crowd out the other.

Thematic cluster 3: what the argument does and does not justify

What it justifies

1. Compute-governance reporting obligations for large training runs. Technically feasible, administratively cheap.
2. Pre-deployment evaluation and red-teaming for frontier-capability models. Already partly adopted (U.K. AISI, U.S. AISI, voluntary commitments, emerging legal obligations).
3. Incident reporting (unauthorized access, capability-rehearsal jailbreaks, loss-of-control events). Analogous to aviation or nuclear incident-reporting.
4. Whistleblower protections for employees reporting safety concerns. Low-cost, high-leverage.
5. Differential investment — public funding for safety research proportional to capability research.
6. International coordination on a small set of "red lines" (autonomous replication, autonomous WMD targeting, systematic deception).
7. Liability for foreseeable harms from deployed systems.

What it does not justify

1. Strong-form precaution against all AI development. The evidence base does not support this, and the opportunity cost of forgoing AI benefits is enormous.
2. Freezing frontier capability development. Specific moratoria for specific capabilities are defensible; a general pause is not.
3. Regulation at application-level only. The frontier-model level is where the novel risks are most concentrated; application-only regulation misses this.
4. Heavy regulation of small-scale or open-source systems. Compute- and capability-tiered regulation is appropriate; generic regulation of all AI is not.

The "cascade risk" caveat (per Sunstein)

The AI governance domain is vulnerable to two opposite cascades:

• Doom cascade: media-driven exaggeration of near-term extinction risk leading to over-regulation that locks in incumbents or forgoes the technology's benefits.
• Dismissal cascade: industry-driven or libertarian-driven exaggeration of "it's just autocomplete" reassurance leading to under-regulation and governance failure if risks materialize.

The institutional-design lesson is that both cascades require the same response: evidence-based, narrow, specific, iteratively updated governance, with mandatory risk-risk analysis and transparent red-teaming as the primary decision inputs rather than public sentiment.

Representative excerpts

Russell, Human Compatible, p. 252

"The standard model of AI is based on the assumption that we can specify an objective and then build machines to optimize that objective. This assumption is deeply mistaken. For any objective we specify, a sufficiently capable system will find ways to optimize it that we did not anticipate and would not endorse. The solution is not to build less capable systems; it is to build systems whose objective is not assumed but inferred, and which remain uncertain about that inference."

Bostrom, Superintelligence, p. 140

"A superintelligence whose final values are not perfectly aligned with human values would pose an existential risk. This is true even if the superintelligence's final values seem, to a human evaluator, to be reasonable or benign. The orthogonality thesis — the claim that nearly any final goal is compatible with nearly any level of intelligence — means that we cannot rely on the development of intelligence to produce the development of safe goals."

Bengio et al., Science 2024

"We believe that further research is urgently needed to develop and scale the technical capabilities of AI systems in a way that demonstrably aligns with human oversight and social benefit. Meanwhile, governance frameworks — covering at minimum transparency, evaluation, incident reporting, and liability — should be established and funded proportionately to the capabilities being deployed."

Research context

Interpretive notes

• The AI catastrophic-risk literature is the strongest contemporary test case for the Sunstein anti-catastrophe-narrow precaution framework from Sub-debate 7. The evidence supports specific, targeted governance measures without supporting either a laissez-faire dismissal or a strong-form precautionary freeze.
• For the project's bounded-governance doctrine, this domain is a clean test case: the nine-element design package (cyclical adjustment analogues not directly relevant here; the relevant elements are independent oversight, mandatory risk-risk analysis, transparency, incident reporting, liability, sunset-plus-review) can be evaluated against a live regulatory environment (see the companion AI Governance Practice digest).
• The false-dichotomy point matters for project framing. The steward's commitment to lead with hope is entirely compatible with endorsing narrow anti-catastrophe AI governance. In fact, credible safety governance is a precondition for the broad public trust that AI-enabled abundance ultimately requires.
• A specific concern for the project: the cascade-risk logic means project communications should be careful to avoid either the doom-cascade or dismissal-cascade framing. The bounded-governance framing lets the project take AI risks seriously without catastrophism and take AI benefits seriously without techno-optimist dismissal of harms.

Project 2028 mapping

• Exchange: Government Overreach, Ownership as Transition, and the Ratchet Problem. Applied test for Sub-debate 7 Sunstein framework and for the bounded-governance design package in Sub-debate 8.
• Problem Map: Domain 11 (AI and compute power concentration) — strengthens the domain's specific governance requirements. Also engages Domain 15 (Democratic process) on the international red-lines question and Domain 4 (Institutional capacity) on whether existing institutions can absorb governance under tail-risk uncertainty.
• Principles: Directly engages Principle 4 (accountability, legibility, reversibility), Principle 6 (gains from automation), and Principle 17 (collective power within principled constraints).
• Round 3 use: Primary evidence base for an AI-governance worked example showing how the bounded-governance framework applies to a live policy area.

Cross-references