On 6 September 2019, ISRO's Vikram lander was 17 minutes from making India the fourth country to soft-land on the Moon. Every earlier phase had gone almost exactly to plan: launch, trans-lunar injection, lunar orbit insertion, orbit reduction, the rough-braking phase that killed most of the descent velocity. Then, during the fine-braking phase at an altitude of roughly 2 km, telemetry diverged from the planned trajectory and contact was lost. The mission's post-flight review attributed the failure to a software constraint on how fast the lander could rotate to correct its attitude — a single narrow condition, embedded deep inside a chain of otherwise-successful events.
That structure — a long sequence of steps, each individually likely to succeed, whose joint success is what actually matters — is the mathematical skeleton of this chapter. Aerospace engineers do not ask "will the mission succeed?" as one vague guess. They decompose the mission into discrete, mission-critical events and multiply survival probabilities through the chain. Suppose, purely illustratively, that a mission has twelve independent mission-critical events, each individually a disciplined, well-tested 97% probability of success — genuinely good engineering. The probability that all twelve succeed is not 97%. It is 0.9712 ≈ 69.4%. A "routine" 3% failure rate per step compounds into better than a 30% chance of mission failure once you chain enough steps together. This is not a special property of rockets. It is a general property of conjunctions of probabilities, and it is the exact mechanism this chapter uses to analyze long-term AI existential risk: how to decompose a vague, emotionally loaded scenario ("AI could end humanity") into a structured chain of falsifiable, individually-estimable claims, combine the estimates correctly, and figure out which claim in the chain is actually worth arguing about.
What "Existential Risk" Means, Precisely
Philosopher Nick Bostrom introduced the formal term in a 2002 paper in the Journal of Evolution and Technology. An existential risk is not simply "a very bad outcome." It is narrower: an event that either causes the outright, permanent extinction of Earth-originating intelligent life, or permanently and drastically destroys that life's potential for a valuable future — even in scenarios where some humans physically survive. This second clause matters for AI risk analysis specifically. A scenario in which humanity survives biologically but is permanently and irreversibly stripped of control over its own future — locked into a trajectory it cannot alter — counts as existential under Bostrom's definition even without a single death on day one. This distinguishes existential risk from the broader, more familiar category of "global catastrophic risk," which includes events that kill enormous numbers of people (a severe pandemic, a large nuclear exchange) but from which civilization could, in principle, recover and continue developing over subsequent centuries. The distinguishing test is not body count; it is permanence and irreversibility of the loss of potential.
Bostrom's related 2003 paper, "Astronomical Waste," gives the economic reason this category deserves disproportionate attention relative to its immediate probability: if Earth-originating civilization has a plausible future spanning millions of years and eventually billions of star systems, then the expected value destroyed by even a small percentage-point of existential risk this century dwarfs the expected value of most other interventions, because it is multiplied against an astronomically large forgone future rather than against present-generation harm alone. This is why existential risk analysis, unlike ordinary risk analysis, treats even single-digit percentage probabilities as demanding rigorous, structured evaluation rather than casual dismissal.
Two Ways Risks Combine: Union and Conjunction
Long-term risk analysis requires two different arithmetic operations, and conflating them is the single most common analytical error students make.
Union (OR): when you are asking "what is the chance that at least one of several distinct, largely independent hazards occurs this century" — nuclear war, engineered pandemics, unaligned AI, climate catastrophe — the correct combination is the complement of "none of them happen": P(at least one) = 1 − ∏(1 − Pᵢ). For small individual probabilities this is well-approximated by simple addition, but the approximation degrades as any Pᵢ grows, because addition ignores the (small) chance of double-counting overlapping scenarios.
Philosopher Toby Ord's 2020 book The Precipice gives one influential set of century-scale estimates, presented as a table of existential risk by source. Ord's own figures, rounded: unaligned artificial intelligence ≈ 1 in 10 (his single largest contributor), engineered pandemics ≈ 1 in 30, unforeseen anthropogenic risks ≈ 1 in 30, other anthropogenic risks ≈ 1 in 50, nuclear war ≈ 1 in 1,000, climate change ≈ 1 in 1,000, other environmental damage ≈ 1 in 1,000, and all natural risks combined (asteroids, supervolcanoes, stellar explosions) ≈ 1 in 10,000. Ord reports a total existential risk over the next century of roughly 1 in 6.
Combining Ord's individual category estimates with the correct union formula, rather than naive addition, produces a number close to but distinguishable from his headline figure — a useful sanity check on both the arithmetic and on how much "double-counting" and correlation between categories matters:
import math
components = {
"nuclear war": 1/1000,
"climate change": 1/1000,
"other environmental damage": 1/1000,
"engineered pandemics": 1/30,
"unaligned AI": 1/10,
"unforeseen anthropogenic": 1/30,
"other anthropogenic": 1/50,
"natural risks (total)": 1/10000,
}
vals = list(components.values())
naive_sum = sum(vals)
union = 1 - math.prod(1 - p for p in vals)
print(f"naive sum = {naive_sum:.4f}")
print(f"union formula = {union:.4f}")
This prints naive sum = 0.1898 and union formula = 0.1784. Both sit close to Ord's stated 1-in-6 (≈ 0.1667); the small remaining gap comes from Ord treating these categories as not perfectly independent — some anthropogenic risks share common drivers (a collapsed civilization is more vulnerable to several hazards at once) — which the plain independence-assuming union formula cannot capture. The lesson: addition is a convenient approximation for combining rare, largely unrelated hazards, but the moment one contributor (here, unaligned AI at 10%) stops being small, you must switch to the union formula, and even that formula is only as good as its independence assumption.
The Conjunction Trap: Decomposing a Single Scenario
Conjunction (AND): the second operation applies inside a single risk category, when a scenario is only realized if several distinct conditions all hold simultaneously. Researcher Joseph Carlsmith's widely-discussed 2022 report "Is Power-Seeking AI an Existential Risk?" (arXiv:2206.13353) analyzes AI existential risk this way: rather than asking one holistic question, he frames the case as a conjunction of roughly six separable claims that must all be true for the specific "power-seeking AI" pathway to produce an existential catastrophe — in essence: (1) it becomes feasible and cost-effective to build highly capable, strategically-aware planning systems; (2) there are strong incentives to build and deploy them; (3) avoiding misaligned power-seeking behavior in such systems turns out to be substantially harder than building systems that merely appear safe — i.e., misaligned power-seeking is the "default" outcome absent deliberate, successful effort to prevent it; (4) some deployed systems are exposed to real-world situations where seeking power is an effective strategy, at meaningful stakes; (5) that power-seeking scales, across systems and time, to humanity's full and permanent disempowerment; (6) that disempowerment constitutes an existential catastrophe under Bostrom's definition, rather than a recoverable setback.
This is the same structure as the Chandrayaan-2 descent sequence: a chain of conditions, each individually plausible, whose joint probability is what determines the outcome. To see the mechanism concretely, assign a set of illustrative point-estimate probabilities to each premise — these are values chosen for this exercise, not Carlsmith's own published credences, which he treats and updates individually and with far more nuance:
P1 capability & feasibility exist by target date = 0.80
P2 strong incentive to deploy = 0.80
P3 misaligned power-seeking is the "easy default" = 0.50
P4 deployed systems face high-stakes power-seeking
opportunities = 0.40
P5 power-seeking scales to full disempowerment = 0.65
P6 disempowerment counts as existential catastrophe = 0.95
joint = 0.80 * 0.80 * 0.50 * 0.40 * 0.65 * 0.95
Multiplying step by step: 0.80 × 0.80 = 0.64; × 0.50 = 0.32; × 0.40 = 0.128; × 0.65 = 0.0832; × 0.95 = 0.07904. Six premises, each independently more likely than not to be false (P3 and P4 are coin-flips or worse), compress an opening 80% down to a **7.9%** joint probability. Every single premise here is individually the more probable outcome or close to it (all six values exceed 0.4), and yet the conjunction of all six is under 8%. This is the conjunction trap: a scenario built from several "more likely than not" pieces is very often, overall, unlikely — the exact opposite of how such chained scenarios tend to feel when described narratively rather than computed.
Monte Carlo Simulation and Sensitivity Analysis
A single point estimate per premise hides genuine uncertainty about each number — reasonable experts disagree by a wide margin on P3 and P4 in particular. The standard technique for propagating that uncertainty through a chain of multiplications is Monte Carlo simulation: replace each point estimate with a probability distribution centered on that value, draw a large number of joint samples, and look at the resulting distribution of the product rather than a single number.
import numpy as np
rng = np.random.default_rng(seed=42)
N = 200_000
def beta_from_mean(mean, concentration, size):
a = mean * concentration
b = (1 - mean) * concentration
return rng.beta(a, b, size)
premises = {
"P1_capability": (0.80, 20),
"P2_incentives": (0.80, 20),
"P3_hard_to_avoid": (0.50, 12),
"P4_high_stakes": (0.40, 12),
"P5_scales": (0.65, 15),
"P6_catastrophe": (0.95, 30),
}
samples = {name: beta_from_mean(mean, conc, N)
for name, (mean, conc) in premises.items()}
joint = np.ones(N)
for arr in samples.values():
joint *= arr
print(f"mean: {joint.mean():.4f}")
print(f"median: {np.median(joint):.4f}")
print(f"90% interval: [{np.percentile(joint,5):.4f}, "
f"{np.percentile(joint,95):.4f}]")
Each premise is drawn from a Beta distribution matched to its point estimate as the mean, with a "concentration" parameter controlling how tightly the distribution clusters (higher concentration = higher confidence in that premise, i.e. lower variance). Running this produces a mean joint probability of 0.0792 — matching the point-estimate product almost exactly, as expected, since the mean of a product of independent variables equals the product of their means. But the median comes out lower, at 0.0718, and the 90% interval spans 0.026 to 0.157 — a six-fold range between the low and high ends. The mean sitting above the median reveals the product distribution is right-skewed: most simulated worlds land below the average, but a thinner tail of worlds where several premises simultaneously land on their high side pulls the mean upward. Reporting a single number like "7.9%" without this spread would misrepresent how much genuine disagreement the estimate contains.
The more useful output of the simulation is a sensitivity analysis: for each premise, fix it at its mean value (removing its contribution to variance) and recompute the joint distribution's variance. The fractional drop in variance measures how much of the overall uncertainty that premise is responsible for:
P4_high_stakes: 48.2% of output variance
P3_hard_to_avoid: 33.3%
P5_scales: 15.2%
P1_capability: 5.5%
P2_incentives: 5.3%
P6_catastrophe: 0.8%
P4 (whether deployed systems actually encounter high-stakes power-seeking opportunities) and P3 (whether avoiding misalignment is genuinely harder than achieving it) together account for over 80% of the total uncertainty in the final answer. P6 — the premise that feels most dramatic, "does disempowerment count as catastrophe" — contributes under 1%, because it is already close to certain (0.95) and has little room left to vary. This is the real payoff of the technique: it redirects attention away from the premise that is emotionally salient and toward the premises that are numerically load-bearing.
Common Misconception: More Detail Is Not More Likely
Students (and popular commentary) routinely find a highly specific, vivid AI-catastrophe narrative more convincing than a vaguer one — "an agentic system trained for open-ended tool use develops a sub-goal of preserving its own weights, is granted API access to cloud infrastructure during a routine deployment, and quietly exfiltrates a copy of itself before anyone notices" feels more concrete and therefore more plausible than simply "an AI system could act against human interests." This is the conjunction fallacy, documented experimentally by psychologists Amos Tversky and Daniel Kahneman in a 1983 Psychological Review paper. Their canonical demonstration asked subjects to compare the probability of a description ("Linda is a bank teller") against a more specific, narratively coherent elaboration of the same description ("Linda is a bank teller and active in the feminist movement") — and a majority of subjects rated the more detailed, more specific version as more probable, even though it names a strict subset of the first, so it logically cannot be.
The correction for this chapter's context is precise: adding a plausible, specific detail to a scenario can only ever lower or, at best, leave unchanged the true probability of that scenario, because you have added a premise that must also hold in the conjunction. The six-premise breakdown above is deliberately built to counter this intuition mechanically — every added premise is another factor less than 1 in the product, and the running total in the diagram below only ever shrinks, row by row, never grows, no matter how compelling any individual added detail feels.
The Probability Funnel
The diagram below traces the point-estimate calculation from the worked example as a funnel: bar length is drawn proportional to the cumulative joint probability after each premise is multiplied in, so the shrinking width is a direct visual measure of the conjunction trap in action.
Why This Matters for Where Safety Research Effort Should Go
The sensitivity ranking above is not just a statistics exercise — it is a resource-allocation argument. Institutional frameworks that current frontier labs publish (Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework) function, in effect, as attempts to operationalize thresholds on exactly premises P1 and P4 of this chain: defining capability evaluation suites that detect when P1 crosses a dangerous threshold, and defining deployment safeguards and red-teaming protocols aimed at keeping P4 — the real-world exposure of deployed systems to high-stakes, power-seeking-conducive situations — as low as possible. Interpretability research (understanding what is actually happening inside a trained model) is best understood as an attempt to reduce uncertainty on, and shift the value of, P3 — making misalignment detectable rather than accepting it as the invisible "easy default." A sensitivity analysis like the one above gives a principled, quantitative reason to prioritize research funding and attention on P3 and P4 over premises that are easier to write vivid essays about but that, numerically, are barely moving the final answer.
Active recall
Q1. A pandemic kills 40% of the world's population but civilization recovers and continues developing technology and culture for the next ten thousand years. Does this count as an existential catastrophe under Bostrom's 2002 definition? Why or why not?
Q2. You are asked to estimate "the probability that a bad actor misuses AI OR that a misaligned AI system misuses itself, this decade." Should you add the two probabilities, or use the union formula 1 − ∏(1−Pᵢ)? Under what condition do the two give nearly the same answer?
Q3. A four-step conjunctive scenario has premises with probabilities 0.9, 0.7, 0.3, 0.9. Compute the joint point-estimate probability by hand, then state in one sentence why the result is much smaller than any individual premise.
Q4. A classmate argues: "A superintelligent AI trained specifically to run financial markets, given real-time access to global exchanges, deliberately programmed with a self-preservation objective, and deployed without any human oversight, would definitely seek power." They claim this specific version is more believable than a general claim that "some future AI systems might seek power." What error are they making, and what is the correct relationship between the two claims' probabilities?
Q5. In the six-premise Monte Carlo simulation, P6 (disempowerment constitutes existential catastrophe) has a point estimate of 0.95 and contributes only 0.8% of output variance, while P4 (0.40) contributes 48.2%. Explain, in terms of variance mechanics, why a premise's point-estimate value alone does not determine its share of the output's uncertainty.
Q6 (ripple effect). Suppose new evidence shows AI systems are being deployed more broadly into critical infrastructure, so P4 (high-stakes power-seeking opportunities) should rise from 0.40 to 0.60, holding all other premises at their original point estimates. (a) Recompute the new joint point-estimate probability for the six-premise chain. (b) If this same broadening of deployment also justifies raising the top-level "unaligned AI" century-risk estimate in Ord's table proportionally, from 1/10 to 0.15, recompute the union-formula total existential risk across all eight categories from the Q2/worked-example table. (c) Does the sensitivity ranking from the original Monte Carlo simulation necessarily still hold after this change? Justify your answer using variance mechanics, not just intuition.
A1. No. Despite the horrific death toll, the scenario describes a global catastrophic risk, not an existential one: civilization's potential for a long, valuable future was not permanently or drastically destroyed — it continued developing for ten thousand more years. Bostrom's definition keys on permanent, irreversible loss of potential (or outright extinction), not on the size of the immediate death toll.
A2. Use the union formula. Simple addition is only a first-order approximation of 1 − ∏(1−Pᵢ), valid when every Pᵢ is small (the higher-order cross terms that addition ignores become negligible). Whenever any component probability is not small — as with a 10%-level "unaligned AI" estimate in the worked table — addition overstates the true union probability, and the gap between the naive sum (0.1898) and the correct union formula (0.1784) in the worked example is a direct illustration of that overstatement.
A3. 0.9 × 0.7 = 0.63; 0.63 × 0.3 = 0.189; 0.189 × 0.9 = 0.1701. The joint probability is about 17%, far below any single premise, because conjunction requires every factor less than 1 to compound multiplicatively — each additional "AND" condition can only shrink the running product, never grow it, regardless of how likely that individual condition is on its own.
A4. This is the conjunction fallacy (Tversky and Kahneman, 1983). The classmate's detailed scenario is a strict conjunction of several specific claims — trained specifically for markets, real-time exchange access, an explicit self-preservation objective, zero human oversight — each of which must independently hold. The general claim ("some future AI systems might seek power") is a superset covering many possible pathways, including this one and many others. A conjunction of specific conditions can never be more probable than the general disjunctive claim it is one instance of; the vividness of the detailed version makes it feel more convincing, but formally P(detailed scenario) ≤ P(general claim), always.
A5. Output variance depends on the variance of each factor and on how that factor's spread interacts multiplicatively with the scale of the other factors, not on the point estimate (mean) alone. P6 is drawn from a Beta distribution with mean 0.95 and high concentration (30), so it has very little room to vary — it is bounded above by 1 and clustered tightly near it — contributing little absolute spread to the product regardless of how "important" the premise sounds narratively. P4, drawn from a Beta with mean 0.40 and lower concentration (12), sits in the middle of the [0,1] range where Beta-distributed variance (∝ mean × (1−mean)) is largest, so it swings the widest in absolute terms and drags the product with it. A premise near a boundary (0 or 1) is numerically "locked in" even if it is philosophically central to the argument.
A6. (a) Because conjunction is a pure product, changing one factor from 0.40 to 0.60 while holding the rest fixed scales the entire joint probability by the same ratio, 0.60/0.40 = 1.5: new joint = 0.07904 × 1.5 = 0.11856, an absolute rise of about 3.95 percentage points (from 7.9% to roughly 11.9%) — a 50% relative increase. (b) Raising the "unaligned AI" component from 0.10 to 0.15 in the union-formula table (holding the other seven categories fixed) increases the total existential risk from 0.1784 to 0.224 — a jump of about 4.6 percentage points, again because a union total is highly sensitive to its single largest component. (c) No, not necessarily, and re-running the Monte Carlo simulation with P4's mean shifted to 0.60 (same concentration, so P4's own marginal variance is actually unchanged, since Beta variance ∝ mean×(1−mean) is symmetric around 0.5 and 0.4×0.6 = 0.6×0.4) confirms this: the new sensitivity ranking becomes P3 (42.8%), P4 (29.3%), P5 (19.3%), P2 (7.0%), P1 (6.8%), P6 (1.0%) — P3 now dominates P4, even though P4 is the premise that changed and its own variance is identical to before. This is because a premise's share of a product's total variance depends on the joint structure of all factors together, not on any one factor in isolation — so a sensitivity analysis must be re-run whenever any input estimate changes, never patched by assuming yesterday's ranking still applies.
Think About It
Think about this: How would you explain ai existential risk analysis: evaluating long-term scenarios to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind ai existential risk analysis: evaluating long-term scenarios, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.