The Compute Arms Race: GPU Economics and Sovereign AI

The queue outside the data center

Through most of 2023, a single NVIDIA H100 GPU carried a waitlist measured in months. Cloud providers rationed allocation to their largest, longest-standing customers first. Smaller labs, university groups, and startups, including a wave of Indian AI companies building on top of large language models, found themselves paying spot-market premiums two to three times the listed cloud rate, or simply going without. This was not a supply hiccup in the way a shortage of a commodity chemical is a hiccup. It was the visible symptom of a structural fact: training and serving modern AI models is bounded by the number of specialized chips a country, company, or lab can get its hands on, at what price, and for how many hours.

By March 2024, the Indian government had responded with the IndiaAI Mission, an outlay of ₹10,372 crore (roughly $1.25 billion) that included a dedicated compute pillar: empanelling private data centers and cloud providers to build out GPU capacity, then offering that capacity to domestic startups, researchers, and academic labs at subsidized hourly rates well below open-market cloud pricing. Reports through 2025 put the empanelled capacity at somewhere north of 18,000 GPUs. This chapter is about the economics that make a decision like that necessary, and about what "sovereign" compute actually buys a nation once it has it. Two threads run through everything below: how the cost of training a model is actually computed from first principles, and why owning the chips is a much narrower kind of independence than it sounds.

Why a matrix multiply needs sixteen thousand cores

A transformer, at its arithmetic core, is a long sequence of matrix multiplications: query, key, and value projections, attention score computation, feed-forward layers. Each of these is embarrassingly parallel, in the technical sense: the value at output position (i, j) of a matrix product depends only on one row of the left matrix and one column of the right matrix, and none of the other output positions. A CPU, built for a long pipeline of sequential, branch-heavy instructions with a handful of powerful cores, is a poor fit. A GPU, built from the start for pixel shading (also embarrassingly parallel), has thousands of small arithmetic units and, since the 2017 introduction of Tensor Cores, dedicated hardware that performs a small fused matrix multiply-accumulate in a single instruction rather than issuing separate multiply and add instructions per element.

The NVIDIA H100 SXM, the workhorse chip of this arms race, has 132 streaming multiprocessors, each carrying its own Tensor Cores, for a combined peak throughput of 989 trillion floating-point operations per second in dense BF16 precision, with no sparsity exploited. That number, 989 TFLOPS, is the ceiling. Nothing you do with software makes a single H100 compute faster than that. What software controls is how close to that ceiling you actually get, and that gap is the first place GPU economics gets interesting.

Pricing a training run: FLOPs, tokens, and the compute-optimal frontier

To cost a training run before you run it, you need to know how many floating-point operations training will consume. Jared Kaplan and colleagues at OpenAI, in "Scaling Laws for Neural Language Models" (2020), showed that the total training compute for a dense transformer is well approximated by a simple formula: C ≈ 6ND, where N is the parameter count and D is the number of training tokens. The factor of 6 comes from counting multiply-accumulate operations across the forward pass (roughly 2N FLOPs per token) and the backward pass (roughly 4N FLOPs per token, since gradients must be computed with respect to both activations and weights).

That formula tells you the cost of a run once you already know N and D. It does not tell you the right D for a given N. In 2022, Jordan Hoffmann and colleagues at DeepMind, in "Training Compute-Optimal Large Language Models," showed that most contemporary large models were badly undertrained relative to their parameter count: labs were scaling parameters aggressively while holding training tokens roughly fixed, which wastes compute. Fitting loss curves across many runs, they found the compute-optimal ratio is close to D ≈ 20N, tokens per parameter. Their own model, Chinchilla, at 70 billion parameters trained on 1.4 trillion tokens (exactly the 20:1 ratio), matched or beat Gopher, a 280-billion-parameter model from the same lab trained on far fewer tokens per parameter, using the same total training compute. Bigger was not better; better-proportioned was better.

This ratio is the first number you need for a cost estimate: pick a target parameter count, multiply by 20 to get the compute-optimal token budget, then apply C ≈ 6ND.

Worked example: costing a 7-billion-parameter model

Suppose you want to compute-optimally train a 7-billion-parameter model.

N = 7 × 10⁹. Compute-optimal tokens: D = 20N = 1.4 × 10¹¹. Training FLOPs: C = 6ND = 6 × (7 × 10⁹) × (1.4 × 10¹¹) = 5.88 × 10²¹ FLOPs.

Now convert FLOPs into GPU-hours, which is what you actually pay for. An H100's 989 TFLOPS peak is never fully realized in practice: memory stalls while waiting for the next batch of activations, communication between GPUs during the backward pass, and imperfect kernel scheduling all eat into it. Practitioners call the fraction of peak actually achieved Model FLOPs Utilization, or MFU, a term that entered common use through Google's PaLM paper (Chowdhery et al., 2022). Well-optimized large-scale training runs typically report an MFU between 30% and 50%. We will use 40%.

Effective per-GPU throughput: 989 × 10¹² × 0.40 = 3.956 × 10¹⁴ FLOPs per second. Time on a single GPU: 5.88 × 10²¹ / 3.956 × 10¹⁴ ≈ 1.4863 × 10⁷ seconds ≈ 4,128.7 hours. That single number, 4,128.7 GPU-hours, is also (by definition, for one GPU) the total compute budget. On a real cluster of many GPUs, communication overhead between devices means the cluster never hits 100% of the sum of individual peak rates; we will assume 90% scaling efficiency, a reasonable figure for a few hundred well-networked GPUs. Total GPU-hours actually consumed: 4,128.7 / 0.90 ≈ 4,587.5.

Here is the same arithmetic as runnable code, so you can trace every step yourself:

def training_flops(n_params, tokens_per_param=20):
    d_tokens = tokens_per_param * n_params
    return 6 * n_params * d_tokens

def gpu_hours(flops, peak_flops_per_s=989e12, mfu=0.40, scaling_efficiency=0.90):
    effective_per_gpu = peak_flops_per_s * mfu
    ideal_gpu_hours = flops / effective_per_gpu / 3600
    return ideal_gpu_hours / scaling_efficiency

def wallclock_hours(total_gpu_hours, n_gpus):
    return total_gpu_hours / n_gpus

def training_cost_usd(total_gpu_hours, price_per_gpu_hour=2.50):
    return total_gpu_hours * price_per_gpu_hour

n = 7e9
flops = training_flops(n)
gh = gpu_hours(flops)
wallclock = wallclock_hours(gh, 512)
cost = training_cost_usd(gh)

print(f"FLOPs: {flops:.3e}")
print(f"GPU-hours: {gh:.0f}")
print(f"Wall-clock on 512 GPUs: {wallclock:.2f} hours")
print(f"Cost: ${cost:,.0f}")

# Output:
# FLOPs: 5.880e+21
# GPU-hours: 4587
# Wall-clock on 512 GPUs: 8.96 hours
# Cost: $11,469

We used an illustrative on-demand rate of $2.50 per GPU-hour; real cloud H100 pricing through 2024-2025 ranged roughly $2 to $5 depending on provider and commitment length, so treat the dollar figure as an order-of-magnitude anchor, not a quote. Two things about this result are worth sitting with. First, $11,469 is a strikingly small number for "training a 7-billion-parameter model," and that is because it is an idealized compute-only floor: it assumes one clean run with no failed hyperparameter searches, no re-runs after a loss spike, no data-cleaning pipeline, and no salaried researchers. Real reported training costs for models in this class run five to ten times higher once those are included. Second, notice that wall-clock time (8.96 hours) and total cost ($11,469) are governed by different variables: GPU count changes only how fast the fixed amount of work gets done, not how much work there is.

Where the interconnect eats your efficiency

That 90% scaling efficiency assumption is not free; it is bought by careful placement of work across the memory hierarchy, and it is why a "training cluster" is not just a pile of GPUs. There is a bandwidth cliff between every tier of the hierarchy. Inside a single H100, the 80 GB of HBM3 memory feeds the compute cores at 3.35 terabytes per second. Between GPUs on the same node, NVIDIA's fourth-generation NVLink, routed through an NVSwitch, sustains 900 gigabytes per second per GPU, roughly a quarter of on-chip bandwidth. Between nodes, an InfiniBand NDR fabric running at 400 gigabits per second (50 gigabytes per second) is the norm, another order of magnitude down.

This is not an incidental engineering detail; it dictates how you split a training job. Tensor parallelism, which splits an individual matrix multiplication across GPUs and requires an all-to-all exchange of partial results after nearly every layer, needs the fast NVLink tier and is essentially never spread across nodes. Data parallelism and pipeline parallelism, which exchange much coarser-grained information (a full gradient sync per step, or one activation tensor between pipeline stages), tolerate the slower InfiniBand tier and are what stitches many nodes into one pod. Get this split wrong, put a tensor-parallel group across an InfiniBand link instead of an NVLink one, and MFU can collapse from 40% to well under 10%, because the GPUs spend most of their time idle, waiting on the network rather than computing.

Anatomy of a training cluster

Sovereign AI: three pillars, one chokepoint

"Sovereign AI" gets used loosely, but it decomposes into three genuinely separable pillars. The first is compute: physical GPU or accelerator capacity that a nation can allocate on its own terms rather than at the discretion of a foreign hyperscaler's pricing and priority queue. The second is data: the legal and infrastructural ability to keep sensitive data (health records, government documents, financial transactions) inside national jurisdiction during training and inference, which matters both for privacy law and for not handing a foreign company a durable advantage built on your citizens' data. The third is the model and talent layer: actually having researchers who can pretrain, fine-tune, and evaluate models, rather than only ever consuming an API from abroad.

The compute pillar is the one most nations can move on fastest, because it does not require inventing a chip industry from scratch, only buying and hosting hardware domestically or via empanelled providers, which is exactly what the IndiaAI Mission's compute component does. It is also the pillar most exposed to a fact the diagram above makes concrete: as of the mid-2020s, essentially every leading-edge AI accelerator (NVIDIA's H100 and its successors, AMD's MI300-class chips) is fabricated at TSMC in Taiwan on a handful of advanced process nodes. There is no second source at comparable yield and density. On top of that single-foundry dependency sits a second, policy-layer chokepoint: United States export control rules, first tightened substantially in October 2022 and again in October 2023, restrict which countries may legally purchase chips above certain performance-density thresholds, initially aimed at China but with ripple effects on export licensing for other markets too. A nation can build data centers, subsidize electricity, and empanel every domestic cloud provider it has, and it still cannot buy chips that a foreign export-licensing regime declines to release.

Different nations have leaned into this reality differently. The UAE's G42 and Saudi Arabia's Humain have pursued large direct compute deals with US chipmakers, betting that political alignment secures license approval. The European Union's EuroHPC initiative funds domestic "AI factories" hosted on European soil, aimed primarily at the data and jurisdiction pillar rather than fabrication independence. China, cut off from the newest export-eligible chips, has pushed hardest on the fabrication pillar itself, scaling domestic accelerators such as Huawei's Ascend line, at a real performance cost relative to the frontier, precisely because the fabrication chokepoint cannot be subsidized around. India's IndiaAI Mission, by contrast, is squarely a compute-pillar and price-access strategy: it makes GPU-hours cheaper and more available to domestic researchers and startups, without changing who fabricates the chip or who controls the export license.

Common misconception: physical location is not sovereignty

A student encountering this topic for the first time will often reason: "once enough GPUs are installed in data centers on Indian soil, India's AI is sovereign." This conflates location with control, and it is wrong for a specific, checkable reason. A GPU installed in an Indian data center is still a TSMC-fabricated NVIDIA chip whose export to India was itself subject to a foreign licensing decision, running firmware and drivers from a foreign vendor, inside a software stack (CUDA and its libraries) that the rest of the world's training code is written against. Full-stack sovereignty would additionally require some real measure of control or diversification over fabrication (or at minimum, insulation from a single foundry and a single licensing regime), the software and driver layer the hardware depends on, and the data and model-training talent layer described above. Buying GPU-hours cheaply, which is what a compute subsidy like IndiaAI's does, is a genuinely useful policy: it lowers the cost of experimentation for domestic researchers and reduces dependence on foreign cloud pricing power. But it leaves the fabrication chokepoint and the export-license chokepoint exactly where they were. Cheaper access to a foreign-controlled supply chain is not the same thing as controlling the supply chain.

Active recall

Attempt each question before reading its answer.

1. Why does distributed training scale sub-linearly with GPU count even when raw compute (FLOPs) scales perfectly linearly with GPU count?
2. Using the same method as the worked example, find the compute-optimal training FLOPs for a 13-billion-parameter model, and estimate how long a single H100 GPU (no multi-GPU overhead, since there is only one device) would take to run that workload at 40% MFU.
3. Take the original 7B worked example (512 GPUs, 40% MFU, 90% scaling efficiency, $2.50/GPU-hour, 4,587 GPU-hours, 8.96 hours wall-clock, $11,469). Now suppose you switch to a Mixture-of-Experts architecture that drops MFU to 25% because of extra all-to-all communication, and you only have 256 GPUs available instead of 512 (scaling efficiency stays 90%). Recompute total GPU-hours, wall-clock time, and cost, and explain which of the three changed for a surprising reason.
4. Give two distinct economic rationales for a government subsidizing GPU-hour pricing for domestic AI researchers and startups, and one risk of doing so.
5. A classmate says: "once we have enough GPUs installed in Indian data centers, India will be fully sovereign in AI." What's wrong with this claim, and what three additional layers would need meaningful domestic control for full-stack sovereignty?
6. An H100 costs roughly $30,000 on its own. A full 8-GPU DGX-class node costs roughly $340,000. What percentage of the node's price is overhead beyond the raw cost of the 8 GPUs, and what does that overhead pay for?

Answers

1. The bottleneck is interconnect bandwidth, not compute. Training requires GPUs to exchange gradients, activations, or partial matrix products at every layer or every step. NVLink and InfiniBand bandwidth do not scale up as you add more GPUs to a job; they stay fixed per link. As GPU count grows, the fraction of wall-clock time spent waiting on communication rather than computing grows too, which is exactly why we modeled it as a fixed "scaling efficiency" (90% in our example) rather than assuming 100%. At cluster sizes in the thousands of GPUs, real-world scaling efficiency commonly drops well below 90%.

2. N = 13 × 10⁹, D = 20N = 2.6 × 10¹¹, FLOPs = 6ND = 6 × (13 × 10⁹) × (2.6 × 10¹¹) = 2.028 × 10²² FLOPs. At 40% MFU, effective single-GPU throughput is 989 × 10¹² × 0.40 = 3.956 × 10¹⁴ FLOPs/s. Time = 2.028 × 10²² / 3.956 × 10¹⁴ ≈ 5.126 × 10⁷ seconds ≈ 14,240 hours ≈ 593 days ≈ 1.63 years. This is roughly 3.45 times the single-GPU time for the 7B model (4,128.7 hours), consistent with FLOPs scaling roughly with N × D = N × 20N = 20N², so doubling N should almost quadruple FLOPs; here N scaled by 13/7 ≈ 1.857, so FLOPs should scale by about 1.857² ≈ 3.45, which matches. This is precisely why nobody trains a 13B model, let alone a frontier model orders of magnitude larger, on a single GPU.

3. New GPU-hours: because GPU-hours = FLOPs / (peak × MFU × scaling_efficiency), and neither FLOPs, peak, nor scaling_efficiency changed, GPU-hours scales inversely with MFU alone: 4,587.5 × (0.40/0.25) = 4,587.5 × 1.6 ≈ 7,340 GPU-hours. Cross-checking by direct computation: single-GPU time at 25% MFU is 5.88 × 10²¹ / (989 × 10¹² × 0.25) ≈ 6,606 hours; divided by 0.90 scaling efficiency gives ≈ 7,340 GPU-hours, matching. Wall-clock time = GPU-hours / GPU count = 7,340 / 256 ≈ 28.67 hours (about 1.2 days), versus 8.96 hours originally, a 3.2x increase driven by both fewer GPUs (2x) and lower MFU (1.6x) multiplying together. Cost = 7,340 × $2.50 ≈ $18,350, versus $11,469 originally, exactly a 1.6x increase. The surprising result: cost rose by exactly the same 1.6x factor as GPU-hours, driven entirely by the MFU drop; halving the GPU count from 512 to 256 changed the wall-clock time but did not change the total cost at all, because the total amount of work (and therefore total GPU-hours purchased) is independent of how many GPUs you spread it across, only how fast it finishes.

4. Two rationales: (a) AI capability-building has positive externalities, spillovers into talent formation, downstream startups, and public research, that a private actor's own return on a GPU-hour does not capture, which is the standard justification for public R&D subsidy in any capital-intensive frontier technology. (b) It functions as a geopolitical and economic hedge: reducing dependence on foreign hyperscalers who set prices and allocation priority unilaterally, similar in logic to subsidizing domestic semiconductor fabs. One risk: underpriced compute without a market signal can be misallocated toward low-value usage or captured by well-connected incumbents rather than the researchers with the highest marginal return, a classic subsidy-allocation problem.

5. Physical location does not equal control. The GPUs are still foreign-designed, foreign-fabricated (via the single TSMC chokepoint), and their export was itself subject to a foreign government's licensing decision; the software stack (CUDA, drivers, firmware) is likewise foreign-controlled. The three additional layers: fabrication (or credible diversification/insulation from a single foundry and licensing regime), the software/driver stack the hardware depends on, and the data-and-talent layer, meaningful domestic control over training data governance and the researchers capable of building and evaluating models, not merely renting API access to models built elsewhere.

6. Raw cost of 8 GPUs: 8 × $30,000 = $240,000. Node price $340,000 minus $240,000 = $100,000 of overhead. $100,000 / $340,000 ≈ 29.4%. That overhead pays for the NVSwitch fabric that gives every GPU 900 GB/s of all-to-all bandwidth to every other GPU in the node, plus the chassis, power delivery, and cooling needed to keep eight 700-watt-class chips running at sustained load, a concrete illustration that the interconnect is not a minor add-on but roughly a third of what you are actually paying for.

Think About It

Think about this: How would you explain the compute arms race: gpu economics and sovereign ai to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind the compute arms race: gpu economics and sovereign ai, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.