Neural Radiance Fields (NeRF): 3D from 2D

A heritage-tech team scanning the Vitthala Temple complex at Hampi for a museum kiosk captures 400 photographs, trains a neural radiance field overnight, and gets back a function that reproduces every camera pose they shot with startling fidelity. Then they try to ship it. Rendering one new viewpoint at a modest 800×800 resolution takes roughly 30 seconds on the workstation that trained it, because computing a single pixel means marching a ray through the volume, invoking a deep multilayer perceptron at somewhere between 64 and 192 points along that one ray, and only then combining the results into a colour. A kiosk needs to redraw the screen 30 to 60 times a second as a visitor drags a finger across the glass. At 30 seconds a frame, across roughly 640,000 pixels per frame, the system is not performing real-time rendering; it is running tens of millions of network evaluations to produce a single still image. Understanding exactly why that number is so large, and what changed in the three years after the first NeRF paper to bring it down to milliseconds, requires opening the one equation that turns a radiance field into a pixel, and then looking at what two very different engineering fixes did to that equation's cost.

The earlier chapter on neural radiance fields introduced the central object: a function F_θ mapping a 3D point and a viewing direction to a colour and a volume density, (x, d) ↦ (c, σ). This chapter opens up what happens between that function and the pixel a viewer actually sees, derives why the raw (x, y, z) coordinates must be transformed before they ever reach the network, and closes by comparing two production-grade fixes for the resulting cost: keeping the network but replacing its input encoding (Instant-NGP), and discarding the network at render time entirely in favour of an explicit, rasterized representation (3D Gaussian Splatting).

From a continuous field to one pixel: the volume rendering integral

A camera ray through pixel p is parameterised as r(t) = o + td, origin o, unit direction d, t ranging over [t_n, t_f]. Treat the scene as participating media, the way physically based renderers treat smoke or fog: σ(x) is the differential probability per unit length that a photon travelling through point x is absorbed or scattered there, and c(x, d) is the colour emitted from that point toward the camera. Define the accumulated transmittance

T(t) = exp( − ∫_tn^t σ(r(s)) ds )

as the probability the ray survives, unabsorbed, from t_n to t. The expected colour returned along the whole ray is then

C(r) = ∫_tn^t_f T(t) σ(r(t)) c(r(t), d) dt

Read T(t)σ(r(t))dt as the probability the ray terminates in the infinitesimal interval [t, t+dt]: it must have survived up to t (probability T(t)), then get absorbed in the next dt (probability σ(r(t))dt). Weighting the local colour by this termination probability and integrating over t gives exactly an expectation over "where did this ray stop." This is the equation the sibling chapter's function feeds into; it is decades older than NeRF itself, first written in this exact form for computer graphics by Nelson Max in 1995 for direct volume rendering of scanned medical and scientific data, long before anyone used a neural network to supply σ and c.

Because σ and c here are arbitrary outputs of an MLP, the integral has no closed form and must be approximated numerically. Split [t_n, t_f] into N intervals at sample points t_i with width δ_i = t_i+1 − t_i, and assume σ and c are constant within each interval (piecewise-constant quadrature). Within interval i, the probability the ray terminates somewhere inside it, given it survived to reach it, is

α_i = 1 − exp(−σ_iδ_i)

which is precisely the "alpha" of classical alpha compositing: it depends jointly on how dense the interval is (σ_i) and how much of it the ray traverses (δ_i), not on σ_i alone. The cumulative transmittance up to sample i telescopes into a product of survivals of every earlier interval,

T_i = exp(−Σ_j<i σ_jδ_j) = ∏_j<i (1 − α_j)

and the whole ray's colour becomes the discrete sum Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng use in their 2020 NeRF paper (ECCV 2020):

Ĉ(r) = Σ_i=1^N T_i α_i c_i

This is front-to-back "over" compositing of N semi-transparent layers, the same operator used to stack layers in image-editing software. Each weight w_i = T_iα_i is non-negative, the weights sum to at most 1, and whatever probability mass is not accounted for by any sample, T_N+1 = ∏_i(1−α_i), is the probability the ray passed through the entire sampled region untouched, effectively "leftover" for a background or for space the sampling missed. A real NeRF ray uses 64 coarse samples plus roughly 128 additional samples placed by an importance-resampling pass that concentrates points where the coarse pass found high weight; the worked example below uses 5 samples, small enough to trace by hand, at the cost of being coarser than production quality.

Worked example: compositing five samples along a ray

A ray passes near the edge of a solid object. The network has been queried at five points, uniform spacing δ_i = 0.2 for every i, returning:

σ = [0.10, 0.20, 0.50, 4.00, 0.05]    c = [0.90, 0.85, 0.80, 0.90, 0.30] (grayscale intensity, 0 to 1, for a single channel; RGB just repeats this per channel)

Compute α_i = 1 − exp(−σ_i·0.2) for each: α = [0.0198, 0.0392, 0.0952, 0.5507, 0.0100]. Note sample 4, sitting on the object's surface, has by far the largest α because its density is an order of magnitude higher than its neighbours; the others represent near-empty space the ray grazes on the way in and out.

Accumulate transmittance as a running product of survivals: T₁ = 1 (nothing has been passed through yet), T₂ = 0.9802, T₃ = T₂(1−0.0392) = 0.9418, T₄ = T₃(1−0.0952) = 0.8521, T₅ = T₄(1−0.5507) = 0.3829. Notice how much T drops crossing sample 4: passing through the dense interval costs the ray more than half its remaining transmittance in one step.

Weights w_i = T_iα_i: [0.0198, 0.0384, 0.0896, 0.4693, 0.0038], summing to 0.6209; the remaining 0.3791 of transmittance escapes past sample 5 to whatever lies beyond (background). The composited colour:

C = Σ w_ic_i = (0.0198)(0.90) + (0.0384)(0.85) + (0.0896)(0.80) + (0.4693)(0.90) + (0.0038)(0.30) ≈ 0.546

Sample 4 alone contributes 0.4223 of the 0.546 total, about 77% of the visible colour, despite being one of five samples and despite α₄ being only 0.55, not 1. This is the traced-by-hand version of the following code, which reproduces every number above:

import math

def volume_render(sigmas, deltas, colors):
    alphas = [1 - math.exp(-s * d) for s, d in zip(sigmas, deltas)]
    T = 1.0
    weights = []
    for a in alphas:
        w = T * a
        weights.append(w)
        T = T * (1 - a)          # T is now T_{i+1}
    C = sum(w * c for w, c in zip(weights, colors))
    return alphas, weights, C, T  # T left over is the escaped transmittance

sigmas = [0.10, 0.20, 0.50, 4.00, 0.05]
deltas = [0.2] * 5
colors = [0.90, 0.85, 0.80, 0.90, 0.30]

alphas, weights, C, leftover = volume_render(sigmas, deltas, colors)
print(round(C, 3), round(leftover, 3))
# prints: 0.546 0.379

A student who thinks a radiance field works like a mesh renderer, where a ray hits the first opaque surface and stops, will expect the answer to be governed entirely by whichever sample has the highest density, with everything before and after irrelevant. The numbers above refute that directly: samples 1, 2, 3, and 5, none of them remotely solid, still contribute 0.123 of the final 0.546, about 22-23% of the visible colour, and the "solid" sample 4 has α₄ = 0.55, not 1, precisely because δ₄ = 0.2 is not infinitesimally small. There is no hard surface anywhere in this computation; there is a continuous density field, discretely integrated, and the appearance of a sharp boundary is an emergent property of one interval's σδ product being much larger than its neighbours', not a special "surface-found" branch in the algorithm.

Why raw coordinates fail: positional encoding and spectral bias

Feed (x, y, z) directly into an eight-layer MLP with smooth activations and it will not learn a radiance field with sharp edges or fine texture, no matter how long it trains. Rahaman et al. (ICML 2019) showed empirically that gradient descent on such networks fits low-frequency components of a target function first and fastest; Tancik et al. (NeurIPS 2020) formalised why, using the neural tangent kernel (NTK). A wide MLP's behaviour under gradient descent is governed by a kernel whose eigenvalues decay rapidly with frequency, so high-frequency target functions correspond to the kernel's smallest eigenvalues and are learned only after impractically many steps, if at all. This is spectral bias: not a bug in a particular implementation, but a structural property of ordinary coordinate-based MLPs.

NeRF's fix is to never let the raw coordinate reach the network. Each scalar coordinate p is mapped through a fixed bank of sinusoids at geometrically increasing frequencies before the first layer:

γ(p) = ( sin(2⁰πp), cos(2⁰πp), sin(2¹πp), cos(2¹πp), …, sin(2^L−1πp), cos(2^L−1πp) )

applied independently to x, y, z and concatenated (the paper uses L=10 for position, turning 3 raw numbers into 60, and L=4 for viewing direction, turning 3 numbers into 24). Tancik et al.'s NTK analysis shows this transforms the original decaying-spectrum kernel into an approximately shift-invariant one whose effective bandwidth is set by the highest encoded frequency, 2^L−1. A larger L lets the same MLP represent finer spatial detail, at the cost of a wider first layer and more compute per query.

The mechanism is visible in a direct computation. Take L=3, so γ(p) has six components, and evaluate at p=0.5:

import math

def positional_encode(p, L):
    out = []
    for k in range(L):
        freq = (2 ** k) * math.pi
        out.append(math.sin(freq * p))
        out.append(math.cos(freq * p))
    return out

print([round(v, 3) for v in positional_encode(0.5, 3)])
# [1.0, 0.0, 0.0, -1.0, -0.0, 1.0]

γ(0.5) = [1, 0, 0, −1, 0, 1] exactly, since π/2, π, and 2π are all lattice points of sine and cosine. Nudge the input by a tiny amount, to p=0.51, and evaluate the same function: the highest-frequency pair, driven by 2²πp = 4πp, moves from (0, 1) to (0.1253, 0.9921), a change of 0.1253 in the sine component for an input change of only 0.01. That is a 12.53× amplification, and it is not a numerical accident: by the chain rule, d/dp sin(2^kπp) = 2^kπcos(2^kπp), which at p=0.5 evaluates to 2²π·cos(2π) = 4π ≈ 12.566, matching the measured amplification. This is the entire mechanism in one number: the highest encoded channel has a local slope of roughly 2^L−1π with respect to the raw coordinate, so a spatial feature that would require an enormous, hard-to-learn slope from a raw-coordinate MLP can instead be represented by an ordinary, easy-to-learn slope acting on an input that itself already varies steeply. With L=10, the top channel's slope is roughly 2⁹π ≈ 1608, letting the network resolve spatial variation on length scales roughly 500 times finer than L=1 would allow, at the cost of 19 extra input dimensions per coordinate axis. (This is unrelated to the sinusoidal positional encoding used in Transformers, which shares the sin/cos form but disambiguates token order in a sequence; here the job is reshaping a continuous spatial coordinate, not marking discrete position.)

Two ways to stop paying for slowness

Positional encoding solves the frequency problem but does nothing about cost: every one of the 64 to 192 samples per ray still triggers a full forward pass through an eight-layer, 256-wide MLP. Two papers, both after the original NeRF, attacked this cost from opposite directions.

Müller, Evans, Schied, and Keller (Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, ACM Transactions on Graphics 41(4), SIGGRAPH 2022) keep the network but replace the fixed sinusoidal encoding with a small set of trainable feature vectors. Space is covered by L resolution levels of grids, from coarse to fine (the paper uses 16 levels). For a query point, each level finds the surrounding voxel corners, hashes each corner's integer coordinates into a fixed-size table of T entries via a spatial hash h(x) = (x₁π₁ ⊕ x₂π₂ ⊕ x₃π₃) mod T, with large primes π₁=1, π₂=2654435761, π₃=805459861 (the technique traces to Teschner et al.'s 2003 spatial hashing for collision detection), looks up an F-dimensional learned feature per corner, and trilinearly interpolates across the 8 corners. Concatenating the interpolated feature across all L levels gives the encoded input, which feeds a network with only two hidden layers of 64 units, versus the original's eight layers of 256. Coarse levels are cheap to look up and generalise fast; fine levels, backed by learned rather than fixed features, capture detail that a fixed sinusoidal basis of the same dimensionality cannot. Most of the representational work moves from sequential MLP depth into parallel table lookups, which is why both training and inference speed up by orders of magnitude on the same hardware.

Kerbl, Kopanas, Leimkühler, and Drettakis (3D Gaussian Splatting for Real-Time Radiance Field Rendering, ACM Transactions on Graphics 42(4), SIGGRAPH 2023) go further and remove the network from the render path altogether. The scene is represented explicitly as millions of 3D anisotropic Gaussians, each carrying a centre μ, a covariance (shape and orientation, via a scale vector and a rotation quaternion), an opacity, and view-dependent colour stored as spherical-harmonic coefficients. Starting from a sparse structure-from-motion point cloud (typically produced by COLMAP), training optimises every Gaussian's parameters by gradient descent against the same photometric loss NeRF uses, interleaved with an adaptive density-control step that clones Gaussians where gradients are large and under-reconstructed, splits over-large ones, and prunes near-transparent ones. Rendering a novel view does not march rays through anything: the Gaussians are projected onto the image plane and rasterized, tile by tile, sorted by depth and alpha-composited per pixel, using the GPU's ordinary rasterization pipeline rather than per-sample network inference. There is no σ(x) or c(x,d) function to query at render time at all; the volume-rendering integral this chapter derived has been replaced by an explicit, differentiable rasterizer operating on a fixed set of primitives.

A real training-time comparison

Method            Year  Representation                Training time (single high-end GPU)   Rendering
NeRF              2020  implicit MLP + ray marching    roughly 1-2 days                       ~30 s per 800x800 frame
Instant-NGP       2022  MLP + multiresolution hash      seconds to a few minutes for           real-time (thousands of fps
                        grid, still ray-marched          comparable quality                     on small scenes)
3D Gaussian       2023  explicit anisotropic            roughly 30-45 minutes                  real-time, commonly well
Splatting               Gaussians, rasterized                                                   over 100 fps at 1080p

The qualitative shift matters more than any single number. Instant-NGP is still, mechanically, the same rendering equation this chapter derived, marched the same way, with a smaller MLP fed a richer, learned encoding instead of a fixed sinusoidal one; its speedup comes from moving work out of sequential MLP depth into parallel hash lookups. 3D Gaussian Splatting is not a faster version of that pipeline; it abandons implicit ray marching entirely for an explicit primitive set that a rasterizer, not a renderer built around per-sample network calls, can draw. That distinction is also the chapter's one deliberate misconception to correct.

Common misconception

Having encountered "Instant-NGP" and "3D Gaussian Splatting" in the same breath, as fast successors to the original NeRF, it is natural to assume they are the same idea wearing different names: two speed tricks bolted onto the same neural radiance field. They are not. Instant-NGP is still an implicit representation: a query point still needs an encoding lookup followed by an MLP forward pass, and an image is still produced by marching rays and applying the T_iα_ic_i compositing formula derived above, just with a cheaper encoding and a smaller network at each of those hundred-plus samples per ray. 3D Gaussian Splatting has no encoding, no MLP, and no ray marching at render time; colour and opacity live directly on a fixed list of explicit 3D primitives, and an image is produced by projecting and rasterizing them, the same family of operation a graphics pipeline uses to draw a triangle mesh. The practical consequence: an Instant-NGP scene can still, in principle, be queried at an arbitrary continuous 3D point with no primitive nearby, because the field is defined everywhere; a Gaussian-splatting scene only has appearance where a Gaussian was placed, and querying "empty" space returns nothing meaningful. Confusing the two obscures why one still needs a GPU with strong random-access memory-lookup throughput and a neural inference stack, while the other can be shipped as, essentially, a very large point cloud and drawn by ordinary rasterization hardware.

Active recall

Attempt each question before reading its answer.

Q1. Write the continuous volume rendering integral and state, in one sentence each, what T(t) and σ(r(t))dt represent.

Q2. In the five-sample worked example, sample 4 has α₄ = 0.5507, meaning the ray has roughly a 55% chance of terminating in that one interval given it survived to reach it. Yet its weight is only 0.4693, not 0.5507. Where did the difference go?

Q3. Compute γ(p) for p = 0.25 with L = 2 (four output numbers). Show each sin and cos evaluation.

Q4. Starting from the original worked example (σ = [0.10, 0.20, 0.50, 4.00, 0.05], δ_i = 0.2 for all i, giving C ≈ 0.546), suppose the sampling is made twice as fine, δ_i = 0.1 for all i, with the same five σ values. Recompute every α_i, every T_i, every weight, the new C, and the new leftover transmittance. Is the final colour close to the original 0.546?

Q5. A colleague says: "Instant-NGP and 3D Gaussian Splatting are the same speed trick, just with different names." What is wrong with this statement, and which of the two still requires a neural network to be evaluated at render time?

Q6. A friend claims NeRF works by ray-casting to find the exact point where a ray hits a solid surface, the way a game engine's z-buffer does, and that σ is just a yes/no occupancy flag. Using the numbers from the worked example, explain concretely why this is wrong.

A1. C(r) = ∫_tn^t_f T(t)σ(r(t))c(r(t),d) dt. T(t) = exp(−∫_tn^tσ(r(s))ds) is the probability the ray has travelled from t_n to t without being absorbed. σ(r(t))dt is the probability of being absorbed in the next infinitesimal step once the ray has reached t; their product, integrated over t, is an expectation of colour over the ray's termination point.

A2. Not all of α₄ converts into visible weight, because w₄ = T₄α₄, and T₄ = 0.8521, not 1. Before the ray even reaches sample 4, it has already lost about 14.8% of its transmittance to samples 1 through 3 (mostly nearly-empty space, but not perfectly empty). 0.8521 × 0.5507 = 0.4693. The missing 0.5507 − 0.4693 = 0.0814 is transmittance that had already been "spent" before sample 4, so it was never available to be captured there. This is why order along the ray matters, not just each interval's own α.

A3. k=0: freq = 1·π = π. sin(π·0.25) = sin(π/4) = 0.7071, cos(π/4) = 0.7071. k=1: freq = 2π. sin(2π·0.25) = sin(π/2) = 1.0, cos(π/2) = 0.0. γ(0.25) = [0.7071, 0.7071, 1.0, 0.0].

A4. Recomputing α_i = 1−exp(−σ_i·0.1): α = [0.0100, 0.0198, 0.0488, 0.3297, 0.0050]. Transmittance: T = [1.0, 0.9900, 0.9704, 0.9231, 0.6188]. Weights: w = [0.0100, 0.0196, 0.0473, 0.3043, 0.0031], summing to 0.3843, leaving 0.6157 of transmittance unaccounted for (versus 0.3791 before). New colour: C = 0.0100(0.90) + 0.0196(0.85) + 0.0473(0.80) + 0.3043(0.90) + 0.0031(0.30) ≈ 0.338. This is far from the original 0.546, and the reason is that δ is not a neutral resolution knob: it multiplies directly into the exponent alongside σ, so halving it while holding σ fixed at each sample point halves how much physical "material" each sample is deemed to represent. The steepest-density sample's contribution collapses hardest, α₄ falls from 0.5507 to 0.3297, because that is exactly where the interval width mattered most. A coarse 5-sample discretisation is not a scaled-down, equally-accurate version of a fine one; this is precisely why production NeRFs use 64 to 192 samples with hierarchical resampling concentrated where density is high, to keep the piecewise-constant approximation close to the true continuous integral.

A5. They differ in what kind of object is being rendered, not just in speed. Instant-NGP keeps the implicit radiance field: at render time, every sample along every ray still triggers a hash-grid lookup followed by a real MLP forward pass, and the resulting σ, c still get composited with the T_iα_ic_i formula from this chapter. 3D Gaussian Splatting has no MLP at render time at all; colour and opacity are stored directly on millions of explicit 3D Gaussians and produced by projecting and rasterizing them, the same category of operation used to draw a mesh. Instant-NGP is the one that still needs a neural network evaluated per sample at render time.

A6. In the worked example, the "solid" sample (index 4, σ=4.00) has α₄ = 0.5507, not 1, so even the densest sample does not deterministically stop the ray; it only has roughly even odds of doing so within its interval. The other four samples, all far less dense, still contribute 0.123 out of the total 0.546, about 22-23% of the final colour. A z-buffer ray-cast would return the colour of exactly one surface point and nothing else; this computation blends five points with continuously varying weights and still leaves 0.379 of transmittance unaccounted for, attributable to whatever lies further along the ray. There is no occupancy flag anywhere in the equation, only a continuous density that determines, probabilistically, how much of each sample's colour survives into the final pixel.

Think About It

Think about this: How would you explain neural radiance fields (nerf): 3d from 2d to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where neural radiance fields (nerf): 3d from 2d is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting neural radiance fields (nerf): 3d from 2d to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind neural radiance fields (nerf): 3d from 2d, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.