NeRF: Neural Radiance Fields

A bronze idol that ordinary 3D scanning cannot handle

Suppose a digital-heritage team wants to put a museum's Chola-era bronze Nataraja on a virtual tour: walk around it, zoom into the anklets, see it from angles no visitor is normally allowed. They shoot 80 photographs on a phone, circling the statue. The standard pipeline is photogrammetry: run structure-from-motion to find matching feature points across photos, triangulate a dense point cloud, then mesh it. This works well on a brick wall or a stone temple facade. It falls apart on polished bronze.

The reason is a hidden assumption. Multi-view stereo matching assumes that a single physical point on the surface looks the same color from every camera position (the Lambertian, matte-surface assumption). Bronze does not obey this. A specular highlight on the statue's shoulder slides to a different location on the surface as the photographer moves, because it is a reflection of the museum lights, not a fixed pigment. Feature matching sees the "same" surface patch reporting different colors in different photos, concludes the match is unreliable, and either discards it (leaving a hole in the mesh) or triangulates it wrong (leaving a spike of noise). The curved, reflective regions, exactly the parts a viewer most wants to see clearly, are the parts classical reconstruction handles worst.

Neural Radiance Fields, introduced by Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng at ECCV 2020, sidestep the problem by never trying to match points across photos at all. Instead of asking "which pixel in photo 12 corresponds to which pixel in photo 47," a NeRF asks a single question repeatedly: "if I stood at this exact 3D location and looked in this exact direction, what color and how much opacity would I see?" It learns to answer that question for every point in space and every viewing direction simultaneously, using nothing but the 80 photographs and their camera poses. Because the answer is allowed to depend on viewing direction, a moving specular highlight is not a bug to be resolved away, it is exactly the kind of view-dependent signal the model is built to represent.

The scene as a function, not a mesh

A NeRF represents a scene as one continuous function F_Θ, implemented as a multilayer perceptron with weights Θ:

F_Θ : (x, y, z, θ, φ) → (r, g, b, σ)

The five inputs are a 3D position (x, y, z) and a viewing direction expressed as two angles (θ, φ) (equivalently a unit vector d). The four outputs are an emitted RGB color and a scalar σ called volume density. Read σ(x, y, z) as the differential probability that a ray terminates at an infinitesimal particle located at that point, the same quantity used in classical volume rendering for smoke, fog, or fire. High σ means "there is solid matter here, a ray is very likely to be absorbed or scattered." Low σ means "this is empty air, a ray passes through almost unaffected."

Two design choices in this function are load-bearing and worth stating explicitly. First, σ is a function of position only, not of viewing direction. This is a physical constraint, not a simplification for convenience: whether a photon can pass through a given point in space is a geometric fact about that point, it cannot depend on which way a camera happens to be looking at it. Second, color is allowed to depend on direction, because the light actually reflected or emitted toward a camera does depend on the angle of observation, this is precisely how specular highlights, sheen on bronze, and glare on glass work. The network architecture enforces this asymmetry directly: position alone is fed through most of the network to produce σ and an intermediate feature vector, and only at the final layers is the viewing direction concatenated in to produce the direction-dependent RGB. A skip connection re-injects the positionally encoded input partway through the eight-layer stack, so the geometric signal does not wash out before reaching the density output.

Nothing here is a voxel grid, a point cloud, or a mesh. The "3D model" is entirely encoded in the numerical weights of the MLP. To find out what color and density exist at a particular point, you do not look anything up in a stored array, you run a forward pass of the network. This is the meaning of "implicit, coordinate-based representation": the scene exists only as the function's behavior, evaluated on demand.

Turning density into a pixel: the volume rendering equation

Knowing (r, g, b, σ) at every point does not by itself give a photograph. To render one pixel, cast a ray r(t) = o + t·d from the camera center o through that pixel in direction d, and integrate what the ray accumulates between a near bound t_n and a far bound t_f:

C(r) = ∫ T(t) σ(r(t)) c(r(t), d) dt,  t from t_n to t_f
where T(t) = exp( -∫ σ(r(s)) ds ),  s from t_n to t

T(t) is accumulated transmittance: the probability the ray has survived, unabsorbed, from t_n up to t. A point deep behind a dense occluder has small T, because the integral of σ in front of it is large, so its color barely reaches the final pixel however bright it is. This is how NeRF represents occlusion without ever computing visibility explicitly, it falls straight out of the exponential decay term.

A neural network cannot evaluate a continuous integral in closed form, so NeRF samples N points along the ray at increasing distances t_1 < t_2 < ... < t_N, with segment lengths δ_i = t_{i+1} - t_i, and uses the exact discretization of the equation above from Max (1995):

α_i = 1 − exp(−σ_i·δ_i)
T_i = exp( −Σ σ_j·δ_j for j < i ) = Π (1 − α_j) for j < i
C   = Σ T_i · α_i · c_i,  i = 1..N

α_i is the local opacity contributed by segment i: if a segment is very dense or long, α_i approaches 1 (fully opaque); if it is nearly empty, α_i approaches 0. T_i is the transmittance that survived every segment before this one. The product T_i·α_i is exactly the weight sample i contributes to the final pixel, and by construction these weights (plus the leftover transmittance that reaches the far bound) sum to 1. This is alpha compositing, the same operation used to layer semi-transparent PNG images, applied along a ray instead of across image layers.

Why raw coordinates blur: positional encoding

Feeding raw (x, y, z) straight into an MLP produces oversmoothed results, missing fine texture and sharp edges. This is not a bug specific to NeRF, it is a documented property of MLPs called spectral bias (Rahaman et al., 2019): gradient descent on a network with a handful of layers preferentially fits low-frequency variation first and struggles to represent high-frequency detail as a function of low-dimensional coordinate input.

NeRF counters this by lifting each scalar coordinate into a higher-dimensional vector of sinusoids at increasing frequencies before it ever reaches the network:

γ(p) = ( p, sin(2⁰πp), cos(2⁰πp), sin(2¹πp), cos(2¹πp), ..., sin(2^(L-1)πp), cos(2^(L-1)πp) )

Position uses L = 10, so each of x, y, z expands to 1 + 2·10 = 21 values, giving a 63-dimensional input. Viewing direction uses only L = 4, giving a 27-dimensional input, because view-dependent color effects vary far more smoothly with angle than fine geometric detail varies with position, so fewer frequency bands are needed and using more would risk fitting angular noise. Here is the encoding traced by hand for one coordinate:

import math

def positional_encoding(p, L):
    out = [p]
    for i in range(L):
        freq = 2 ** i
        out.append(math.sin(freq * math.pi * p))
        out.append(math.cos(freq * math.pi * p))
    return out

print([round(v, 4) for v in positional_encoding(0.5, 2)])

Trace it by hand: p = 0.5, L = 2. For i = 0, freq = 1: sin(π·0.5) = sin(90°) = 1.0, cos(π·0.5) = cos(90°) ≈ 0. For i = 1, freq = 2: sin(2π·0.5) = sin(180°) ≈ 0, cos(2π·0.5) = cos(180°) = −1.0. The list printed is [0.5, 1.0, 0.0, 0.0, -1.0] (the two "0.0" entries are actually on the order of 1e-16 in floating point, rounding to zero, not exact algebraic zero). One scalar became five numbers spanning multiple frequencies, giving the MLP the raw material to represent sharp variation instead of only smooth blur.

Worked example: compositing one ray by hand

Take a ray through the statue with four sample points, equally spaced by δ = 0.2 along the ray. Suppose the network has already been queried at each point and returned:

sample 1: σ = 0.1  color = red    (1, 0, 0)
sample 2: σ = 0.5  color = green  (0, 1, 0)
sample 3: σ = 2.0  color = blue   (0, 0, 1)
sample 4: σ = 0.3  color = yellow (1, 1, 0)

Sample 3 is far denser than the rest, a stand-in for the ray finally hitting solid bronze after passing through less dense material (patina, dust, or simply the network being uncertain closer to the surface boundary). Compute opacity and transmittance step by step: α₁ = 1 − e^(−0.1·0.2) = 1 − e^(−0.02) ≈ 0.0198. α₂ = 1 − e^(−0.1) ≈ 0.0952. α₃ = 1 − e^(−0.4) ≈ 0.3297. α₄ = 1 − e^(−0.06) ≈ 0.0582. Transmittance accumulates as a running product of survival probabilities: T₁ = 1, T₂ = (1 − α₁) ≈ 0.9802, T₃ = T₂·(1 − α₂) ≈ 0.8869, T₄ = T₃·(1 − α₃) ≈ 0.5945. Verify this in code rather than trusting the hand arithmetic:

import math

def volume_render(sigmas, deltas, colors):
    T = 1.0
    weights = []
    for sigma, delta in zip(sigmas, deltas):
        alpha = 1 - math.exp(-sigma * delta)
        weights.append(T * alpha)
        T *= (1 - alpha)
    C = [0.0, 0.0, 0.0]
    for w, c in zip(weights, colors):
        for k in range(3):
            C[k] += w * c[k]
    return weights, C, T

sigmas = [0.1, 0.5, 2.0, 0.3]
deltas = [0.2, 0.2, 0.2, 0.2]
colors = [(1, 0, 0), (0, 1, 0), (0, 0, 1), (1, 1, 0)]

weights, C, T_bg = volume_render(sigmas, deltas, colors)
print([round(w, 4) for w in weights])
print([round(c, 4) for c in C])
print(round(T_bg, 4))

Tracing the loop: iteration 1 gives alpha ≈ 0.0198, weights = [0.0198], T updates to 0.9802. Iteration 2 gives alpha ≈ 0.0952, appends T·alpha ≈ 0.0933, T updates to 0.8869. Iteration 3 gives alpha ≈ 0.3297, appends 0.8869·0.3297 ≈ 0.2924, T updates to 0.5945. Iteration 4 gives alpha ≈ 0.0582, appends 0.5945·0.0582 ≈ 0.0346, T updates to 0.5599. The three printed lines are [0.0198, 0.0933, 0.2924, 0.0346], [0.0544, 0.1279, 0.2924], and 0.5599.

Read the result physically. The rendered color (0.0544, 0.1279, 0.2924) is mostly blue, because sample 3, the dense "surface" point, contributes the largest single weight (0.2924) even though it is only the third of four samples. Sample 4's yellow barely registers (weight 0.0346) despite having a respectable density of its own, because it sits behind the dense sample 3 and most of the transmittance that would have carried its color forward has already been absorbed. And 0.5599 of the ray's transmittance survives past all four samples into whatever lies beyond, meaning this ray has not yet hit a fully opaque surface within the four points sampled; on a real background-free render that leftover would pick up a background color.

Training without a single 3D label

The network never sees a ground-truth depth map, mesh, or point cloud during training. What it does need is the training photographs themselves plus, for every photograph, the camera's position and orientation in a shared coordinate frame. These poses are not measured by hand, they are recovered automatically by running structure-from-motion (COLMAP is the standard tool) on the photo set before NeRF training even starts, exploiting exactly the point-matching approach that struggles on the reflective object itself but works well enough on the surrounding matte geometry and texture to fix camera positions.

Given poses, one training step samples a random batch of rays, a few thousand pixels drawn from across the training photographs. For each ray, the current network is queried at sampled points, volume rendering composites those into a predicted color exactly as in the worked example above, and that predicted color is compared against the actual pixel value from the photograph using squared error. Because every step of the pipeline, positional encoding, the MLP forward pass, and the alpha-compositing sum, is differentiable, the gradient of this photometric loss flows backward through the entire rendering equation into the MLP's weights, and Adam gradient descent updates them. No stage of training ever touches 3D geometry directly. The σ field that eventually looks like a solid bronze idol emerges purely as a side effect of the network being forced to agree, simultaneously, with 80 different 2D photographs of the same underlying 3D reality. Geometry is what falls out when enough viewpoints must all be explained by one consistent function.

Misconception: "NeRF stores a 3D model you can just open in Blender"

The natural assumption, especially coming from photogrammetry where the output really is a mesh file, is that training a NeRF produces some kind of stored 3D asset, a voxel grid or point cloud sitting on disk that a viewer just plays back. It does not. The trained artifact is a set of MLP weights, nothing else. There is no array anywhere holding "the color at (2.3, 1.1, 0.7)"; that value only comes into existence when you run a forward pass with those coordinates as input.

This has two direct consequences worth internalizing. First, rendering a new view is not free at inference time either: every pixel of every new image requires marching a ray through dozens of sample points, each one a full forward pass through the eight-layer MLP, which is why the original NeRF took roughly 30 seconds to render a single 800×800 frame on a contemporary GPU, and why later variants exist specifically to attack this cost, as discussed below. Second, if you actually want a mesh, a `.obj` file to drop into a game engine, you have to build one afterward as a separate step: evaluate σ on a dense regular grid of points and run an algorithm like marching cubes to extract a surface where density crosses a threshold. The network itself never held that grid, you constructed it by querying the function many times after training finished. A second, related trap: because everything the network knows lives in weights fit to one specific set of 80 photographs, that trained network is useless on a different statue in a different room. Unlike an image classifier trained once on millions of images and then applied to any new photo, a vanilla NeRF is retrained from scratch for every new scene.

The pipeline end to end

The diagram below traces one ray from camera through the whole learning loop, using the exact weights computed in the worked example, so the widths of the composited bar are the real numbers, not illustrative guesses.

Where the field went next

The rendering-cost bottleneck named above, hundreds of MLP forward passes for every single pixel, is exactly what later work targeted. Instant Neural Graphics Primitives (Müller et al., SIGGRAPH 2022) replaced the plain positional encoding with a small multiresolution hash table that a much smaller MLP can query, cutting training from hours to seconds by moving most of the representational capacity out of network weights and into a fast lookup structure. 3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023) went further and dropped the implicit MLP function entirely, representing the scene as an explicit set of millions of small colored, oriented ellipsoids ("splats") that can be rasterized directly, without marching any rays through a network at all, enabling real-time frame rates. Both descendants keep NeRF's central idea, learning a scene from only photographs and poses by making rendering differentiable and comparing against real pixels, while attacking the one part of the original design, per-pixel network evaluation along a ray, that this chapter's volume-rendering derivation shows is the expensive step.

Active recall

Attempt these before reading the answers.

1. Why does the NeRF architecture feed viewing direction into only the last layers of the network, after density has already been predicted from position alone?
2. In the worked example, if σ₃ had been 20.0 instead of 2.0 (a near-opaque surface), what happens to w₄, and why does that match physical intuition?
3. Why does NeRF training need a known camera pose for every photograph, and where do those poses normally come from?
4. Does a NeRF trained on one statue generalize to rendering a different statue it has never seen? Contrast this with an image classifier trained on ImageNet.
5. Using the worked example's numbers, what is the rendered pixel color if the leftover transmittance (0.5599) is composited against a pure white background instead of being left as zero?
6. Why does positional encoding use more frequency bands for position (L = 10) than for viewing direction (L = 4)?

Worked answers

1. Density is a geometric property of a point in space, whether that point contains solid matter, so it is physically impossible for it to depend on which direction a camera happens to be looking from. Color, by contrast, is what is actually reflected or emitted toward the camera, and real materials (metal sheen, glass glare) genuinely change appearance with viewing angle. Making density a function of position only, and color a function of both, encodes this physical constraint directly into the architecture rather than hoping the network learns it from data.

2. With σ₃ = 20.0, α₃ = 1 − e^(−20·0.2) = 1 − e^(−4) ≈ 1 − 0.0183 = 0.9817. T₃ is unaffected (it only depends on samples before 3), staying at roughly 0.8869, but T₄ = T₃·(1 − α₃) ≈ 0.8869·0.0183 ≈ 0.0162, down from 0.5945. So w₄ = T₄·α₄ ≈ 0.0162·0.0582 ≈ 0.00094, about 36 times smaller than the original 0.0346. Sample 4 becomes nearly invisible to the final pixel, exactly as expected: an almost fully opaque surface at sample 3 blocks nearly all the light that would have reached the camera from anything behind it, the same way a real solid object occludes what is behind it.

3. Rendering a training ray requires knowing exactly which line through 3D space produced a given pixel, which is only computable if the camera's position and orientation are known. Without correct poses, the predicted ray would not correspond to the actual ray that produced the ground-truth pixel, and the photometric loss would be comparing unrelated things, giving the network a meaningless gradient. Poses are recovered before NeRF training starts by running structure-from-motion (typically COLMAP) on the photo set.

4. No. A vanilla NeRF's MLP weights are fit entirely to one specific scene's radiance field; querying it with coordinates from a different object's geometry produces output the network was never trained to produce, since it has never seen any other scene. An ImageNet classifier, by contrast, is trained across millions of diverse images specifically so its learned features transfer to new, unseen photos. Every new NeRF scene needs its own training run from scratch, which is exactly the cost that hash-grid and Gaussian-splatting variants were built to reduce.

5. Compositing over a white background adds the leftover transmittance times the background color to the rendered result: C_final = C + T_leftover·(1, 1, 1) = (0.0544, 0.1279, 0.2924) + 0.5599·(1, 1, 1) = (0.6143, 0.6878, 0.8523), a pale blue-gray rather than a saturated blue, because more than half the ray's energy budget went to the background rather than any of the four colored samples.

6. Fine geometric detail (sharp edges, textures) varies rapidly with position and needs many frequency bands to represent, whereas view-dependent color effects like specular highlights vary comparatively smoothly as viewing angle changes, so fewer bands suffice. Using more bands than necessary on direction would let the network fit high-frequency angular noise instead of genuine reflectance behavior, and would also cost more computation in the color branch, which is evaluated at every sample along every ray.

Think About It

Think about this: How would you explain nerf: neural radiance fields to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where nerf: neural radiance fields is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting nerf: neural radiance fields to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind nerf: neural radiance fields, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.