Video Understanding: Temporal Modeling

In December 2021 the RBI made Video KYC (V-CIP) a legal substitute for a bank officer meeting you in person to open an account. A customer holds their face to a phone camera; software must decide, in real time, whether it is looking at a living person or at a printed photo, a phone screen replaying a video, or a 3D mask. The obvious first move — run a face-recognition model on the incoming frame and check that the face matches the ID photo — solves the wrong problem. A high-resolution printed photo of the same person matches just as well. What the system actually needs is proof of life: ask the user to blink, and check that the eye genuinely goes open → closing → closed → opening → open, in that order, over time. A single frame — even a perfect one — cannot contain this proof. Liveness is not a property of a frame. It is a property of a sequence. This chapter is about the machinery that lets a model reason across frames instead of within one: temporal modeling.

What Actually Changes When Images Become Video

A single image is a tensor of shape H×W×C (height, width, channels). A video clip is T×H×W×C — you have added a time axis T. This looks like a small change in notation and is in fact the entire difficulty of the subject, because of one property the new axis has that the others do not: order matters causally, not just spatially. Swap two rows of an image and you get a different but still physically plausible image. Swap two frames of a video — put the eye-closed frame before the eye-open frame — and you get a sequence that no living eye produces. A model that is blind to frame order cannot tell the two apart.

The most tempting first design for a video classifier is therefore also the most dangerous: late fusion. Run your favourite 2D CNN or ViT image classifier independently on every frame, and average the resulting predictions. This is fast, reuses all your image-pretrained weights, and — this is the misconception this chapter exists to correct — feels like it "sees the whole video" because every frame passed through the network. It does not. Averaging is a symmetric operation: it cannot recover which value came first.

Worked Example: Mean-Pooling Cannot See Order

Suppose a per-frame classifier outputs the probability that the subject's eye is open, sampled at five moments during a liveness check. A genuine blink produces a sequence like this, read left to right in time:

seq_A = [0.90, 0.60, 0.10, 0.60, 0.90]   # open -> closing -> closed -> opening -> open

Now consider a spoof: an attacker has spliced a single closed-eye frame into an otherwise-open video, or a low-frame-rate replay jumbles the timing. The same five numbers appear, in a different order:

seq_B = [0.90, 0.10, 0.90, 0.60, 0.60]   # jump straight to closed, then back open, then half-close twice

This is not how a human eyelid moves — a real blink is a single smooth descent and a single smooth ascent, not two half-closes with a full re-open in between. It is exactly the kind of artifact a splicing attack or a corrupted frame order produces. Feed both sequences through a mean-pool late-fusion head:

import numpy as np

seq_A = np.array([0.90, 0.60, 0.10, 0.60, 0.90])
seq_B = np.array([0.90, 0.10, 0.90, 0.60, 0.60])

print(round(seq_A.mean(), 2))                  # 0.62
print(round(seq_B.mean(), 2))                  # 0.62
print(sorted(seq_A) == sorted(seq_B))           # True

Trace it by hand: both lists contain exactly the multiset {0.90, 0.90, 0.60, 0.60, 0.10}. Sum = 3.10 either way; divide by 5; mean = 0.62 for both. The classifier head sees identical evidence for the genuine blink and the impossible splice, because mean-pooling is a symmetric function of its inputs — it is mathematically incapable of depending on order. This is not a training failure or a bug you could fix with more data; it is a structural property of the aggregation operator. Any model whose only cross-frame operation is an average, a max, or any other permutation-invariant pooling will have this blind spot regardless of how good its per-frame CNN is. Genuine temporal modeling requires an aggregation step that is sensitive to sequence — a recurrent unit that consumes frames one at a time and updates a hidden state, a 3D convolution whose kernel spans the time axis with a fixed direction, or an attention mechanism that is told each token's timestamp. All three appear below.

Making Motion Explicit: Optical Flow and the Aperture Problem

Before deep networks learned to infer motion implicitly, classical computer vision computed it explicitly, via optical flow: for each pixel, estimate the velocity (u, v) at which its brightness pattern is moving. The starting assumption is brightness constancy — a physical point keeps its intensity as it moves: I(x+u, y+v, t+1) ≈ I(x, y, t). A first-order Taylor expansion of the left side around (x, y, t) gives the optical flow constraint equation:

Ix·u + Iy·v + It = 0

where Ix, Iy are the image's spatial gradients and It is the frame-to-frame intensity change. One equation, two unknowns — underdetermined at a single pixel. The Lucas–Kanade method fixes this by assuming (u, v) is constant over a small window and stacking the constraint from every pixel in that window into a least-squares system.

Take a 3×3 window straddling a vertical eyelid-skin boundary — intensity 10 (skin, dark in this channel) to the left, 90 (sclera, bright) to the right — as the eye opens and the bright region grows leftward by one pixel between frame t and frame t+1:

Frame t   (cols x=0,1,2):      Frame t+1:
 10  10  90                     10  90  90
 10  10  90                     10  90  90
 10  10  90                     10  90  90

Using central differences at the middle column (x=1), for every row y ∈ {0,1,2}:

Ix(1,y) = [I(2,y) - I(0,y)] / 2 = (90-10)/2   = 40
Iy(1,y) = [I(1,y+1) - I(1,y-1)] / 2  (row values identical along y) = 0
It(1,y) = I_t+1(1,y) - I_t(1,y)      = 90-10  = 80

All three rows give the identical triple (40, 0, 80) because the pattern is perfectly uniform along y — a straight vertical edge has no vertical gradient anywhere along it. Summing over the 3-pixel window:

ΣIx² = 3×40² = 4800     ΣIy² = 0        ΣIxIy = 0
ΣIxIt  = 3×40×80 = 9600  ΣIyIt = 0

The Lucas–Kanade normal equations, [ΣIx² ΣIxIy; ΣIxIy ΣIy²]·[u;v] = -[ΣIxIt; ΣIyIt], become:

4800·u + 0·v = -9600   =>  u = -2
   0·u + 0·v =  0      =>  v is UNDETERMINED (any v satisfies 0=0)

The v row vanished entirely — the matrix is singular. This is the classical aperture problem, appearing here not as a hand-wavy diagram but as an honest singular linear system: along a perfectly straight edge with zero gradient in one direction, motion along that direction is invisible to the math, no matter how the pixels actually moved. Note also that u = -2, not -1, even though the edge visibly shifted by exactly one pixel — the Taylor linearization is only exact for infinitesimal motion, and a full 80-intensity-unit jump over one pixel is a sharp discontinuity, not a smooth gradient, so the first-order approximation overshoots the true displacement. It gets the direction right (negative, i.e. leftward, matching the bright region's actual leftward growth) but not the exact magnitude. This is precisely why production optical-flow pipelines run Lucas–Kanade on an image pyramid, coarse to fine: at a heavily downsampled resolution a one-pixel shift is a sub-pixel shift, which the linear approximation handles well, and each finer pyramid level only has to correct a small residual.

Two lessons carry forward into deep video architectures: (1) a single straight-edge window cannot fully determine motion — real trackers actively select corner-like windows with gradient in two directions (the Shi–Tomasi criterion), and (2) explicit optical flow is expensive per-pixel algebra that deep networks would rather learn end-to-end. Both concerns shaped the two architecture families below.

3D Convolutions: Learning Motion Without Computing Flow

A 2D convolution kernel has shape (C_out, C_in, k_h, k_w) and slides over (H, W) only — applied to a video, it must be run independently per frame, which is exactly the order-blind late-fusion trap from before if nothing follows it. A 3D convolution extends the kernel to (C_out, C_in, k_t, k_h, k_w) and slides over (T, H, W) jointly, so a single output value is a weighted sum over a small block of consecutive frames — the network now has direct access to short-range motion, the same way a 2D kernel has direct access to spatial edges.

The cost is real. Compare a layer with C_in = C_out = 64 and a 3×3 spatial kernel, run per-frame, against the same channel counts with a full 3×3×3 kernel:

2D per-frame:  64 × 64 × 3 × 3         = 36,864 parameters
3D (3x3x3):    64 × 64 × 3 × 3 × 3    = 110,592 parameters   (3× more)

Every extra temporal tap multiplies the parameter count by 3, which is why the field converged on the (2+1)D factorization (used in R(2+1)D and related action-recognition networks): replace one 3×3×3 spatial-temporal kernel with a 1×3×3 spatial convolution followed by a 3×1×1 temporal convolution, with a nonlinearity (ReLU) in between:

spatial (1x3x3):   64 × 64 × 1 × 3 × 3 = 36,864
temporal (3x1x1):  64 × 64 × 3 × 1 × 1 = 12,288
total                                       = 49,152 parameters

Full 3D divided by factorized: 110,592 / 49,152 = 2.25× fewer parameters for the same receptive field, plus an extra ReLU the joint kernel doesn't get, which empirically makes the network easier to optimize. Notice this ratio, 27/(9+3) = 2.25, depends only on the kernel size k=3 (k³ vs k²+k) and not on the channel count — it is a structural property of the decomposition, not a lucky choice of numbers.

Stack L such temporal layers (kernel size 3, stride 1) and the temporal receptive field — how many raw frames influence one output — grows as RF = 1 + 2L, exactly like the spatial receptive-field formula from ordinary CNNs. Four layers give RF = 1 + 8 = 9 frames of context: enough to span most of a blink at typical 15–30 fps sampling, but notice that this context window is fixed by architecture, unlike a recurrent network's, which can in principle carry information indefinitely.

Two-Stream Networks and I3D: Reusing Image Pretraining

An older and still-influential design sidesteps learning motion from scratch by feeding it in explicitly: the two-stream network (Simonyan & Zisserman, 2014) runs a spatial-stream CNN on raw RGB frames (what things look like) and a separate temporal-stream CNN on stacked optical-flow fields, computed classically exactly as above (how things move), and fuses their predictions late. It works well but pays the optical-flow computation cost per clip and needs two full networks.

I3D (Inflated 3D ConvNet) found a cheaper route to the same 2D-pretraining benefit for a single 3D network: take an ImageNet-pretrained 2D CNN and "inflate" every 2D kernel into 3D by repeating it T times along the new time axis, then dividing every copied weight by T. Why divide? If the video input is a static image repeated T times (the standard sanity check), an un-normalized inflated kernel would sum T identical copies of the 2D convolution's contribution and inflate the output by a factor of T; dividing each of the T copies by T exactly cancels that, so at initialization the 3D network reproduces the original 2D network's output on any still image, and all the ImageNet pretraining transfers intact before a single frame of video training happens.

Recurrent and Attention-Based Fusion

3D convolutions and I3D still fuse motion within a fixed, short local window. For a task like liveness detection, where the discriminating pattern (open→closing→closed→opening→open) unfolds over the whole clip and must be checked for the right global order, two other families are more natural.

A CNN+LSTM pipeline extracts one embedding vector per frame with a shared 2D CNN (identical weights on every frame, so no extra motion-specific parameters), then feeds the resulting sequence of embeddings e_1, e_2, …, e_T into a recurrent unit one step at a time. Each step updates a hidden state: h_t = f(e_t, h_{t-1}). Because the recurrence consumes the sequence strictly left to right and each hidden state depends on everything before it, seq_A and seq_B from the worked example above are no longer indistinguishable — the LSTM's hidden state after seeing "closed, then open, then closed again" looks nothing like its hidden state after seeing the smooth single-dip blink, even though the five raw numbers are the same multiset either way. This directly repairs the mean-pooling failure.

Video transformers push further: instead of a shared CNN summarizing each frame into one vector, split every frame into patches (as in a Vision Transformer) and let self-attention operate over the whole T × H × W grid of patch tokens. Full joint space-time self-attention lets every patch attend to every other patch across all frames — maximally expressive, but the cost of attention scales with the square of the token count, and video has far more tokens than a single image. TimeSformer and similar architectures use divided space-time attention instead: each patch first attends only to the same spatial location across all T frames (a purely temporal attention pass), then attends to all patches within its own frame (a purely spatial pass) — two cheap passes instead of one expensive joint one.

The saving is not marginal. Take T = 8 frames of a 14×14 patch grid (HW = 196 patches/frame, standard ViT-Base granularity), so there are T·HW = 1568 tokens total:

Joint attention pairs:   (T·HW)²          = 1568²         = 2,458,624
Divided - temporal pass: T²·HW            = 64 × 196      =    12,544
Divided - spatial pass:  T·HW²            = 8 × 38,416    =   307,328
Divided total:                                                =   319,872

Speed-up: 2,458,624 / 319,872 ≈ 7.7×

Nearly an order of magnitude cheaper, for a model that still lets information reach every frame (via the temporal pass) and every spatial location within a frame (via the spatial pass) — just not in one single expensive joint operation. This is the same "spatial then temporal, separately" instinct as (2+1)D convolution, arrived at independently for attention.

Assembling a Liveness Pipeline

Putting the pieces in order, a realistic Video-KYC liveness pipeline looks like this: decode the incoming clip and uniformly sample T = 16 frames; run face detection and crop each frame to a fixed face box; pass each crop through a shared 2D CNN or ViT backbone (weights tied across all 16 frames — this is the only step where per-frame processing alone is correct, because it is only extracting appearance, not deciding liveness); stack the 16 resulting embeddings into a sequence; feed that sequence into an order-sensitive temporal module — an LSTM, a stack of temporal convolutions, or divided space-time attention — whose final hidden state or pooled output goes to a small classifier head producing P(live). The diagram below traces this shape, using the LSTM variant since it maps most directly onto the blink-sequence example this chapter has followed throughout.

Active Recall

Attempt every question before reading the answer beneath it.

1. A liveness clip yields per-frame "eye open" probabilities [0.95, 0.5, 0.05, 0.5, 0.95]. A classmate says: "the mean of these is 0.59, so the model must have detected a blink." What is wrong with this reasoning?
2. In the Lucas–Kanade worked example, why did the row for v vanish completely (0·v = 0) instead of just giving a noisy estimate?
3. Compute the parameter count for a 3D convolution with C_in = C_out = 32 and a 3×3×3 kernel, and for its (2+1)D factorized equivalent (1×3×3 followed by 3×1×1, same channel counts). What is the reduction ratio, and why is that ratio independent of the channel count?
4. I3D inflates a 2D kernel into 3D by copying it T times along the time axis and dividing every copy by T. What breaks if you skip the division?
5. For T = 4 frames and a 7×7 patch grid per frame (HW = 49), compute the number of attended pairs for full joint space-time attention versus TimeSformer-style divided space-time attention, and the resulting speed-up factor.
6. Besides liveness detection, name one scenario where two videos could contain the exact same set of frames in different order, such that only a temporal model — not a per-frame classifier — could tell them apart.

Answers.

1. The reasoning conflates "mean is low" with "a blink occurred." Mean-pooling is permutation-invariant: the sequence [0.05, 0.5, 0.95, 0.5, 0.95] — eye slams shut first, then reopens and stays mostly open — has the exact same five values, hence the exact same mean of 0.59, but is not a plausible single blink. A low mean only tells you the eye was closed-ish somewhere in the clip; it cannot certify that the closing happened as one smooth, correctly-ordered dip. Only an order-sensitive aggregator (LSTM, temporal attention, causal temporal convolution) can verify the shape of the sequence, not just its average.

2. Because in that window the intensity pattern was exactly constant along the y-direction (every row of the frame had the same value column-by-column), the vertical gradient Iy was exactly 0 at every sampled pixel — not approximately small, but algebraically zero, since the edge was perfectly vertical. Every term in ΣIy², ΣIxIy, and ΣIyIt is then forced to zero, so the second row of the normal equations becomes the tautology 0 = 0: satisfied by every value of v. This is the aperture problem in its purest form — a straight edge genuinely does not constrain motion parallel to itself, no matter how much data you feed the least-squares fit, because that direction carries no information in the window at all.

3. 3D: 32×32×27 = 27,648. Factorized: spatial 32×32×9 = 9,216 plus temporal 32×32×3 = 3,072, total 12,288. Ratio: 27,648 / 12,288 = 2.25. The ratio reduces algebraically to k³ / (k² + k) for kernel size k = 3, i.e. 27/12 — the channel count C_in×C_out appears as a common factor in both numerator and denominator and cancels out, so it never affects the ratio, only the absolute parameter counts.

4. Without dividing by T, feeding the network a static image repeated T times (the standard way to check that inflation preserved the pretrained behaviour) would make every inflated 3D convolution output exactly T× too large — because the same 2D contribution gets summed T times along the copied time axis. That breaks the identity with the original 2D network's output, so the pretrained ImageNet weights would not actually behave like themselves at the start of video training, undermining the whole point of inflating a pretrained model instead of training 3D filters from scratch.

5. Joint: (T·HW)² = (4×49)² = 196² = 38,416. Divided: temporal pass T²·HW = 16×49 = 784, spatial pass T·HW² = 4×2,401 = 9,604, total 10,388. Speed-up: 38,416 / 10,388 ≈ 3.7×.

6. Any video whose meaning is defined by direction of motion rather than by what is present, e.g. CCTV footage of a passenger walking through an IRCTC platform gate: "entering the platform" and "leaving the platform" can be literally the same set of frames of the same person walking through the same gate, played forward versus backward — every individual frame is identical between the two cases, so a per-frame classifier gives identical outputs on both, but a temporal model consuming the frames in their given order can tell entry from exit because it is sensitive to which frame came before which.

Think About It

Think about this: How would you explain video understanding: temporal modeling to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where video understanding: temporal modeling is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting video understanding: temporal modeling to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind video understanding: temporal modeling, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.