Optical Flow: Estimating Motion

On 23 August 2023, ISRO's Vikram lander began its final descent to the lunar surface with no GPS to tell it how fast it was moving sideways. GPS needs a constellation of satellites broadcasting timed signals — the Moon has none. Yet a lander drifting horizontally at the wrong speed when it touches down can tip over or crash. Among Vikram's sensors was a downward-pointing Lander Horizontal Velocity Camera, photographing the lunar surface several times a second during descent. ISRO's own description of the instrument is blunt about its job: measure the lander's horizontal velocity from successive images of the ground below. Between one photograph and the next, craters and boulders on the surface shift a few pixels across the frame. Measure that pixel shift, know the camera's height and field of view, and you can convert "the ground moved this many pixels in this direction" into "the lander is moving this many metres per second in this direction." That conversion — apparent motion of brightness patterns in an image sequence, turned into a velocity estimate — is exactly the problem this chapter solves. It is called optical flow, and the same mathematics that helps a spacecraft measure its own velocity without GPS is what lets OpenCV track a cricket ball across frames, what early action-recognition networks used as an input stream before end-to-end learning took over, and what video codecs exploit when they encode "this block moved eight pixels right" instead of storing a whole new block of pixels.

What optical flow actually measures

Precision matters here, because the name is misleading in a way that produces a real misconception (addressed properly later in this chapter). Optical flow does not measure the true 3-D motion of objects in the world. It measures the apparent 2-D motion of brightness patterns in the image plane — where did this intensity value go between frame t and frame t+dt? Most of the time the two agree: a boulder that is physically moving also produces a shifting brightness pattern. But they can diverge, and a rigorous treatment has to keep the distinction sharp from the first definition.

Formally: let I(x, y, t) be the intensity of the pixel at position (x, y) at time t. Given two consecutive frames, I(x, y, t) and I(x, y, t+dt), optical flow assigns to every pixel a displacement (dx, dy) — equivalently a velocity (u, v) = (dx/dt, dy/dt) — describing how far that pixel's brightness pattern appears to have moved. Do this for every pixel and you get a dense vector field over the whole image: the flow field. Do it only for a sparse set of distinctive points (corners, textured patches) and you get sparse flow — cheaper, and what most real-time trackers actually compute.

From brightness constancy to an equation

Everything starts from one assumption, and naming it explicitly matters because it is also the assumption that breaks in the misconception section below. The brightness constancy assumption says: as a point in the scene moves from (x, y) to (x+dx, y+dy) over a short time dt, its intensity does not change:

I(x, y, t) = I(x + dx, y + dy, t + dt)

Now expand the right-hand side as a first-order Taylor series around (x, y, t), which is valid when the motion dx, dy, dt is small:

I(x+dx, y+dy, t+dt) ≈ I(x,y,t) + I_x·dx + I_y·dy + I_t·dt

where I_x = ∂I/∂x, I_y = ∂I/∂y are the spatial image gradients (computable with the same Sobel-type convolution you would already recognise from edge detection) and I_t = ∂I/∂t is the frame-to-frame intensity difference. Substituting brightness constancy, the I(x,y,t) terms cancel on both sides, leaving:

I_x·dx + I_y·dy + I_t·dt = 0

Divide through by dt and define u = dx/dt, v = dy/dt — the horizontal and vertical flow velocities:

I_x·u + I_y·v + I_t = 0

This is the optical flow constraint equation (OFCE), sometimes called the brightness constancy equation. Every quantity except u and v is directly measurable from the two frames — I_x and I_y from a spatial gradient filter on frame t, I_t from subtracting frame t from frame t+dt. What you get is one linear equation relating the two unknown velocity components at each pixel.

One equation, two unknowns: the aperture problem

One equation cannot fix two unknowns. Geometrically, the OFCE only constrains the flow component along the gradient direction — call it the normal flow, magnitude -I_t / |∇I| — and says nothing about the component perpendicular to the gradient, i.e. along an edge. Look at a diagonal edge through a small window (an "aperture"): if the edge slides sideways along its own length, no change in brightness is visible inside the window at all, so that component of motion is genuinely unrecoverable from this equation alone. This is the aperture problem, and it is not a numerical inconvenience to be optimised away — it is a real information-theoretic limit of what a local brightness measurement can tell you.

You can see it algebraically too. Suppose a small window sits on a perfectly straight intensity ramp, so every pixel in the window has the same gradient, say I_x = 2, I_y = 3 everywhere. Three pixels give three copies of the same equation 2u + 3v = -I_t, not three independent constraints. Stack them into a system and try to solve for (u, v) by least squares (the method the next section builds properly) and the coefficient matrix has rank 1: summing I_x² = 12, I_y² = 27, I_xI_y = 18 over three identical-gradient pixels gives a determinant of 12·27 - 18² = 324 - 324 = 0. Singular. No unique solution exists — confirming, algebraically, exactly what the geometric aperture argument predicted.

Lucas–Kanade: pooling equations from a neighbourhood

The aperture problem is unsolvable with information from a single pixel, but it is often solvable with information from a small neighbourhood, provided that neighbourhood contains more than one gradient direction. This is the insight behind the Lucas–Kanade method (1981): assume the flow (u, v) is constant across a small window W around each pixel (say 5×5 or 7×7), then write down the OFCE for every pixel in that window and solve all of them at once.

For N pixels in the window, this gives N equations in the same 2 unknowns:

I_x1·u + I_y1·v = -I_t1
I_x2·u + I_y2·v = -I_t2
   ...
I_xN·u + I_yN·v = -I_tN

In matrix form, A[u, v]^T = b, where A is N×2 and generally overdetermined (N > 2). Solve it in the least-squares sense — minimise ||A[u,v]^T - b||² — which gives the normal equations:

(A^T A) [u, v]^T = A^T b

A^TA works out to the 2×2 matrix [[ΣI_x², ΣI_xI_y], [ΣI_xI_y, ΣI_y²]]. This exact matrix — built from sums of squared and cross gradients over a window — is the structure tensor, and it is not a coincidence that the same matrix reappears in the Harris corner detector: A^TA is invertible precisely when the window contains gradients pointing in more than one direction, which is precisely the condition for a window to look like a corner rather than a flat edge or a blank patch. Lucas–Kanade tracking is reliable at corners and richly textured patches, unreliable on a single straight edge (rank-deficient A^TA, the aperture problem again), and completely blind on a flat, textureless region (A^TA is the zero matrix — no gradient information at all).

Worked example: solving a Lucas–Kanade window by hand

Take a 3-pixel window where a Sobel-type filter has already produced these gradients and frame differences (this is what you would read off real pixels — the next section shows that step explicitly):

Each row gives one equation I_xu + I_yv = -I_t, so b = [4, 7, 3]^T. Build the normal equations by hand:

ΣI_x² = 1²+3²+2² = 14        ΣI_x I_y = 1·2+3·1+2·(-1) = 3
ΣI_y² = 2²+1²+(-1)² = 6      ΣI_x b  = 1·4+3·7+2·3 = 31
                                ΣI_y b  = 2·4+1·7+(-1)·3 = 12

A^T A = [ 14  3 ]      A^T b = [ 31 ]
        [  3  6 ]              [ 12 ]

Solve the 2×2 system by Cramer's rule. det(A^TA) = 14·6 - 3·3 = 84 - 9 = 75.

u = (31·6 - 3·12) / 75 = (186 - 36) / 75 = 150/75 = 2
v = (14·12 - 3·31) / 75 = (168 - 93) / 75 = 75/75  = 1

So the recovered flow at this pixel is (u, v) = (2, 1): two pixels of rightward motion, one pixel downward, per frame. This is exactly checkable in code:

import numpy as np

Ix = np.array([1, 3, 2])
Iy = np.array([2, 1, -1])
It = np.array([-4, -7, -3])

A = np.stack([Ix, Iy], axis=1)   # shape (3, 2)
b = -It

flow = np.linalg.solve(A.T @ A, A.T @ b)
print(flow)   # [2. 1.]

Note this window was constructed with three different gradient directions (1,2), (3,1), (2,-1) — not multiples of each other — which is exactly why A^TA came out invertible instead of singular like the ramp example earlier. That is not a coincidence; it is the corner condition made concrete with numbers.

Worked example: the same pipeline from raw pixels — and where the linear model bends

The table above assumed I_x, I_y, I_t had already been computed. Here is the full pipeline starting from actual pixel intensities, which also exposes something the textbook version hides: the OFCE is a first-order (linear) approximation, and real intensity is not always locally linear.

Define a synthetic 5×5 frame with intensity I(x, y) = x² + 2xy (a curved field, chosen deliberately so the approximation error below is visible and explicable rather than accidental):

frame1[y][x] = x**2 + 2*x*y     # x, y = 0..4

frame1 =
[[ 0  1  4  9 16]
 [ 0  3  8 15 24]
 [ 0  5 12 21 32]
 [ 0  7 16 27 40]
 [ 0  9 20 33 48]]

Now shift the whole scene by the true flow (u, v) = (1, 0) — one pixel right, none down — by setting frame2(x, y) = frame1(x-1, y):

frame2 =
[[ 1  0  1  4  9]
 [-1  0  3  8 15]
 [-3  0  5 12 21]
 [-5  0  7 16 27]
 [-7  0  9 20 33]]

Take the interior 3×3 window (x, y ∈ {1,2,3}) so central differences never run off the grid. At each of the 9 pixels compute I_x = (I(x+1,y) - I(x-1,y))/2, I_y = (I(x,y+1) - I(x,y-1))/2 on frame1, and I_t = frame2(x,y) - frame1(x,y):

Ix = [4, 6, 8, 6, 8, 10, 8, 10, 12]
Iy = [2, 4, 6, 2, 4,  6, 2,  4,  6]
It = [-3, -5, -7, -5, -7, -9, -7, -9, -11]

A = np.stack([Ix, Iy], axis=1)
b = -np.array(It)
flow = np.linalg.solve(A.T @ A, A.T @ b)
print(flow)   # [0.88461538 0.        ]

v comes back exactly 0, matching the true v = 0. But u comes back 552/624 = 23/26 ≈ 0.885, not exactly the true u = 1. This is not an arithmetic slip — it is the Taylor linearisation showing its limit. I(x, y) = x² + 2xy is linear in y for any fixed x (no y² term), so the y-direction central difference is exact and the OFCE holds exactly in that direction. But it is quadratic in x, and a first-order Taylor expansion drops the x² curvature term; that dropped term is exactly what shows up as the 0.115-pixel error in the recovered u. Real Lucas–Kanade implementations do not stop at one linear solve for this reason: they warp frame2 back by the current estimate, recompute I_t against the warped frame, and re-solve — a Newton iteration that converges to the true flow even when the intensity surface is curved — and for motions larger than a pixel or two, they do this inside an image pyramid (coarse, blurred, small-motion estimate refined at successively higher resolution). That combination, pyramidal iterative Lucas–Kanade, is what OpenCV's calcOpticalFlowPyrLK implements, and it is the general family of technique behind image-based velocity sensing on a descending spacecraft: small apparent per-frame motion, refined estimate, repeated every frame.

The misconception: "optical flow" is not "the motion field"

Students who have just derived the OFCE tend to walk away believing optical flow is an object's true motion, recovered from images. It is worth correcting explicitly, because the gap between the two is not an edge case — it is structural, and it follows directly from the brightness constancy assumption itself. Optical flow measures how a brightness pattern moves. The true 3-D motion projected into 2-D is called the motion field. They usually agree, but here are two classic cases (Horn, 1986) where they do not:

• Motion with zero flow. A perfectly smooth, uniformly lit sphere rotating in place produces a motion field that is clearly nonzero at every visible point — the surface is spinning. But if the sphere has no texture and the lighting is uniform, every point on it always looks like the same shade of gray as its neighbours; frame t and frame t+dt are pixel-for-pixel identical. I_t = 0 everywhere, so the OFCE gives (u, v) = (0, 0) everywhere. Optical flow reports "nothing moved." The motion field disagrees completely.
• Flow with zero motion. Now hold the same sphere perfectly still and instead move a point light source around it. Nothing in the scene physically moves, so the true motion field is zero everywhere. But the highlight and the shading gradient sweep across the surface as the light moves, so I_t ≠ 0 in the region the highlight crosses, and the OFCE reports nonzero flow where nothing moved at all.

The lesson generalises: optical flow is a well-defined, computable quantity derived entirely from image intensities under an assumption (brightness constancy) that can fail whenever appearance changes without motion (lighting, reflections, transparency, shadows) or motion happens without appearance changing (untextured surfaces). This is precisely why the Vikram lander's velocity camera needs the lunar surface to be sufficiently textured (craters, rock shadows, regolith patterns) under stable illumination — a featureless patch of dust in flat light would give the algorithm nothing to lock onto, and the flow it reports would default toward zero regardless of the lander's true horizontal speed.

Beyond a single pixel pair: pyramids and dense flow

Lucas–Kanade as derived here has two working limits worth naming precisely, both already visible in the pipeline above. First, the linearisation only holds for small motion — the Taylor expansion drops second-order terms, and the raw-pixel example showed exactly how that shows up as error when a per-pixel curvature term is neglected. Second, a single window assumes uniform flow inside it, which breaks down at motion boundaries (the edge of a moving object against a static background) where two different true flows genuinely coexist inside one window. Coarse-to-fine pyramidal Lucas–Kanade addresses the first limit by estimating flow on a heavily downsampled image (where a 20-pixel real motion looks like a 2-pixel motion, well inside the linear regime), then propagating and refining that estimate at each finer level. It does not fix the second limit — for that, dense variational methods such as Horn–Schunck replace the "constant flow in a window" assumption with a global smoothness penalty solved over the whole image, trading Lucas–Kanade's sharp corner-following for flow fields that vary smoothly everywhere, including at true object boundaries where that smoothness is itself a modelling compromise. Both families reduce, at their algebraic core, to the same OFCE derived in this chapter.

Active recall

Attempt these before reading the answers below.

1. Starting from I(x,y,t) = I(x+dx, y+dy, t+dt), derive the optical flow constraint equation and name the one assumption the derivation depends on.
2. A pixel has I_x = 4, I_y = -2, I_t = 10. What is the component of flow along the gradient direction (the normal flow)? Can you determine the full (u, v) from this alone?
3. A 3-pixel Lucas–Kanade window has gradients (2, 4), (1, 2), (3, 6) at its three pixels. Without solving anything, predict whether A^TA will be invertible, and justify it in one line.
4. A window has exactly two pixels with gradients (1, 0) and (0, 2), and true flow (u, v) = (3, -1). Compute I_t at each pixel, then solve the 2×2 system for (u, v) and check it recovers the true flow.
5. Explain why a spinning, uniformly-lit, textureless sphere yields zero optical flow despite obviously moving. Name the two quantities that disagree in this example.
6. Why does Lucas–Kanade fail near a motion boundary, where a foreground object's edge slides across a static background, even though both regions individually have well-conditioned A^TA?

Answers

1. Expand the right side to first order: I(x,y,t) + I_xdx + I_ydy + I_tdt. Setting this equal to I(x,y,t) (brightness constancy — a moving point's intensity does not change) cancels the I(x,y,t) terms, leaving I_xdx + I_ydy + I_tdt = 0. Dividing by dt and writing u = dx/dt, v = dy/dt gives I_xu + I_yv + I_t = 0. The dependency is the first-order Taylor truncation plus brightness constancy — both assumptions, not facts about the world.

2. The normal flow magnitude is -I_t/|∇I| = -10/√(16+4) = -10/√20 ≈ -2.236, directed along ∇I = (4,-2) (normalised, that direction is (4,-2)/√20). No — a single equation in two unknowns cannot fix (u, v) fully; only the along-gradient component is determined, which is the aperture problem restated numerically.

3. Not invertible. All three gradient vectors are scalar multiples of (1, 2) — (2,4)=2·(1,2), (3,6)=3·(1,2) — so every row of A is a multiple of the same direction, A has rank 1, and A^TA is singular (determinant zero), exactly like the ramp example in the aperture-problem section.

4. I_t1 = -(1·3 + 0·(-1)) = -3, so equation 1 is 1·u + 0·v = 3 → u = 3. I_t2 = -(0·3 + 2·(-1)) = 2, so equation 2 is 0·u + 2·v = -2 → v = -1. Recovered (u, v) = (3, -1), matching the true flow exactly — because the two gradients are orthogonal, the 2×2 system is already diagonal and each equation isolates one unknown directly, no least squares needed.

5. The true 3-D motion field is nonzero everywhere on the sphere's surface (every surface point is physically rotating). But with no texture and uniform lighting, every point looks identical to its neighbours at every instant, so frame t and frame t+dt are pixel-identical: I_t = 0 everywhere, and the OFCE returns (u,v) = (0,0) at every pixel. The two quantities that disagree are the motion field (nonzero, true 3-D motion projected to 2-D) and optical flow (zero, apparent brightness-pattern motion).

6. Lucas–Kanade assumes a single (u, v) is constant across the whole window. Straddle a motion boundary and the window contains pixels from two regions with genuinely different true flows; least squares then returns one compromise vector that is wrong for both regions (or, worse, an ill-conditioned mix if the two motions happen to produce gradients that partially cancel). A^TA being invertible only guarantees a unique answer to the least-squares problem as posed — it says nothing about whether the constant-flow-in-a-window assumption underlying that problem was valid in the first place.

Think About It

Think about this: How would you explain optical flow: estimating motion to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where optical flow: estimating motion is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting optical flow: estimating motion to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind optical flow: estimating motion, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.