An orbit that never takes discrete jumps
When ISRO's flight dynamics team steers a spacecraft like Chandrayaan-3 from a geosynchronous transfer orbit toward the Moon, it never asks "where will the spacecraft be after step 1, step 2, step 3?" It asks a different question: given the spacecraft's current position and velocity, and the combined pull of Earth, Moon and Sun at this instant, what is its instantaneous rate of change? That question is a differential equation, and it gets handed to a numerical integrator, typically a Runge-Kutta variant, which decides for itself how finely to slice time. Near a lunar flyby, where gravity changes fast, it takes tiny steps. During a long coasting arc, where almost nothing changes, it takes large ones. The trajectory itself is a continuous curve; the number of steps the solver used to compute it is an implementation detail, chosen adaptively to hit a target accuracy, not fixed in advance.
A ResNet image classifier is built the opposite way. Feed an image in, and it passes through layer 1, then layer 2, then layer 3, all the way to layer 50, each one nudging a hidden representation h into a slightly different h. The number of layers is not something the network decides at inference time. You, the architect, bake it in before training starts, and it never changes, no matter how easy or hard the input is. A Neural ODE is the idea of applying ISRO's orbit-propagation trick to a neural network: replace the fixed staircase of discrete layers with a continuous trajectory h(t), governed by a differential equation, and let a numerical solver decide, per input, how much computation it actually needs to spend to compute that trajectory accurately.
The residual block was already doing Euler's method
Start from something you already know cold: the residual connection. In a ResNet, layer l maps hidden state h_l to h_(l+1) by
h_(l+1) = h_l + f(h_l, θ_l)
where f is a small block (say conv, batchnorm, ReLU, conv) and θ_l are that layer's own weights. Now compare this to the simplest numerical method for solving an ordinary differential equation: the forward Euler method. If dh/dt = f(h(t), t, θ) is the ODE and you want to step from time t to time t + Δt, Euler's method says
h(t + Δt) ≈ h(t) + Δt · f(h(t), t, θ)
Set Δt = 1 and drop the explicit time argument, and the two equations are identical. A stack of L residual layers is exactly the forward-Euler integration of some differential equation, using a fixed step size of 1, for L steps, with a fresh set of weights θ_l plugged in at every step. This is not a loose analogy; it is the literal derivation behind the 2018 paper that introduced the idea, "Neural Ordinary Differential Equations" by Chen, Rubanova, Bettencourt and Duvenaud. Once you see a ResNet as "Euler's method with a crude, fixed step size and a brand-new function reused at every step," an obvious question follows: what happens if you tie the weights across depth, so f becomes one function f(h, t, θ) instead of a different f_l at every l, shrink the step size toward zero, and hand the integration to a real numerical solver instead of a single crude update rule?
Depth becomes an integral, not a count
That question is the entire definition of a neural ODE block. Instead of specifying a number of layers L, you specify a single vector field f(h(t), t, θ), parameterized by one small network whose weights are shared across the whole depth of the block, and you define the hidden state as the solution to the initial value problem
dh(t)/dt = f(h(t), t, θ), h(0) = x
The block's output is h(T) for some chosen T (commonly T = 1), obtained not by looping over discrete layers but by calling a numerical ODE solver:
h(T) = h(0) + ∫₀ᵀ f(h(t), t, θ) dt = ODESolve(f, h(0), 0, T, θ)
"Depth" has stopped being an integer you choose before training. It has become the number of times an adaptive solver evaluates f while integrating from 0 to T to hit a requested error tolerance, exactly like the number of steps ISRO's integrator spends propagating a lunar transfer. A well-behaved f early in training might need only six evaluations to integrate accurately; the same block, once its dynamics grow stiffer later in training, might need thirty. Loosen the requested tolerance and you get a cheaper, blurrier approximation of the same continuous transformation; tighten it and you get a costlier, more faithful one. Accuracy and compute have become a dial you turn at inference time, not an architectural decision frozen at design time.
The diagram below places the two pictures side by side: the ResNet's fixed staircase of equal-sized Euler hops on the left, and the neural ODE's single continuous trajectory, sampled at solver-chosen, unevenly spaced points, on the right.
Worked example: same compute budget, very different accuracy
To make "the solver decides how many evaluations to spend" concrete, integrate a toy neural ODE block by hand. Let the vector field be the simplest nonlinear-in-general, linear-in-this-instance function, dh/dt = θh, with θ = 0.5 and h(0) = 1, and integrate out to T = 1. This has a closed-form answer, which is exactly why it makes a good check: h(t) = e^(θt), so the true value is h(1) = e^0.5 = 1.6487213 (to seven digits). Any numerical scheme is graded against this.
Four-step Euler. Split [0, 1] into 4 steps of Δt = 0.25. Each Euler step multiplies the current value by (1 + Δt·θ) = (1 + 0.125) = 1.125:
def euler_step(h, t, dt, theta):
return h + dt * theta * h
h, t, theta, dt = 1.0, 0.0, 0.5, 0.25
for _ in range(4):
h = euler_step(h, t, dt, theta)
t += dt
print(round(h, 6))
# 1.125
# 1.265625
# 1.423828
# 1.601807After 4 function evaluations of f, Euler lands at h(1) ≈ 1.601807 against the true 1.648721, an error of 0.046914, about 2.85% relative error. Now double the step count to 8 steps of Δt = 0.125 (factor 1.0625 per step): (1.0625)^8 works out to 83521² / 65536² = 6,975,757,441 / 4,294,967,296 ≈ 1.624166. Error drops to about 0.024555, roughly 1.49% relative. Doubling the number of steps roughly halved the error, which is exactly what you'd expect: forward Euler has global error that scales linearly with step size, O(Δt). This is the "add more layers, get more accuracy" behaviour you already know from stacking ResNet blocks.
Same 4 evaluations, a smarter solver. Now integrate the identical ODE with one step of the classical fourth-order Runge-Kutta method (RK4), the kind of solver a neural ODE library actually calls. RK4 spends 4 evaluations of f inside a single step, exactly matching the compute budget of the 4-step Euler run above:
def f(h, t, theta):
return theta * h
def rk4_step(h, t, dt, theta):
k1 = f(h, t, theta)
k2 = f(h + dt / 2 * k1, t + dt / 2, theta)
k3 = f(h + dt / 2 * k2, t + dt / 2, theta)
k4 = f(h + dt * k3, t + dt, theta)
return h + (dt / 6.0) * (k1 + 2 * k2 + 2 * k3 + k4)
h = rk4_step(1.0, 0.0, 1.0, 0.5)
print(round(h, 7))
# 1.6484375Trace it: k1 = 0.5, k2 = 0.5·(1 + 0.5·0.5) = 0.625, k3 = 0.5·(1 + 0.5·0.625) = 0.65625, k4 = 0.5·(1 + 1·0.65625) = 0.828125. The weighted sum is 0.5 + 2(0.625) + 2(0.65625) + 0.828125 = 3.890625, divided by 6 gives 0.6484375, plus the initial 1 gives h(1) = 1.6484375. Against the true 1.6487213, the error is only 0.0002838, about 0.0172% relative. For the identical 4 evaluations of f, RK4's error is roughly 165 times smaller than Euler's. This is precisely why a neural ODE library reaches for an adaptive, higher-order solver (torchdiffeq's default is a Dormand-Prince variant, dopri5, itself a Runge-Kutta method with automatic step-size control) instead of naive fixed-step Euler: for the same number of forward passes through f, you get an integration that is orders of magnitude closer to the true continuous transformation, or equivalently, you can hit the same accuracy with far fewer evaluations than Euler needs.
import torch
import torch.nn as nn
from torchdiffeq import odeint # (assumed helper, not shown — adaptive-step solver, e.g. dopri5)
class ODEFunc(nn.Module):
def __init__(self):
super().__init__()
self.net = nn.Sequential(nn.Linear(4, 20), nn.Tanh(), nn.Linear(20, 4))
def forward(self, t, h):
return self.net(h) # dh/dt = f(h, theta); one shared block for all "depth"
func = ODEFunc()
h0 = torch.randn(1, 4)
t = torch.tensor([0.0, 1.0])
h1 = odeint(func, h0, t)[-1] # solver decides how many evaluations of func to spendTraining without storing every layer: the adjoint method
Ordinary backpropagation through L stacked layers needs every intermediate activation h_1 through h_L kept in memory, because the backward pass walks back through the same computation graph the forward pass built. Storage grows linearly with depth, O(L). Naively backpropagating through an ODE solver would be worse: you would need to store every intermediate state the solver touched, and that count is itself variable and can be large.
The 2018 paper's other contribution was to avoid this entirely using the adjoint sensitivity method, a tool from classical optimal control theory. Define the adjoint a(t) = ∂L/∂h(t), the sensitivity of the training loss L to the hidden state at time t. It turns out a(t) itself obeys a differential equation, derivable by the chain rule in the continuum limit:
da(t)/dt = −a(t)ᵀ · (∂f/∂h)
Starting from a(T) = ∂L/∂h(T), which you get for free from the loss at the final output, you integrate this adjoint ODE backward in time from T down to 0, using the same kind of numerical solver as the forward pass. Along the way you also recompute h(t) backward (running the original dynamics in reverse, since the ODE is time-reversible), rather than reading it out of stored memory. The gradient with respect to the shared parameters θ falls out as a third quantity integrated alongside the first two: dL/dθ = ∫₀ᵀ a(t)ᵀ · (∂f/∂θ) dt. The upshot: training needs only the solver's working state at any one instant, not a stored copy of every intermediate activation. Memory becomes O(1) in depth. The price you pay is that recomputing h(t) backward instead of storing it costs additional evaluations of f, and if f is not exactly invertible along its trajectory in practice (numerical drift accumulates over long integrations), this recomputation can itself introduce small errors, which is a genuine, documented limitation of the plain adjoint method, not a free lunch.
Common misconception: "continuous depth must be strictly more powerful"
Because a neural ODE integrates for a continuous interval instead of stopping after a fixed number of discrete layers, it is tempting to assume it is a strict superset of what any finite-depth network can express, "infinite depth" sounds like it should dominate any finite one. This is false, and the reason is worth sitting with because it comes straight from the existence-and-uniqueness theorem for ODEs (Picard-Lindelöf) that you meet in differential-equations coursework.
If f is well-behaved enough for a unique solution to exist from every starting point (which is required for training to even make sense, otherwise the "output" of a given input is ambiguous), then two different trajectories h(t) starting from two different initial states can never cross at the same time t. If they did cross, running the ODE backward from that shared crossing point would have to split into two different paths, contradicting uniqueness. A direct consequence in one dimension: the flow map φ_T, taking h(0) to h(T), must preserve order. If a < b initially, then φ_T(a) < φ_T(b) for every T, always.
Now try to build a one-dimensional neural ODE that computes g(x) = −x, the simple reflection through the origin. You would need φ_T(1) = −1 and φ_T(−1) = 1. But 1 is greater than −1, and order preservation demands φ_T(1) stay greater than φ_T(−1) for all T. Sending 1 to −1 and −1 to 1 flips that order, which no continuous, well-posed one-dimensional flow can ever do. No choice of f, no matter how large the network computing it, escapes this: it is a structural fact about ODE trajectories, not a training or capacity limitation. A single ResNet layer, by contrast, can trivially compute h → −h with one linear map; discrete steps are allowed to "teleport" across space in a way a continuous flow cannot. This is exactly the observation behind Augmented Neural ODEs (Dupont, Doucet and Teh, 2019): the practical fix is to give the state extra, otherwise-unused dimensions to route trajectories around each other in a higher-dimensional space, the same way two roads that cannot cross on a flat map can pass over and under each other with a flyover. Continuous depth buys you adaptive, memory-cheap computation; it does not, on its own, buy you extra expressive power over a discrete network of the same width, and in this specific sense it is provably more restricted.
Where the continuous formulation actually earns its keep
The most direct payoff shows up whenever your data does not arrive on a fixed clock tick. A UPI transaction stream, a hospital ICU's vital-sign readings, or sensor logs from a satellite pass all generate observations at irregular, unevenly spaced timestamps. An RNN or a fixed-depth network expects a regular grid and forces you to pad, bucket, or interpolate the gaps, throwing away exactly the timing information that often matters most. A neural ODE has no such requirement: because h(t) is defined continuously, you can evolve the hidden state from whatever timestamp the previous observation had straight to whatever timestamp the next one has, using the solver's own adaptive step control to bridge a five-minute gap and a five-hour gap correctly, without any special-casing. This is the basis of "Latent ODEs" for irregularly sampled time series, one of the strongest empirically-demonstrated use cases from the original line of work, alongside continuous normalizing flows, where the continuous-time change-of-variables formula replaces an expensive discrete Jacobian-determinant computation with a cheap trace, making certain generative models tractable at a scale their discrete counterparts are not.
Active recall
Attempt each question before reading its answer.
Q1. Show algebraically that a single ResNet residual layer is exactly one step of forward Euler applied to dh/dt = f(h).
Q2. A neural ODE block integrates dh/dt = 0.5h from h(0)=1 to t=1 using 8 equal Euler steps instead of 4. Compute h(1) and compare its relative error to the 4-step case (2.85%). What pattern does this confirm about Euler's method?
Q3. In your own words, explain why the adjoint sensitivity method lets a neural ODE train with O(1) memory in depth, and name one cost this savings does not eliminate.
Q4. Why can no one-dimensional neural ODE represent g(x) = −x, no matter how the vector field f is parameterized?
Q5. An adaptive solver spends 20 evaluations of f integrating one input through a neural ODE block and only 6 evaluations for a different input. What does that gap tell you about the network's behaviour near each input?
Q6. A neural ODE trained to convergence needs, on average, 25 evaluations of f per forward pass, each costing about as much as one ResNet layer. Is it therefore strictly cheaper to run than a 25-layer ResNet? Justify your answer.
A1. Forward Euler for dh/dt = f(h(t),t,θ) with step Δt gives h(t+Δt) ≈ h(t) + Δt·f(h(t),t,θ). Setting Δt = 1 and writing h(t) as h_l, h(t+Δt) as h_(l+1), and f(h(t),t,θ) as f(h_l,θ_l) gives h_(l+1) = h_l + f(h_l,θ_l), which is precisely the residual update rule. The residual connection is Euler integration with unit step size and a distinct weight set reused at each step.
A2. With 8 steps, Δt = 0.125 and the per-step multiplier is 1 + 0.125·0.5 = 1.0625. Since 1.0625 = 17/16, h(1) = (17/16)^8 = 83521²/65536² = 6,975,757,441/4,294,967,296 ≈ 1.624166. The true value is e^0.5 ≈ 1.648721, so the error is about 0.024555, roughly 1.49% relative, versus 2.85% for 4 steps. Doubling the number of steps roughly halved the error, confirming Euler's global error is O(Δt), first-order convergence: linear in step size, not the dramatically faster convergence a higher-order method like RK4 achieves for the same evaluation budget.
A3. Standard backpropagation needs every intermediate activation kept in memory because the backward pass reads them off the forward computation graph, giving O(L) memory for L layers. The adjoint method instead treats a(t) = ∂L/∂h(t) as the solution of its own ODE, da/dt = −a(t)ᵀ·∂f/∂h, and integrates it backward from T to 0 using a solver, recomputing h(t) backward alongside it rather than reading it from storage. Only the solver's current working state is ever held, independent of how many evaluations depth required, so memory is O(1) in depth. What this does not eliminate is compute: recomputing h(t) backward costs additional evaluations of f, and because floating-point integration is not perfectly reversible, recomputed trajectories can drift slightly from the true forward ones over long integrations.
A4. Because f is required to be well-behaved enough to guarantee a unique solution from every initial condition (Picard-Lindelöf), distinct trajectories can never cross at the same time t; if they did, running the ODE backward from the crossing point would have to split into two paths, which uniqueness forbids. In one dimension this forces the flow map to be strictly order-preserving: if a < b at t=0, then φ_T(a) < φ_T(b) for every T. Representing g(x) = −x would require φ_T(1) = −1 and φ_T(−1) = 1, which reverses the order of 1 and −1 and is therefore impossible for any continuous flow, regardless of how f is parameterized.
A5. An adaptive solver takes smaller steps, hence more evaluations, exactly where the local error estimate says the trajectory is curving quickly, meaning f or its derivatives are changing rapidly along that path. Needing 20 evaluations instead of 6 means the block's dynamics are "stiffer" or more nonlinear near that particular input's trajectory, so the network is implicitly spending more effective computation on it. This gives neural ODEs an input-adaptive compute budget that a fixed-depth ResNet, which spends the same 25 layers on every input regardless of difficulty, cannot offer.
A6. No, not necessarily. Training also requires solving the adjoint ODE backward, which itself calls f repeatedly to recompute h(t) and integrate a(t), typically costing a comparable or greater number of evaluations than the forward pass. So a full training step's forward-plus-backward compute can exceed a 25-layer ResNet's forward-plus-backward pass, even though the neural ODE's memory footprint stays O(1) instead of O(25) stored activations. The real advantage demonstrated here is constant memory and a tunable accuracy-compute tradeoff, not automatically lower total FLOPs.
Think About It
Think about this: How would you explain neural odes: continuous depth to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind neural odes: continuous depth, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.