The flicker problem
Suppose you already have a working text-to-image diffusion model and you want a text-to-video model in a hurry. The lazy approach: run the image model once per frame, twenty-four times a second, feeding it the same prompt each time. Ten seconds of footage, 240 independent calls, done. This does not work, and the reason it fails is worth deriving exactly, because the fix defines every video generation system that followed, including Sora.
A diffusion model starts each generation from a fresh draw of Gaussian noise and denoises it step by step toward an image. Two independent calls to the same model, even with an identical prompt, start from two independent noise vectors and walk two different denoising trajectories. Let x and y be the initial noise vectors for two consecutive frames, each a vector in R^d with independent, identically distributed standard normal entries. Their dot product has mean zero, since E[x·y] = Σᵢ E[xᵢ]E[yᵢ] = 0 by independence. Its variance is Var(x·y) = Σᵢ E[xᵢ²yᵢ²] = Σᵢ E[xᵢ²]E[yᵢ²] = d, because each term contributes exactly 1. So the dot product has standard deviation √d. For large d, both ‖x‖ and ‖y‖ concentrate tightly around √d (a standard concentration-of-measure fact for high-dimensional Gaussians), so the cosine similarity cos(x,y) = (x·y)/(‖x‖‖y‖) concentrates around (x·y)/d, giving a standard deviation of roughly √d/d = 1/√d.
Plug in a realistic number. A modest video latent frame with, say, 3,600 spatial positions gives 1/√3600 = 1/60 ≈ 0.017. Two independently sampled frame-noise vectors are cosine-similar to within about ±0.017 of zero — statistically indistinguishable from orthogonal. Nothing carries over from frame t to frame t+1 except the shared text prompt, which only constrains the semantic content, not the pixel-level texture, camera micro-jitter, or lighting speckle. The result is exactly what you would predict: a technically correct sequence of frames, each individually plausible, that flickers when played back, because high-frequency detail is being redrawn from scratch 24 times a second with no mechanism forcing consistency between draws. Every real video generation system exists to close this gap: instead of denoising each frame separately, denoise the entire clip jointly, so that one shared computation — not 240 unrelated ones — determines the pixels of every frame at once.
From pixels to a spatio-temporal tensor
The denoising diffusion probabilistic model (DDPM) framework of Ho, Jain, and Abbeel (2020) defines a forward process that gradually corrupts data x₀ into noise over T steps, q(xₜ|xₜ₋₁), and trains a network εθ to reverse it by predicting the noise added at each step, minimizing E[‖ε − εθ(xₜ, t)‖²]. For images, x₀ is an H×W×3 tensor. The direct extension to video treats a clip as a 4-D tensor of shape T×H×W×3 and asks the network to denoise the whole block at once. Video Diffusion Models (Ho et al., 2022) did exactly this: a 3-D U-Net where 2-D spatial convolutions are extended with a temporal axis, and attention is factorized into a spatial pass (attend across pixels within a frame) and a temporal pass (attend across frames at a fixed pixel), rather than one unrestricted 3-D attention over every pixel of every frame simultaneously. Make-A-Video (Singer et al., 2022) and Imagen Video pushed the same idea further, adding temporal super-resolution networks that interpolate extra frames after the base model generates a low-frame-rate clip.
Factorized attention is a compute necessity, not a stylistic choice: full 3-D self-attention over every spacetime position costs O(N²) in the number of positions N = T·H·W, and video multiplies the token count of an already-expensive image transformer by the frame count. The worked example later in this chapter makes this concrete with real numbers. But even factorized 3-D convolution over raw pixels is wasteful, because most of what a video's pixels encode — smooth gradients, redundant background, sensor noise — carries little information relevant to denoising. The next move, borrowed directly from image diffusion, is to stop operating on pixels at all.
Compressing before you diffuse
Latent diffusion (Rombach et al., 2022 — the paper behind Stable Diffusion) trains a variational autoencoder (VAE) to compress an image into a smaller latent grid, runs the diffusion process entirely in that latent space, and only decodes back to pixels at the very end. Stable Diffusion's VAE compresses spatially by a factor of 8 in each dimension, which cuts the token count the diffusion transformer has to attend over by 64× (8 × 8, one factor per spatial dimension) — and because attention compute scales with the square of token count, that 64× fewer tokens translates into roughly 64² ≈ 4,096× less attention compute.
Video systems extend this with a spatio-temporal VAE: the encoder compresses both the spatial dimensions and the temporal dimension, so a 240-frame clip might become a latent tensor with far fewer than 240 frames along its temporal axis, each latent frame carrying more channels to compensate. This is not free — a temporal compression factor of, say, 4 means the decoder must hallucinate plausible sub-latent-frame motion when reconstructing pixels, which is exactly the kind of learned, content-aware interpolation a VAE decoder is trained to do, unlike naive frame-blending. The payoff is enormous: diffusion now runs over a latent tensor that is smaller than the raw video by roughly the product of the spatial and temporal compression factors, and every subsequent stage of the pipeline — patchifying, attention, denoising — operates on this compressed representation instead of raw pixels.
Spacetime patches: tokens for video
Vision Transformers (Dosovitskiy et al., 2021) established that an image can be cut into fixed-size non-overlapping patches, each patch flattened and linearly projected into a token, and a standard transformer run over the resulting token sequence exactly as if it were a sentence. Sora's technical report, "Video Generation Models as World Simulators" (OpenAI, 2024), applies the same idea to the compressed latent video tensor: cut it into spacetime patches — small blocks spanning a few pixels of height, width, and a slice of time — and treat each patch as one token, analogous to a token in a language model.
The reason this specific choice matters goes beyond compute. A fixed 3-D convolutional grid bakes in a fixed spatial resolution and a fixed number of frames, because convolution kernels and pooling layers are wired to a specific tensor shape. A patch-and-transformer design has no such constraint: a longer clip, a wider aspect ratio, or a higher resolution simply produces more tokens in the sequence, and the same transformer processes them, the way a language model handles a longer sentence as a longer token sequence rather than needing a different architecture. This is the architectural reason Sora is described as training natively on videos "at their native size" — varying durations, resolutions, and aspect ratios — rather than the older convention of resizing and center-cropping every training clip to one fixed square resolution, which throws away composition information and forces awkward padding or cropping at inference time to hit a fixed shape. Patchifying converts the entire duration/resolution problem into "how many tokens," which a transformer already knows how to handle.
The Diffusion Transformer backbone
Peebles and Xie's "Scalable Diffusion Models with Transformers" (ICCV 2023) replaced the U-Net backbone of latent diffusion with a plain Vision-Transformer-style backbone operating on the patch tokens, calling the result DiT. Each DiT block is a standard transformer block — self-attention over the token sequence, then a per-token MLP — with one addition: conditioning information (the diffusion timestep and, for text-to-image or text-to-video, a text embedding) is injected via adaptive layer normalization, specifically a variant the paper calls adaLN-Zero. Instead of adding conditioning as extra tokens in the sequence (which would lengthen it and raise attention cost quadratically) or via cross-attention layers, adaLN-Zero regresses a per-channel scale, shift, and gate from the conditioning vector and applies them around each block's normalization layers. The gate parameters are initialized so that, at the start of training, every block computes the identity function — a design chosen because it made large transformer diffusion models measurably faster and more stable to train than alternative conditioning schemes the paper compared against.
The paper's headline empirical result is a scaling law: holding everything else fixed, image quality (measured by FID) improves smoothly and predictably as total training compute increases, whether that compute increase comes from a bigger model or from finer patches (which means more tokens per image). Concretely, DiT's own ablation across patch sizes p ∈ {8, 4, 2} shows that shrinking the patch — and therefore increasing the token count for a fixed image — improves sample quality monotonically at every model size tested, at the direct cost of more attention compute. This tradeoff is the same one that governs Sora's spacetime patches: finer spacetime patches mean better fidelity and more expensive attention, and the worked example below quantifies exactly how expensive.
The full pipeline
Worked example: the cost of attention over a video
The pipeline diagram names the quantities; here is what they cost in practice, worked from first principles with round, clearly-labeled illustrative numbers (OpenAI has not disclosed Sora's exact compression factors or patch size, so treat the figures below as order-of-magnitude reasoning, not published specifications).
Take a 10-second clip at 24 frames per second and 1280x720 resolution: T = 240 raw frames. Suppose the spatio-temporal VAE compresses space by 8x in each spatial dimension (the same ratio Stable Diffusion's VAE uses) and time by 4x — giving a latent tensor of T' = 60, H' = 160, W' = 90. Patchify with spatial patch size p = 2 and no further temporal patching:
T_tok, H_tok, W_tok = 60, 160 // 2, 90 // 2 # 60, 80, 45
N = T_tok * H_tok * W_tok # 60 * 80 * 45
full_attn_cost = N ** 2
spatial_cost = (H_tok * W_tok) ** 2 * T_tok
temporal_cost = (T_tok ** 2) * (H_tok * W_tok)
factorized_cost = spatial_cost + temporal_cost
print(N, full_attn_cost, factorized_cost, round(full_attn_cost / factorized_cost, 2))
Tracing this by hand: N = 60 * 80 * 45 = 216,000 tokens. Unrestricted 3-D self-attention costs O(N²) for the query-key score matrix, so full_attn_cost = 216,000² = 46,656,000,000. Factorized attention instead runs spatial attention independently within each of the 60 latent frames — cost (80·45)² = 3,600² = 12,960,000 per frame, times 60 frames, giving spatial_cost = 777,600,000 — plus temporal attention independently at each of the 3,600 spatial locations across the 60 frames — cost 60² = 3,600 per location, times 3,600 locations, giving temporal_cost = 12,960,000. Summing, factorized_cost = 790,560,000. The code above prints 216000 46656000000 790560000 59.02: factorizing attention into a spatial pass and a temporal pass cuts the attention compute for this clip by roughly 59x relative to full 3-D attention over the same token grid, for identical model width d (which cancels in the ratio, since both costs scale with the same per-pair cost d). This is the concrete reason every video diffusion architecture discussed in this chapter uses some form of factorized or windowed attention rather than dense attention over the raw spacetime token grid — dense attention is not merely slower, it is tens of times more expensive for a completely ordinary clip length.
How Sora specifically assembles these pieces
Per OpenAI's technical report, Sora is best read as the DiT recipe scaled up and generalized to video with three specific choices layered on top of the generic pipeline above. First, videos and images are unified into one training distribution: an image is simply a video with T = 1, so the same spacetime-patch tokenizer and the same DiT backbone train on both, with no separate image-only pathway. Second, training happens at each clip's native resolution, duration, and aspect ratio rather than after resizing and cropping to one fixed shape, which the patch-token representation makes tractable — a widescreen clip and a vertical clip simply produce different-shaped token grids, both consumed by the same transformer, avoiding the letterboxing and cropping artifacts that fixed-resolution training bakes into a model's outputs. Third, text conditioning uses highly descriptive captions produced with the re-captioning technique introduced for DALL-E 3 (Betker et al., 2023): rather than training on the short, often generic captions that accompany raw video data, a captioning model is used to generate long, detailed descriptions of the training clips, which measurably improves how faithfully the trained model follows detailed text prompts at inference time.
The report also describes Sora exhibiting behaviors consistent with learning some implicit world structure — object permanence when a subject is briefly occluded, roughly consistent 3-D camera motion — as an emergent property of training a large transformer on large amounts of video at scale, rather than a modeled or hand-engineered mechanism. That framing (the report's title calls video models "world simulators") is a claim about emergent capability from scale, distinct from the architectural mechanics covered above, and it is worth keeping the two separate: the spacetime-patch-plus-DiT design explains how the model computes each denoising step, not why the resulting behavior generalizes to physically plausible motion.
Production realities: training and serving
Video diffusion models strain systems in ways an image model or an autoregressive language model does not. Attention over hundreds of thousands of tokens per training example (as the worked example above shows) means the activation memory for even one clip can dominate GPU memory, which is why large-scale training of these models relies on splitting the token sequence itself across multiple GPUs — sequence parallelism — rather than only splitting the model's layers or parameters across devices the way a large language model typically does. Serving is a different problem from training: a diffusion model produces output through an iterative sampling loop, typically dozens of sequential forward passes through the full network (an early DDPM sampler used on the order of 1,000 steps; modern samplers commonly use 20 to 50), and — unlike autoregressive text generation — there is no KV-cache to reuse across steps, because each step reprocesses the entire token sequence with a different noise level, not one new token appended to a growing context. Classifier-free guidance (Ho and Salimans, 2022), the standard technique for making text conditioning strongly steer the output, runs the network twice per step — once with the text conditioning and once without — and combines the two predictions, roughly doubling the per-step compute; batching the two passes into one call can hide much of that as extra latency by using spare GPU parallelism, but it does not reduce the underlying FLOP cost.
Because sampling latency is the product of step count and per-step cost, and per-step cost for video is already large due to token count, the field has invested heavily in cutting step count without cutting quality: progressive distillation (Salimans and Ho, 2022) trains a student sampler to match two teacher steps in one, halving step count repeatedly, while consistency models (Song, Dhariwal, Chen, and Sutskever, 2023) train a network to map any noisy point on a diffusion trajectory directly to the clean sample, enabling few-step or even single-step generation. Both lines of work exist because, for video specifically, the naive multiply-by-many-steps cost is large enough that a production service generating clips on demand needs every available lever — fewer steps, cached unconditional passes, batched requests — to keep latency and GPU-hour cost within a viable budget.
Common misconception
The most natural wrong mental model for a video diffusion system is that it works like a next-frame predictor: generate frame 1, then generate frame 2 conditioned on frame 1, and so on, the way a video codec predicts future frames from past ones, or the way a language model predicts the next token from previous tokens. This is not how Sora or the DiT-based systems in this chapter work. The entire spacetime latent volume is denoised jointly — the diffusion process operates on the whole clip's tokens at once, at every step, with temporal attention layers letting information flow freely in both directions across time, not just forward. This is precisely why the flicker-problem derivation at the start of this chapter identifies the actual failure mode correctly: the fix is not "make frame generation causal," it is "make frame generation joint." It also explains capabilities that a causal, frame-by-frame model could not easily have: Sora's report describes extending a generated video backward in time from a starting clip, which is straightforward for a model that denoises an entire spacetime block non-causally, and awkward for a model architecturally committed to generating strictly forward in time.
Active recall
Attempt each question before reading its answer.
1. Why does running an image diffusion model independently, frame by frame, on the same prompt produce visible flicker rather than a smooth video, even though every individual frame looks correct?
2. A convolutional video model is trained only on 128x128, 16-frame clips. Why does a spacetime-patch-and-transformer design handle a request for a 512x288, 96-frame clip more naturally than that convolutional model does?
3. In the worked example, spatial patch size is increased from p = 2 to p = 4, with T' = 60, H' = 160, W' = 90 unchanged and only spatial (not temporal) patching affected. Recompute N, the full attention cost, the factorized cost, and the full/factorized ratio, and state which of the two factorized terms now dominates.
4. Why does DiT inject the timestep and text conditioning through adaLN-Zero rather than by concatenating them as extra tokens to the sequence?
5. A studio wants a 30-second, 4K (3840x2160) clip from the same model and compression ratios as the worked example. Estimate the new token count N relative to the original 216,000, and explain qualitatively why the factorized attention cost grows faster than N does.
Answers.
1. Each independent call starts from an independent Gaussian noise vector. For high-dimensional noise, two independent draws are essentially orthogonal (cosine similarity concentrates near 0, with standard deviation about 1/√d), so the fine-grained texture each frame settles into during denoising is statistically unrelated to its neighbor's. Only the shared text prompt links the frames, and a prompt constrains semantic content, not pixel-level detail, so consecutive frames diverge in exactly the high-frequency detail the eye is most sensitive to — the visible symptom is flicker.
2. A convolutional (or fixed-grid) architecture is wired to one input tensor shape by its kernel and pooling structure, so a different resolution or frame count requires resizing/cropping to fit, or a different network. A patch-and-transformer design converts any input shape into a token sequence of whatever length that shape implies; the transformer itself has no fixed sequence-length requirement (the same way a language model handles sentences of varying length), so a larger or differently-shaped clip is simply a longer or differently-composed token sequence for the identical network to process.
3. With p = 4: H_tok = 160 // 4 = 40, W_tok = 90 // 4 = 22, T_tok unchanged at 60. N = 60 * 40 * 22 = 52,800. full_attn_cost = 52,800² = 2,787,840,000. spatial_cost = (40·22)² · 60 = 880² · 60 = 46,464,000. temporal_cost = 60² · (40·22) = 3,600 · 880 = 3,168,000. factorized_cost = 46,464,000 + 3,168,000 = 49,632,000. The ratio is 2,787,840,000 / 49,632,000 ≈ 56.17 — down from 59.02. The spatial term still dominates numerically (46.5M vs 3.2M), but it shrank faster than the temporal term did going from p=2 to p=4, so temporal attention is now a larger fraction of the factorized total than it was at p=2 — coarsening the spatial patch size disproportionately cheapens the spatial term and leaves temporal attention as the relatively slower-shrinking cost.
4. Concatenating conditioning as extra tokens lengthens the sequence that every attention layer processes, and attention cost scales quadratically with sequence length, so even a few extra conditioning tokens raise the cost of every single attention operation in the network. adaLN-Zero instead folds the conditioning vector into per-channel scale, shift, and gate parameters applied around each block's normalization, at a cost that scales linearly with the number of channels and is paid once per block regardless of how many tokens the sequence contains — cheaper, and, per the DiT paper's ablations, also better-optimizing because the zero-initialized gate makes each block start as an identity function.
5. 3840x2160 at 24fps for 30s gives 720 raw frames. With the same 8x spatial / 4x temporal compression: latent T' = 180, H' = 270, W' = 480. With p = 2: T_tok = 180, H_tok = 135, W_tok = 240, so N = 180 * 135 * 240 = 5,832,000 — exactly 27x the original 216,000 tokens. Full attention cost scales as N², so it would grow by exactly 27² = 729x. Factorized cost is dominated by the spatial term, (H_tok·W_tok)² · T_tok, which squares the per-frame spatial token count — and per-frame spatial token count itself grew a lot (from 3,600 to 32,400, exactly 9.0x) because more pixels are packed into every single frame at 4K. Squaring a 9.0x increase gives 81x, times the additional T_tok growth (60 to 180, 3x), so the spatial term alone grows on the order of 243x — far more than the 27x growth in raw token count. The qualitative lesson: because the spatial attention term is quadratic in per-frame resolution but only linear in duration, scaling up resolution is much more expensive for a video diffusion model's attention cost than scaling up duration by the same factor — which is why production video systems typically treat resolution and duration as separately tunable, and price or throttle them differently, rather than treating "make the video bigger" as one uniform cost.
Think About It
Think about this: How would you explain video generation systems: from concepts to sora to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind video generation systems: from concepts to sora, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.