Sliding Window Attention: Efficient Long Context Processing

A day of Test cricket breaks a chatbot

Suppose you build an assistant that answers questions about an ongoing Test match by reading the full ball-by-ball commentary as its context: "What happened to the third umpire's decision in the 62nd over?" To answer that, the model needs the commentary text tokenized and fed in as the sequence it attends over. Estimate the size of that sequence with a few honest assumptions: about 90 overs get bowled in a day, each over has 6 balls, a Test runs 5 playing days, and each ball's commentary line averages about 20 tokens once tokenized. That gives 90 × 6 = 540 balls per day, × 5 days = 2,700 balls, × 20 tokens = 54,000 tokens for the full match transcript — and that is before adding scorecards, player stats, or previous-innings summaries the assistant might also need.

Standard self-attention, the mechanism you have already met in the transformer, computes a compatibility score between every token and every token before it in the sequence, then mixes information according to those scores. For a sequence of length n, the number of such pairs works out (derived precisely in the next section) to n(n+1)/2. At n = 54,000, that is 54,000 × 54,001 / 2 = 1,458,027,000 — close to 1.46 billion pairwise scores, for a single attention head in a single layer, before you even multiply by the embedding dimension to get actual floating-point operations. Real models stack dozens of layers and dozens of heads. The quadratic term is the entire problem: double the context length and the attention cost roughly quadruples, not doubles.

But notice something about the question itself: answering "what happened in over 62" mostly needs the commentary from nearby overs — over 60, 61, 62. It rarely needs token-by-token detail from over 3. Most of the information a token needs to interpret itself lives close to it in the sequence. Sliding window attention is the architectural decision to take that observation seriously: restrict each token's direct attention to a fixed-size local neighbourhood, and let a different mechanism — depth, not width — carry information over longer ranges. This is exactly the design choice made in production models such as Mistral 7B, and it is what this chapter builds from first principles.

Full self-attention, precisely

For a sequence of n tokens with query, key, and value vectors Q, K, V ∈ ℝ^n×d, causal self-attention (the kind used in autoregressive language models, where token i may only look at tokens 1..i, never the future) computes, for each query position i:

score(i, j) = (q_i · k_j) / sqrt(d),  for all j ≤ i
weights(i, :) = softmax(score(i, :))
output(i) = Σ_j weights(i, j) · v_j

The mask "j ≤ i" is what makes this causal: it is a full lower-triangular mask. Count how many (i, j) pairs survive that mask for a toy sequence of n = 8 tokens (positions 0 to 7, 0-indexed): row i keeps i + 1 entries (positions 0..i), so the total is 1 + 2 + 3 + … + 8 = n(n+1)/2 = 8 × 9 / 2 = 36 score computations. In general, full causal attention costs Θ(n²) score computations and, since each score is a d-dimensional dot product, Θ(n²d) floating-point operations. That is the wall the commentary-bot hits at n = 54,000.

Restricting the mask: what "sliding window" means

Sliding window attention keeps the causal rule (no looking at the future) but adds a second restriction: no looking too far into the past either. Fix a window size w. Token i is now only allowed to attend to keys in the range [max(0, i − w + 1), i] — itself and the w − 1 tokens immediately before it. Formally:

mask(i, j) = True   if  max(0, i - w + 1) ≤ j ≤ i
mask(i, j) = False  otherwise

Every query now attends to at most w keys, regardless of how long the full sequence n is. Work out the n = 8, w = 3 case by hand, row by row (row i lists which columns j are allowed):

i=0: {0}            → 1 key
i=1: {0,1}          → 2 keys
i=2: {0,1,2}        → 3 keys
i=3: {1,2,3}        → 3 keys
i=4: {2,3,4}        → 3 keys
i=5: {3,4,5}        → 3 keys
i=6: {4,5,6}        → 3 keys
i=7: {5,6,7}        → 3 keys
Total = 1+2+3+3+3+3+3+3 = 21

Compare to the 36 pairs full causal attention needed for the same n = 8: a 41.7% reduction, even at this tiny scale, because the first w − 1 rows are still growing toward the window (they simply have fewer than w tokens behind them to attend to) while every row from i = w − 1 onward is capped at exactly w. The general formula, for n ≥ w, splits into that ramp-up region and the flat region:

windowed_pairs(n, w) = w(w+1)/2 + (n - w)·w

Check it against the n = 8, w = 3 case: 3×4/2 + (8−3)×3 = 6 + 15 = 21. Matches the hand count exactly. Now push it to the commentary-bot's scale: n = 54,000 tokens with a Mistral-style window w = 4,096. Windowed pairs = 4096×4097/2 + (54,000−4096)×4096 = 8,390,656 + 204,406,784 = 212,797,440 — about 213 million score computations, versus the 1,458,027,000 (≈1.46 billion) full causal pairs computed above. That is roughly a 6.85× reduction (about 85.4% fewer pairwise scores) at this one context length, and the ratio only grows as n grows further, because windowed cost scales linearly in n while full attention's cost scales quadratically.

A fully worked example: computing actual outputs

Counting pairs shows the compute savings but hides what the mechanism actually does to numbers. Work a complete forward pass by hand for a toy case: n = 4 tokens, window w = 2 (each token attends to itself and the one token immediately before it), embedding dimension d = 1 so that dot products are plain scalar multiplication and the softmax scaling factor sqrt(d) = 1. Pick:

q = [1, 2, 1, 3]   (query scalars for tokens 1..4)
k = [1, 1, 2, 1]   (key scalars)
v = [10, 20, 30, 40]  (value scalars)

Step 1 — raw scores s(i, j) = q_i · k_j for every pair (full matrix, before masking):

        j=0  j=1  j=2  j=3
i=0:     1    1    2    1
i=1:     2    2    4    2
i=2:     1    1    2    1
i=3:     3    3    6    3

Step 2 — apply the causal sliding-window mask for w = 2 (row i keeps only j ∈ {i−1, i}, clipped at 0):

        j=0    j=1    j=2    j=3
i=0:     1    -inf   -inf   -inf
i=1:     2      2    -inf   -inf
i=2:   -inf     1      2    -inf
i=3:   -inf   -inf     6      3

Step 3 — softmax each row over only its finite entries. Row 3 is the interesting one: scores {6, 3} for j ∈ {2, 3}. exp(6) ≈ 403.43, exp(3) ≈ 20.09, sum ≈ 423.52, giving weights 403.43/423.52 ≈ 0.9526 on j = 2 and 20.09/423.52 ≈ 0.0474 on j = 3. Doing this for every row:

        j=0     j=1     j=2     j=3
i=0:   1.0000    0       0       0
i=1:   0.5000  0.5000    0       0
i=2:     0    0.2689  0.7311    0
i=3:     0      0    0.9526  0.0474

Step 4 — output(i) = Σ_j weights(i,j)·v_j:

output(0) = 1.0000×10                              = 10.0000
output(1) = 0.5000×10 + 0.5000×20                   = 15.0000
output(2) = 0.2689×20 + 0.7311×30                   = 27.3106
output(3) = 0.9526×30 + 0.0474×40                   = 30.4743

Every one of these numbers was verified by running the equivalent NumPy computation rather than trusting the hand arithmetic; they match to four decimal places. Two things are worth noticing before moving on. First, token 0 (the sequence start) has only itself in its window, so its output is trivially its own value — the ramp-up effect from the pair-counting section, now visible at the level of actual numbers. Second, token 3's output leans heavily on token 2's value (30) rather than its own (40), even though both are inside its window. That is not a bug; it falls out of how attention weighs things, and it sets up the misconception below.

Building and running the mask in code

The mask itself is a small, mechanical function. Here it is in NumPy, constructing exactly the n = 8, w = 3 mask worked out earlier:

import numpy as np

def sliding_window_mask(n, w):
    mask = np.full((n, n), False)
    for i in range(n):
        lo = max(0, i - w + 1)
        mask[i, lo:i + 1] = True
    return mask

n, w = 8, 3
mask = sliding_window_mask(n, w)
print(mask.astype(int))
print("Allowed pairs:", int(mask.sum()))
print("Full causal pairs:", n * (n + 1) // 2)

Running this prints exactly the banded triangular pattern derived by hand above:

[[1 0 0 0 0 0 0 0]
 [1 1 0 0 0 0 0 0]
 [1 1 1 0 0 0 0 0]
 [0 1 1 1 0 0 0 0]
 [0 0 1 1 1 0 0 0]
 [0 0 0 1 1 1 0 0]
 [0 0 0 0 1 1 1 0]
 [0 0 0 0 0 1 1 1]]
Allowed pairs: 21
Full causal pairs: 36

Notice the shape: it is not a triangle any more, it is a band — a diagonal stripe of fixed width w that slides down the matrix as i increases. That band is the entire idea; everything else in this chapter is consequences of that one geometric change. In a real transformer layer, this boolean mask is added to the raw score matrix as 0 where True and −∞ where False, exactly as done by hand in Step 2 above, before the softmax and the weighted sum over V.

Diagram: the mask band and what depth buys back

Correcting a common misconception

The natural first reaction to sliding window attention is: "if a token can only see w tokens back, then any information more than w positions away is simply gone — the model must be strictly worse than full attention at long-range reasoning." This is wrong, and the worked example above already contains the reason, just not yet stated.

Attention layers are stacked. A token's representation after layer 2 is not a function of the raw input tokens within its window; it is a function of the layer-1 outputs of the tokens within its window — and each of those layer-1 outputs was itself already a mixture of tokens within their windows. Information composes across depth exactly the way it does in a convolutional network with stacked local kernels: the receptive field grows by w − 1 positions with every additional layer. Verify this by computing reachability directly rather than asserting it: build the n×n boolean mask M for w = 3, and compute which positions the last token can reach after L "hops" through the layer stack (M, then M², then M³, keeping the union of all lower powers since a token's information persists once it has arrived). Doing this computation gives receptive field 3 after 1 layer, 5 after 2 layers, 7 after 3 layers — matching the formula RF(L) = (w−1)L + 1 exactly (3 = 2×1+1, 5 = 2×2+1, 7 = 2×3+1). A 32-layer stack with w = 4,096, the actual configuration published for Mistral 7B, reaches RF = 4,095×32 + 1 = 131,041 — a theoretical reach four times longer than the model's stated 32,768-token context window, achieved without a single layer ever computing a full n×n attention matrix.

So the corrected statement is: sliding window attention does not delete long-range information, it relocates the work of carrying it from a single wide attention operation to many layers of narrow ones. The real trade-off is not "can vs. cannot access distant tokens," it is that access becomes indirect and diffused across depth rather than a single sharp lookup, and if a genuinely rare but critical long-range fact needs a direct, undiluted path, pure depth-based mixing can dilute it. This is precisely why systems like Longformer add a handful of explicitly global tokens that every position may attend to directly regardless of the window, and why streaming-inference systems keep a few fixed "attention sink" tokens outside the sliding window — both are patches for the one real limitation of pure sliding window attention, not evidence that the window "forgets."

The tiny worked example also demonstrates a second, related misconception worth naming explicitly: token index 3's output leaned on token index 2's value (weight 0.9526) far more than on its own value (weight 0.0474), even though token index 3's own key was inside its own window. That is not the window failing; it is softmax correctly reporting that q₃·k₂ = 3×2 = 6 was a much stronger match than q₃·k₃ = 3×1 = 3. Attention weight is driven by query–key alignment, never by how large a candidate's value vector happens to be. A student who expects "the model mostly uses its own token's information, plus a little of the window" is imposing a rule attention was never built to follow.

The other reason production systems adopt this: bounded memory during generation

Compute savings are only half of why sliding window attention matters in deployed systems. During autoregressive generation, a transformer decodes one token at a time, and to avoid recomputing attention over the whole prefix at every step, it caches the key and value vectors of every previous token — the KV cache. With full causal attention, that cache grows by one entry per generated token, without bound: after generating 50,000 tokens, the cache holds keys and values for all 50,000. Memory for the KV cache scales as Θ(n), and at large n it can dominate GPU memory even more than the model's own weights.

Sliding window attention caps this directly: since token i only ever attends to the w tokens behind it, keys and values from more than w steps back are provably never read again, so they can be dropped from the cache the moment they age out of the window. The KV cache becomes a fixed-size rolling buffer of w entries — Θ(w) memory, independent of how long the generated sequence grows. This is the concrete engineering payoff that made the technique attractive for real deployments, not merely a nice complexity-theory result: it lets a model serve very long conversations or documents without its memory footprint growing with them.

Where this runs in production, and what it costs

Mistral 7B (Mistral AI, 2023) is the clearest public example: every attention layer uses a causal sliding window of w = 4,096 tokens, stacked across 32 layers, which the model's own technical report cites as giving a theoretical receptive field around 131K tokens — the exact number derived above. Longformer, built for long-document encoder tasks like question answering over full papers, pairs a local sliding window with a small set of task-specific global tokens (such as the [CLS] token, or every question-token in QA) that get full bidirectional attention to and from everything, patching exactly the diffusion weakness discussed above. Streaming-inference systems for chat-style deployment use "attention sink" tokens — keeping just the first few tokens of a conversation permanently in the KV cache alongside the sliding window — because empirically, transformers assign unusually large attention mass to early tokens regardless of their content, and dropping them destabilizes generation quality even when they carry little information themselves.

The trade-off is real and should not be sugar-coated: a pure sliding-window model needs either substantial depth or an auxiliary global/sink mechanism to match full attention's ability to make a single, direct, one-hop connection between two arbitrary distant tokens. Many current long-context systems therefore use a hybrid: most layers run cheap sliding-window attention, while a small number of layers (or a small number of designated tokens) retain full or global attention, buying most of the compute and memory savings while keeping a small number of direct long-range paths available. Sliding window attention is best understood not as a replacement for full attention but as the default local mechanism, with full attention reserved for where it is actually needed.

Active recall

Attempt these before reading the answers below.

1. For n = 10 tokens with a causal sliding window w = 4, how many query–key pairs get an actual attention weight computed? How many would full causal attention need?
2. Derive the general closed-form windowed_pairs(n, w) formula from scratch (do not just recall it), and use it to check your answer to Q1.
3. A model uses w = 2,048 with 32 stacked layers and is fed a 1,000,000-token context (e.g. an entire codebase). What is the theoretical receptive field after all 32 layers? Is that enough to connect any two arbitrary tokens in the input in one forward pass?
4. In the n = 4, w = 2 worked example, token index 3 (the 4th token) put weight 0.9526 on token index 2 and only 0.0474 on itself, even though token 3 was inside its own window. Explain why, in terms of the actual numbers involved — do not just say "that's how softmax works."
5. Longformer (an encoder-only model) uses a symmetric window (w/2 tokens on each side of position i). Explain precisely why a decoder-only autoregressive language model, generating one token at a time, cannot use a symmetric window the same way.
6. Why does capping the sliding window at w tokens also cap the KV cache at a fixed size during generation, rather than just reducing compute? What in the definition of the mask guarantees this?

Answers

1. Row i (0-indexed, i = 0..9) keeps min(i+1, 4) keys. Rows 0–3 keep 1, 2, 3, 4 keys (sum 10); rows 4–9 (six rows) each keep 4 keys (sum 24). Total = 10 + 24 = 34 pairs. Full causal attention needs n(n+1)/2 = 10×11/2 = 55 pairs. The window saves (55−34)/55 ≈ 38.2% of the pairwise computations even at this tiny scale.

2. Split the sum into the ramp-up region (rows 0 to w−1, where row i keeps i+1 keys, summing to the triangular number w(w+1)/2) and the flat region (the remaining n−w rows, each keeping exactly w keys, summing to (n−w)w). Adding them: windowed_pairs(n, w) = w(w+1)/2 + (n−w)w. Substituting n = 10, w = 4: 4×5/2 + 6×4 = 10 + 24 = 34, matching Q1 exactly.

3. RF(32) = (w−1)×32 + 1 = 2,047×32 + 1 = 65,504 + 1 = 65,505 tokens. That is nowhere near enough to span a 1,000,000-token context in one forward pass (65,505 « 1,000,000) — two tokens more than about 65,505 positions apart cannot influence each other directly through pure sliding-window composition at this depth. This is exactly why real million-token-context systems cannot rely on plain sliding window attention alone and add global tokens, memory mechanisms, or occasional full-attention layers.

4. Token 3's query is q₃ = 3. Its window contains keys k₂ = 2 and k₃ = 1, giving raw scores s(3,2) = 3×2 = 6 and s(3,3) = 3×1 = 3. Softmax turns a difference of 3 in the logits into a large ratio: exp(6)/exp(3) = exp(3) ≈ 20.1, so token 2 receives about 20× the weight token 3 receives on itself, normalizing to weights ≈ 0.9526 and 0.0474. The cause is that token 3's key (2) aligned better with token 3's own query (3) than token 3's own key (1) did — nothing about value magnitude (30 vs 40) entered the weight calculation at all.

5. During autoregressive generation, token i is produced before any token after it exists, and training uses teacher forcing under the causal constraint that token i's prediction may never depend on tokens i+1, i+2, … (doing so would leak the answer during training and be physically impossible at inference, since future tokens have not been generated yet). A symmetric window requires attending to positions after i, which violates that constraint. Encoder-only models like Longformer are not generating a sequence token by token; the entire input is available at once (e.g., for classification or extractive QA), so there is no future to protect against, and a symmetric window is safe.

6. The mask guarantees that once a key/value pair at position j falls outside every future query's window — that is, once the current decoding position i satisfies i − j ≥ w — mask(i′, j) is False for that j and all subsequent positions i′ > i as well, since the window only slides forward. No query from that point on will ever be allowed to read position j again, so it is provably safe to evict it from the cache. This is a direct consequence of the window being defined as [i−w+1, i]: the lower bound strictly increases with i, so old positions permanently exit the allowed range and never re-enter it.

Think About It

Think about this: How would you explain sliding window attention: efficient long context processing to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where sliding window attention: efficient long context processing is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting sliding window attention: efficient long context processing to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind sliding window attention: efficient long context processing, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.