T5: Text-to-Text Transfer Transformer

An IRCTC support desk gets a WhatsApp message: "mera PNR 8241093456 confirm nahi hua, refund kab milega?" One message, four separate jobs: detect that it is a complaint (classification), pull out the PNR number (extraction), decide whether it needs escalation on a 1–5 urgency scale (regression), and draft a one-line summary for the human agent's queue ("Refund pending for unconfirmed PNR 8241093456"). Build this with the transformer models from the previous chapters and you are staring at four different systems: BERT with a classification head bolted onto its [CLS] vector for urgency scoring, a separately fine-tuned BERT with a token-tagging head for PNR extraction, and GPT prompted or fine-tuned on its own for the summary line. Each task needs its own output layer, sometimes its own loss function, and its own training run. T5 — the Text-to-Text Transfer Transformer, introduced by Colin Raffel and colleagues at Google in 2019 — asks a blunt question: what if every one of these were just text in, text out? What if urgency scoring produced the string "4", extraction produced the string "8241093456", and summarization produced a sentence, all from the exact same model, the exact same vocabulary, and the exact same decoding procedure? That reframing is the entire contribution of T5, and it happened to also produce, at the time, some of the strongest transfer-learning results in NLP.

Why "encoder-only" and "decoder-only" don't give you this for free

BERT is an encoder: every token attends to every other token in both directions, which makes it superb at building a contextual representation of an input, but it has no built-in mechanism for producing variable-length output text. You graft on a task head — a linear layer over [CLS] for classification, a span-start/span-end pair for extractive QA — and that head only knows how to do one job. GPT is a decoder: it generates text autoregressively one token at a time, so it can produce output naturally, but every input token can only see tokens to its left, which throws away context when you need to deeply understand a whole source sentence before translating or summarizing it.

T5 keeps both halves. It is a full encoder-decoder transformer: an encoder stack that reads the entire input bidirectionally, the way BERT does, followed by a decoder stack that generates output autoregressively, the way GPT does, with the decoder cross-attending to every encoder position at every generation step. The output side is always text, produced token by token from the same softmax over the same fixed vocabulary regardless of task. That single design choice — always decode text, never add a task-specific head — is what "text-to-text" means, and it is what lets one pretrained checkpoint be fine-tuned on translation, summarization, sentiment classification, and semantic-similarity regression without adding a single new parameter.

Architecture: an ordinary transformer with two deliberate simplifications

Strip away the framing and T5's network is close to the original 2017 encoder-decoder transformer, with two changes the paper found improved stability at scale. First, normalization: instead of standard LayerNorm (which subtracts the mean and adds a learned bias), T5 rescales activations by their root-mean-square only — no mean subtraction, no additive bias anywhere in the normalization or the feed-forward layers — and applies it before each sub-layer rather than after (pre-norm), which keeps gradients better behaved in very deep stacks. Second, position information: instead of the sinusoidal absolute positions of the original transformer or BERT's learned absolute positions, T5 uses a relative position bias — a learned scalar added directly to each attention logit, indexed only by how far apart the query and key positions are, bucketed on a logarithmic scale (32 buckets by default, so nearby positions get fine-grained distinctions and far-apart positions get coarse ones), and that same small set of learned biases is shared identically across every layer of a stack. The practical effect is a model that handles sequences longer than anything seen in training more gracefully than absolute-position schemes, since "17 tokens apart" means the same thing whether it happens near the start or the end of a long document.

Everything else is the familiar shape from earlier chapters: multi-head self-attention, position-wise feed-forward blocks, residual connections. The encoder stack's self-attention is fully bidirectional, exactly like BERT's. The decoder stack has three attention-and-feed-forward sub-layers per block instead of one: causal self-attention over the tokens generated so far (so position t cannot see position t+1, preserving the autoregressive property), cross-attention, where the decoder's queries attend over the encoder's final output states as keys and values, and a feed-forward block. Cross-attention is the channel through which everything the encoder understood about the source text reaches the generation process — without it, the decoder would only ever know what it has generated so far, never the input it is supposed to be transforming.

The pretraining objective: span corruption on C4

T5 is pretrained on the Colossal Clean Crawled Corpus (C4) — roughly 750 gigabytes of English text filtered out of Common Crawl web dumps, with heuristics stripping boilerplate, offensive content, and code, leaving mostly well-formed prose. The pretraining objective is not BERT's per-token masking. BERT replaces roughly 15% of individual tokens with [MASK] and predicts each one independently. T5 instead corrupts contiguous spans of tokens — runs of consecutive tokens, not scattered singletons — replacing each entire span with one sentinel token drawn from a reserved set (<extra_id_0>, <extra_id_1>, … up to <extra_id_99> in the standard 32,000-token SentencePiece vocabulary), and the decoder's target is only the sentinels and the spans they replaced, concatenated — not the full original sentence. This is what makes it genuinely text-to-text: corrupt an input string, and the target is another string, both drawn from the same vocabulary the model will use for every downstream task.

Two hyperparameters govern the corruption: the corruption rate (default 15% of input tokens, matching BERT's rate) and the mean span length (default 3 tokens per span, sampled so that corrupted tokens land in contiguous runs rather than scattered individually).

Worked example: corrupting one sentence exactly the way T5's pretraining does

Take a twenty-token sentence describing exactly the kind of event the IRCTC assistant above would need to process:

"Rohan booked a train ticket from Mumbai to Pune using IRCTC and paid with UPI before the deadline this morning"

Counting tokens (splitting on whitespace, ignoring subword detail for clarity): Rohan(1) booked(2) a(3) train(4) ticket(5) from(6) Mumbai(7) to(8) Pune(9) using(10) IRCTC(11) and(12) paid(13) with(14) UPI(15) before(16) the(17) deadline(18) this(19) morning(20) — 20 tokens. At the default 15% corruption rate, the expected number of corrupted tokens is 0.15 × 20 = 3, and at the default mean span length of 3, that lands almost exactly on a single contiguous span of 3 tokens. Suppose the sampler selects positions 4–6, the span "train ticket from". A short Python function reproduces exactly what the T5 data pipeline does to that span:

def corrupt_span(tokens, start, length, sentinel_id=0):
    span = tokens[start:start + length]
    corrupted = (tokens[:start]
                 + [f"<extra_id_{sentinel_id}>"]
                 + tokens[start + length:])
    target = ([f"<extra_id_{sentinel_id}>"] + span
              + [f"<extra_id_{sentinel_id + 1}>"])
    return corrupted, target

tokens = ("Rohan booked a train ticket from Mumbai to Pune using "
          "IRCTC and paid with UPI before the deadline this morning").split()
corrupted, target = corrupt_span(tokens, start=3, length=3)
print(" ".join(corrupted))
print(" ".join(target))

Trace it by hand: tokens[3:6] is the zero-indexed slice covering positions 4, 5, 6, which is ["train", "ticket", "from"] — the span. corrupted concatenates the first 3 tokens, one sentinel, and the remaining 14 tokens (positions 7–20), giving 3 + 1 + 14 = 18 tokens. target is the sentinel, the 3 span tokens, and the next sentinel: 5 tokens. Running the two print calls produces exactly:

Rohan booked a <extra_id_0> Mumbai to Pune using IRCTC and paid with UPI before the deadline this morning
<extra_id_0> train ticket from <extra_id_1>

The first line is what the encoder reads — 18 tokens, three shorter than the original because the removed span cost 3 tokens but its sentinel cost only 1. The second line is everything the decoder must produce — just 5 tokens, versus 20 for the full original sentence. That gap matters at training-set scale: reconstructing only the corrupted spans, rather than the whole sequence the way a text-generating BERT-style objective would, makes each training step cheaper and keeps the target sequence short even when the source document is long. Note the trailing <extra_id_1>: it is not decorative. If a second span had been corrupted, its content would immediately follow, opened by that same sentinel; since there is no second span here, <extra_id_1> instead functions as the terminator — the signal telling the decoder to stop generating.

From denoising to downstream tasks: the text-to-text framing

Pretraining teaches the network to reconstruct missing spans; it does not by itself know how to translate or classify. T5 gets task behavior the same way every transfer-learning transformer does — supervised fine-tuning on downstream data — but frames every downstream task's input and output as plain text, prefixed with a short string naming the task. Translating a sentence becomes the input string "translate English to German: That is good." with target "Das ist gut." Judging whether a sentence is grammatically acceptable (the CoLA task) becomes the input "cola sentence: Anyone who has visited Paris will tell you it's beautiful." with the target simply the word "acceptable" or "unacceptable". Even regression is coerced into text: the STS-B benchmark asks for a sentence-similarity score between 0 and 5 with one decimal place, and T5 handles it by rounding the target to the nearest 0.2 and emitting it as a string like "3.8" — the model never sees a regression loss; it is still doing next-token prediction over a vocabulary that happens to contain the digits 0–9 and a decimal point.

Because every task now shares one input/output format, T5 can be fine-tuned on many tasks at once by mixing their examples into a single training stream, rather than fine-tuning one checkpoint per task. Naively mixing in proportion to each dataset's size fails in practice: a machine-translation corpus with tens of millions of sentence pairs would swamp a small classification dataset with a few thousand examples, so the model would barely see the small task during training. T5 uses example-proportional mixing with a temperature: each dataset's sampling probability is raised to the power 1/T for a temperature T > 1, which flattens the distribution and gives small datasets more relative weight than their raw size would earn — the same trick that would apply if a multilingual Indian-language pipeline mixed a large Hindi corpus with much smaller Marathi, Tamil, or Bengali corpora and needed the low-resource languages to not disappear from training entirely.

What the systematic study actually found

The T5 paper is as well known for the scale of its ablation study as for the model itself: holding compute roughly fixed, the authors swept architecture (encoder-decoder vs. decoder-only vs. a "prefix-LM" that is decoder-only but allows bidirectional attention within the input segment), pretraining objective (BERT-style masking, sentence deshuffling, prefix-based language modeling, and span corruption at several corruption rates and span lengths), unlabeled dataset (C4 versus unfiltered Common Crawl versus Wikipedia versus news-only text), and model size, one variable at a time. Three findings carry forward. Encoder-decoder outperformed decoder-only and prefix-LM architectures at matched parameter count on most tasks, confirming that bidirectional encoding of the source before generation begins is worth its extra parameters. Span corruption beat both BERT-style per-token masking and plain language modeling as a pretraining objective, and the exact corruption rate and span length mattered less than the choice to corrupt contiguous spans at all — several nearby rates performed within a narrow band of each other, while token-level masking without span structure was consistently worse. And scale dominated every other lever: given a fixed compute budget, training a bigger model for fewer steps beat training a smaller model for more steps — the same lesson that motivated releasing T5 at five sizes together (Small, 60M parameters; Base, 220M; Large, 770M; 3B; and 11B), each roughly an order of magnitude apart, specifically so the scaling trend could be read off directly instead of pieced together from unrelated papers.

Common misconception: task prefixes are not zero-shot prompts

Because "translate English to German: That is good." looks exactly like a natural-language instruction, it's tempting to assume T5 works the way instruction-tuned chat models do — that you can hand it any prefix you invent and it will improvise a sensible response, the way a model trained for instruction-following handles a novel phrasing it has never seen before. That is not what is happening. T5's prefixes are fixed strings chosen by the researchers and paired with specific supervised datasets during fine-tuning; the model learns "the string that starts with cola sentence: is answered from the CoLA label vocabulary" the same way it learns any other input–output mapping from labeled examples, not by inferring intent from the words in the prefix. Feed T5 a prefix it never saw fine-tuning data for — say, "list three factors affecting monsoon rainfall:" — and there is no guarantee of a coherent answer, because nothing in training associated that string with that behavior. The instruction-following generalization that makes prefixes feel like natural-language commands is a capability of later models trained explicitly for it, with instruction tuning and human-feedback fine-tuning; T5's prefixes are closer to a routing key selected in advance than a request understood on the fly.

Active recall

Attempt these before reading the answers.

1. Why can T5 handle classification, regression, and generation with the same model and no added parameters, while BERT needs a new head per task?
2. In the worked example, the target ended with <extra_id_1> even though only one span was corrupted. What job is that token doing?
3. Why does T5 use a full encoder-decoder rather than a decoder-only architecture, given decoder-only models can also be trained on text-to-text data?
4. A classmate says: "T5's prefixes prove it understands instructions, just like a modern instruction-tuned chat model." What's the flaw in that claim?
5. A document has 600 tokens. Using T5's default corruption rate of 15% and mean span length of 3, estimate the number of spans corrupted, the encoder input length, and the decoder target length.
6. Why does span corruption train faster per step than a hypothetical "BERT, but it generates text" objective that reconstructed the entire original sequence?

Answers

1. Because T5's output is never routed through a task-specific head — every task's answer is generated as text from the same decoder softmax over the same fixed vocabulary. The differences between tasks live entirely in the input text (the task prefix) and the target text, not in the model's parameters or architecture, so no new weights are ever added for a new task.
2. <extra_id_1> is the sentinel that would open the next corrupted span. Since there is no next span here, it functions as an end-of-target marker — the only signal telling the decoder to stop generating for this example. Without it, the model has no learned reason to halt after "from".
3. The encoder can attend bidirectionally over the entire source before generation begins, so even the first generated token benefits from context that appears later in the source sentence. A decoder-only model processes source and target in one causal stream, so earlier source tokens are encoded without ever seeing later source context. The paper's architecture ablation found encoder-decoder outperformed decoder-only and prefix-LM variants at matched parameter count on most tasks.
4. T5's prefixes are fixed strings tied to specific datasets seen during supervised fine-tuning — the model learns a lookup from "this exact prefix" to "this label format," not general instruction comprehension. An instruction-tuned chat model is explicitly trained on a huge variety of differently-phrased instructions so it generalizes to novel ones; T5 was never trained for that and gives no such guarantee on a prefix it hasn't seen fine-tuning examples for.
5. Corrupted tokens: 0.15 × 600 = 90. Number of spans: 90 ÷ 3 = 30. Encoder input length: 600 total tokens − 90 removed + 30 sentinels (one per span) = 540 tokens. Decoder target length: 90 span tokens + 31 sentinels (one opening each of the 30 spans, plus one final terminator) = 121 tokens.
6. The target sequence only contains the corrupted spans and their sentinels, not the untouched majority of the sentence, so the decoder produces far fewer tokens per example (5 out of 20 in the worked example, roughly a quarter) than a full-sequence reconstruction objective would. Fewer target tokens means fewer autoregressive decoding steps and a shorter loss computation per training example, so more examples can be processed for the same compute budget.