Few-Shot Learning Techniques

ISRO's Chandrayaan orbiter returns hyperspectral images of the lunar surface faster than anyone can label them. A given patch of regolith might be basalt, anorthosite, or olivine, but confirming which one requires an actual spectroscopic reading from an instrument like the Pragyan rover's Alpha Particle X-ray Spectrometer — and across an entire mission that yields, at best, a few dozen confirmed samples per mineral, not the tens of thousands of labeled images a convolutional network normally needs to learn a category from scratch. New craters get photographed every orbit, and every new patch needs a mineral label today, from a labeled set that will never grow past a handful of examples per class — landing another rover to collect "more data" is not an option on a semester's notice. This is not a data-engineering inconvenience to be fixed by scraping more images. It is exactly the constraint few-shot learning was built for: given only K labeled examples of a class the model has never trained on, classify a new example correctly, and do it well from the very first attempt.

From "big data" supervision to N-way K-shot classification

A standard image classifier — the kind you have already trained with a CNN — is trained and tested on the same fixed set of classes, using thousands of labeled examples per class. Few-shot learning changes both halves of that setup. The classes seen at test time are novel: they never appeared during training. And the number of labeled examples per novel class, K, is small — typically 1 to 5. The standard way to pose this precisely is N-way K-shot classification: at test time you are handed a support set of N classes with K labeled examples each (N × K examples total), plus a query example belonging to one of those N classes, and your job is to say which one. Two labeled basalt patches, two anorthosite patches, two olivine patches, and one unlabeled patch to classify — that is 3-way, 2-shot.

The support set is not a training set in the ordinary sense — at test time there is little or no gradient descent run on it. The heavy lifting happened earlier, during meta-training: the model was trained not on the three lunar minerals it will eventually see, but on hundreds of unrelated, abundant image categories — terrestrial rock types, industrial material textures, whatever labeled data exists in bulk — with the eventual test classes held out completely. During meta-training, the model is shown thousands of simulated N-way K-shot "episodes." Each episode samples a fresh random group of N classes from the training pool, builds a support set and a query set from those classes, and scores the model on how well it classifies the query set using only the support set. Across thousands of such episodes, spanning thousands of different class groupings that never repeat, the model is forced to learn something more general than "what basalt looks like" — it learns how to tell classes apart from a handful of examples, for classes it has never seen. That learned skill, not any memorized label, is what transfers to the three real lunar minerals at test time.

Metric-based few-shot learning: Prototypical Networks

The cleanest way to turn "how to tell classes apart from a few examples" into an algorithm is to stop trying to learn a classifier at all, and learn a distance function instead. This is the idea behind Prototypical Networks (Snell, Swersky & Zemel, 2017):

An embedding network f_θ — a CNN encoder, meta-trained exactly as described above — maps every image, support or query, into a fixed-size embedding vector. For each class k in the current episode's support set, compute its prototype: the mean of the embeddings of that class's K support examples.

c_k = (1/K) * Σ f_θ(x_i)     for every support example x_i of class k

To classify a query example x, embed it with the same f_θ, then measure its squared Euclidean distance to every class prototype. Convert distances into a probability distribution with a softmax over the negative distances, so a closer prototype gets a higher probability:

p(y = k | x) = exp(-d(f_θ(x), c_k)) / Σ_j exp(-d(f_θ(x), c_j))

Notice what does not happen at test time: no weight updates, no backpropagation on the support set, no risk of the network overfitting to three lunar rock images. The K examples only ever get averaged into a single vector, the prototype. Everything the model "knows" about telling minerals apart lives frozen inside f_θ, learned during meta-training on hundreds of unrelated classes. Snell et al. also report a detail worth remembering precisely because it is counterintuitive: squared Euclidean distance in the embedding space consistently beats cosine distance for this method, even though cosine similarity is the more familiar choice for comparing embeddings elsewhere in deep learning. The reason is structural — the class mean under squared Euclidean distance corresponds to the maximum-likelihood cluster center for a Gaussian (or more generally, any Bregman-divergence exponential-family) model, which makes the prototype a mathematically well-justified summary of the class. Cosine distance does not have that property, so replacing the K examples with their mean is no longer principled once you switch metrics.

A closely related, older idea worth knowing by name is the Siamese network (Koch et al., 2015): instead of computing class prototypes, it directly learns a similarity score between pairs of examples, and is the natural fit for 1-shot verification rather than classification — matching a scanned signature against a single specimen on file, for instance, rather than choosing among several candidate classes. Matching Networks (Vinyals et al., 2016) sit between the two: they compare the query to every individual support example (not a class mean) using an attention-weighted similarity, which can help when a class's support examples are not tightly clustered, at the cost of more computation per query. Prototypical Networks' simplification — replace the whole support set with one mean vector per class — turns out to work at least as well on the standard benchmarks for K > 1, which is why it remains the default starting point for metric-based few-shot learning.

Worked example: classifying a lunar rock patch, prototype by prototype

Suppose the meta-trained encoder f_θ has already reduced every image patch to a 2-dimensional embedding (real systems use hundreds of dimensions; two is enough to trace every step by hand). This is a 3-way, 2-shot episode.

Step 1 — support embeddings. The encoder produces:

Basalt:       s1 = (1.0, 2.0)   s2 = (1.4, 1.8)
Anorthosite:  s3 = (4.0, 1.0)   s4 = (3.6, 1.4)
Olivine:      s5 = (2.0, 5.0)   s6 = (2.4, 4.6)

Step 2 — compute the three prototypes (coordinate-wise mean of each class's two support embeddings):

c_Basalt      = ((1.0+1.4)/2, (2.0+1.8)/2) = (1.2, 1.9)
c_Anorthosite = ((4.0+3.6)/2, (1.0+1.4)/2) = (3.8, 1.2)
c_Olivine     = ((2.0+2.4)/2, (5.0+4.6)/2) = (2.2, 4.8)

Step 3 — embed the query. A new, unlabeled crater-wall patch is embedded by the same f_θ as q = (1.6, 2.2).

Step 4 — squared Euclidean distance from q to each prototype.

d(q, c_Basalt)^2      = (1.6-1.2)^2 + (2.2-1.9)^2 = 0.16 + 0.09 = 0.25
d(q, c_Anorthosite)^2 = (1.6-3.8)^2 + (2.2-1.2)^2 = 4.84 + 1.00 = 5.84
d(q, c_Olivine)^2     = (1.6-2.2)^2 + (2.2-4.8)^2 = 0.36 + 6.76 = 7.12

Step 5 — softmax over the negative distances. exp(-0.25) ≈ 0.77880, exp(-5.84) ≈ 0.0029088, exp(-7.12) ≈ 0.0008088. Their sum is ≈ 0.7825184, so:

p(Basalt)      ≈ 0.77880 / 0.78252 ≈ 0.9952
p(Anorthosite) ≈ 0.00291 / 0.78252 ≈ 0.0037
p(Olivine)     ≈ 0.00081 / 0.78252 ≈ 0.0010

The query is classified as basalt with about 99.5% confidence — unsurprising once you notice its squared distance to the basalt prototype (0.25) is more than twenty times smaller than its distance to either other prototype. The same computation as runnable code:

import numpy as np

support = {
    "Basalt":       np.array([[1.0, 2.0], [1.4, 1.8]]),
    "Anorthosite":  np.array([[4.0, 1.0], [3.6, 1.4]]),
    "Olivine":      np.array([[2.0, 5.0], [2.4, 4.6]]),
}

# Step 1-2: prototype = mean embedding per class
prototypes = {c: v.mean(axis=0) for c, v in support.items()}

# Step 3: query embedding, from the same encoder f_theta
query = np.array([1.6, 2.2])

# Step 4: squared Euclidean distance to every prototype
dists = {c: float(np.sum((query - p) ** 2)) for c, p in prototypes.items()}

# Step 5: softmax over the negative distances
classes = list(dists.keys())
neg_d = np.array([-dists[c] for c in classes])
probs = np.exp(neg_d) / np.sum(np.exp(neg_d))

for c, p in zip(classes, probs):
    print(f"{c}: {p:.4f}")

Output:

Basalt: 0.9952
Anorthosite: 0.0037
Olivine: 0.0010

The diagram below traces this exact episode end to end: the raw support patches on the left, the shared frozen encoder in the middle, the embedding space with the three prototypes and the query plotted using the coordinates above, and the resulting softmax probabilities on the right.

Optimization-based few-shot learning: MAML

Model-Agnostic Meta-Learning (Finn, Abbeel & Levine, 2017) reaches the same goal by an entirely different route. Instead of learning a fixed embedding and comparing distances, it learns an initialization θ for an ordinary neural network — one that sits only a few gradient steps away from a good classifier for any new task drawn from the same task distribution. Meta-training is a nested, bi-level optimization. For each sampled episode T_i, an "inner loop" takes one or a few ordinary gradient-descent steps on that episode's support set, starting from the current θ:

θ'_i = θ - α * ∇_θ L_Ti(θ)

The adapted parameters θ'_i are then evaluated on that same episode's query set, and the resulting query loss — still viewed as a function of the original θ, through the inner-loop update above — drives an "outer loop" update to θ itself:

θ ← θ - β * ∇_θ Σ_i L_Ti(θ'_i)

The outer-loop gradient passes through the inner-loop gradient step, which means computing it involves a second derivative of the loss. That extra derivative is precisely what makes MAML's initialization special: θ is optimized explicitly to be a starting point from which a handful of ordinary gradient steps, on any new task's small support set, lands close to a good classifier for that task. At test time on the three lunar minerals, you take the meta-trained θ, run one or two ordinary gradient steps against the 3×K support examples, and evaluate the result on the query. Unlike Prototypical Networks, this genuinely does use gradient descent on the support set at test time — but only one or two steps, from a starting point specifically trained to make that safe, in contrast to the hundreds of steps ordinary fine-tuning would use (and which would overfit badly with only a handful of examples).

MAML's appeal is generality: because the inner loop is just gradient descent on whatever loss you supply, MAML applies to any differentiable model and task type — classification, regression, even reinforcement-learning control — whereas Prototypical Networks are specific to classification via distance comparison. The cost is compute: the second derivative through the inner loop is expensive to backpropagate, which is why cheaper first-order approximations (foMAML, and the related Reptile algorithm) exist as practical, if slightly less precise, substitutes in production systems.

A third route: in-context few-shot learning

Large pretrained language models perform a strikingly different kind of few-shot learning, one you have already met in the NLP unit. Place K labeled examples directly in the text prompt — input, correct label, input, correct label, and so on — followed by a new unlabeled input, and the model predicts a continuation that supplies that input's label, with zero gradient updates and zero weight changes at inference time. All the adaptation happens through the forward pass alone, conditioning on the examples sitting in the context window. This is called in-context learning, and it works because the pretraining objective — next-token prediction over enormous, varied text — implicitly forces the model to become good at pattern completion across the wildly different local "mini-tasks" it encounters within long documents (a table being filled in, a Q&A thread, a translated pair of sentences). That general skill ends up transferring to explicit few-shot prompts at inference time, even though no episodic few-shot objective of the kind Prototypical Networks or MAML use was ever part of training.

It is worth being precise about what genuinely differs here. Prototypical Networks and MAML are meta-learning methods: a specific training procedure — episodic training, or bi-level optimization — engineered to produce a model built for few-shot adaptation. In-context learning is an emergent capability of ordinary large-scale language-model pretraining; no few-shot-specific objective produced it. All three share a goal — perform well given only K examples of something not seen in that exact form during training — but arrive by mechanically different means. In NLP practice today, "few-shot learning" most often means prompting a large model, not running episodic meta-training at all; it is a mistake to assume the word always implies the Snell/Finn-style machinery.

Common misconception

The name invites a specific, wrong reading: "few-shot learning means training a model with only a few labeled examples." That is not what makes any of these three techniques work. In Prototypical Networks and MAML, the K examples of the novel class only ever appear at test time — as a support set to average into a prototype, or to power one or two adaptation steps starting from a specially-trained initialization. In-context learning likewise sees its K examples only inside a test-time prompt. The actual training that makes few-shot classification possible — meta-training across hundreds of disjoint classes, or pretraining on internet-scale text — happens beforehand, on plenty of data, and is what the "few" examples at test time are able to lean on.

Test this against the failure case directly: take a plain, randomly initialized CNN and try to fit it straight to the three lunar-mineral classes with two support images each, using ordinary backpropagation until the training loss on those six images goes to zero. It will fail at classifying the query, not because six images are inherently too little information in principle, but because an untrained encoder has no useful embedding space yet, and gradient descent on six images with millions of free parameters will happily drive loss to zero by memorizing those six images' idiosyncratic pixels — sensor noise, exact lighting, the precise crop — rather than learning anything that generalizes to a seventh image of the same mineral. This is a straightforward capacity-versus-sample-size mismatch, the same overfitting risk you already know from ordinary supervised learning, just pushed to its extreme at K=1 or K=2. Prototypical Networks sidestep it entirely by never introducing new trainable parameters at test time — the "learning" already happened, once, during meta-training on classes that have nothing to do with lunar minerals. So the "few" in few-shot learning describes the labeled data available for the specific new classes at test time; it says nothing about how much data the overall method consumed to get there, and the amount consumed beforehand is usually large.

Active recall

Attempt each question before reading its answer.

1. In 5-way 1-shot classification, how many images make up the support set, and how does the support set differ in role from the query set?
2. A 2-way, 1-shot episode uses 1-dimensional embeddings. Class P's single support example embeds to 0.0; class Q's to 3.0. A query embeds to 1.0. Using the Prototypical Network rule, compute p(P | query) and p(Q | query).
3. Why does fine-tuning a full CNN's weights on K=1 example per class typically fail, while a Prototypical Network succeeds with the same K=1?
4. What is the key structural difference between MAML and Prototypical Networks in terms of what happens to model parameters at test time, when the support set is presented?
5. Why must the classes used during meta-training be strictly disjoint from the classes evaluated at meta-test time?
6. In-context few-shot learning in a large language model updates no weights at all. What actually produces the adaptation to the examples placed in the prompt?

Answers

1. The support set has 5 classes × 1 example = 5 labeled images total. The query set contains separate, unlabeled example(s) drawn from those same 5 classes; the model must classify the query using only what it can extract from the support set (plus whatever it learned during meta-training) — the support labels are the only supervision available for these particular classes.

2. With 1-dimensional embeddings and K=1, each prototype is simply the single support embedding: c_P = 0.0, c_Q = 3.0. Squared distances from the query (1.0): d(q,P)² = (1.0-0.0)² = 1.0; d(q,Q)² = (1.0-3.0)² = 4.0. Softmax over negative distances: exp(-1.0) ≈ 0.36788, exp(-4.0) ≈ 0.01832, sum ≈ 0.38620. So p(P|query) ≈ 0.36788/0.38620 ≈ 0.9525, and p(Q|query) ≈ 0.01832/0.38620 ≈ 0.0475. The query is classified as P with about 95% confidence, since its squared distance to c_Q (4.0) is four times its squared distance to c_P (1.0).

3. Fine-tuning updates every weight of the CNN to minimize loss on just 1 example per class. With millions of free parameters and only a handful of images, gradient descent memorizes the exact pixels of those images rather than learning a generalizable notion of the class, so it fails on any new image of the same class. A Prototypical Network introduces zero new trainable parameters at test time — the K support examples are only averaged into a prototype using an encoder that was already trained (during meta-training, on unrelated classes with abundant data) to produce a useful embedding space. There is nothing left to overfit.

4. Prototypical Networks: no parameter updates at all on the support set — a forward pass through the frozen encoder, followed by averaging and a distance computation. MAML: the meta-trained parameters θ are explicitly updated by one or a few ordinary gradient-descent steps on the support set (the inner loop), producing task-specific parameters θ' that are then used to classify the query.

5. If meta-training classes overlapped with meta-test classes, the model would already have seen labeled examples of the supposedly "novel" test classes during meta-training. Reported few-shot accuracy would then be inflated by ordinary memorization rather than measuring the model's ability to generalize to genuinely new categories — a data-leakage problem that would make the evaluation meaningless as a test of few-shot generalization.

6. The adaptation happens entirely within the forward pass: the transformer's self-attention layers let the model's prediction for the new query token(s) be conditioned on every labeled example present earlier in the context window, exploiting pattern-completion behavior learned during large-scale pretraining. No gradient is computed and no parameter is changed; the "learning" is really inference over the prompt, not training.

Think About It

Think about this: How would you explain few-shot learning techniques to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.