Multi-Task Learning: Shared Representations

One dispatch engine, three predictions

A food-delivery platform operating at Swiggy or Zomato scale has to answer three separate questions for every order the moment it is placed: how many minutes will this delivery take, what is the probability this order gets cancelled before the rider reaches the restaurant, and what is the probability this customer reorders from the same restaurant within the next seven days. The obvious engineering plan is three separate deep networks, one per question, each trained end to end on its own labelled data. That plan has a real cost. All three questions depend on the same underlying facts about the order: how far the restaurant is from the customer, how backed up the kitchen currently is, what time of day it is, how many riders are free nearby. Three independent networks each have to rediscover these regularities from scratch, using only their own labels, and their own labels are not equally generous. Every order eventually has a ground-truth delivery time, so the ETA network sees millions of clean examples a week. Cancellations are rarer, and "reorders within seven days" is rarer still, maybe three orders in a hundred. A network trained only on that thin positive signal tends to either overfit to noise or fail to learn a feature space rich enough to generalise at all.

Multi-task learning (MTL) with shared representations is the architectural answer to this exact situation: build one encoder that turns the raw order features into a shared internal representation, and let each task attach its own small, task-specific head on top of that shared representation. The encoder is shaped by the gradients from all three losses at once, not by any one of them alone. This chapter works out precisely what that sharing does to the mathematics of training, where it helps, where it actively hurts, and how to read the difference off a real gradient computation.

From independent networks to a shared encoder

In ordinary single-task supervised learning you fit a function f_t(x; θ_t) that maps input features x to a prediction for one task t, minimising a loss L_t over parameters θ_t that belong to that task alone. With T tasks trained independently you have T completely separate parameter sets and T completely separate optimisation problems; nothing learned while fitting the ETA network is available to the cancellation network.

Hard-parameter-sharing MTL restructures this. Define a shared encoder g(x; θ_shared) that produces an internal representation z = g(x; θ_shared), common to every task. Each task then applies its own head h_t(z; θ_t) to produce its prediction ŷ_t = h_t(g(x; θ_shared); θ_t). Training minimises a single combined objective:

L(θ_shared, θ_1, ..., θ_T) = Σ_t λ_t · L_t(h_t(g(x; θ_shared); θ_t), y_t)

where λ_t is a scalar weight for task t. The parameters θ_shared receive a gradient contribution from every task in the sum; each θ_t receives a gradient only from its own task's loss, since the other tasks' losses do not pass through θ_t at all. This asymmetry, one set of parameters pooling signal from everything, task-specific sets isolated to their own signal, is the entire mechanism of shared-representation learning. Everything interesting in this chapter, both the benefit and the failure mode, comes from what happens when several tasks' gradients land on the same shared parameters simultaneously.

There is a second family, soft parameter sharing, where each task keeps its own full parameter set but a regularisation term penalises the distance between corresponding parameters across tasks (cross-stitch networks are a well-known example). It avoids some interference problems at the cost of roughly T times the parameter count. This chapter focuses on hard sharing, since it is the dominant industrial pattern and the one whose gradient arithmetic is easiest to trace exactly.

Why sharing helps: one representation, more signal per parameter

The argument for hard sharing is a statistical-efficiency argument, not a computational-convenience one, though it delivers convenience too. A feature detector inside θ_shared, say, a unit that has learned to estimate "how backed up is this kitchen right now" from order-count and prep-time history, is useful for predicting ETA directly, and it is also useful for predicting cancellation, because customers cancel more often when the wait grows long, and it is weakly useful for predicting reorders, because a smooth first experience predicts repeat behaviour. When the ETA task's abundant, clean gradient signal pushes the shared encoder to build a good kitchen-congestion feature, the cancellation and reorder heads get that feature for free; they only have to learn a thin mapping from an already-informative z to their own output, rather than learning the whole representation from their own scarce labels. Rich Caruana's original 1997 formulation of multi-task learning called this effect eavesdropping: a data-poor task listens in on features that a data-rich, related task was forced to discover. The shared representation also acts as an implicit regulariser. A representation that has to serve three different objectives at once cannot overfit to idiosyncrasies of any single one of them; anything the encoder learns has to survive contact with all three losses, which rules out a wide class of spurious correlations that a single-task network would happily memorise.

Where sharing hurts: negative transfer and gradient conflict

Nothing in the combined objective guarantees that the tasks agree on what the shared representation should look like. If task A's gradient on θ_shared points one way and task B's gradient points the opposite way, the shared parameter update is their sum, which can be smaller than either gradient alone, or can move the shared representation somewhere that is mediocre for both tasks rather than good for either. This is called negative transfer, and it is the central risk that any real MTL system has to manage, not a rare edge case. The general diagnostic is the cosine similarity between two tasks' gradient vectors on the shared parameters, cos θ = (g_A · g_B) / (‖g_A‖ ‖g_B‖). A positive cosine means the tasks are pulling the representation in broadly compatible directions (constructive, positive transfer); a negative cosine means they are pulling against each other (destructive, negative transfer). The worked example below computes this quantity directly and shows both effects arising from the same two-task setup, just with different head weights.

Worked example: tracing one gradient step through a shared weight

Strip the dispatch network down to the smallest system that still has the property we care about: one shared scalar weight feeding two task heads. This keeps every arithmetic step checkable by hand while preserving exactly the mechanism that matters.

Shared layer:       z   = w · x                (one shared feature)
ETA head:           ŷ_A = a · z                (task A: predicted extra minutes)
Cancellation head:  ŷ_B = b · z                (task B: predicted cancellation risk)
Per-task loss:       L_A = ½(ŷ_A − y_A)²,  L_B = ½(ŷ_B − y_B)²   (MSE, for clean derivatives)
Combined loss:        L   = L_A + L_B           (λ_A = λ_B = 1 here)

Take a single normalised congestion reading x = 1.5, a shared weight currently at w = 0.8, head weights a = 1.2 (ETA) and b = −0.5 (cancellation, negative because this head currently associates high congestion with slightly lower measured cancellation risk in its training data), and targets y_A = 2.0 minutes and y_B = 0.4.

Step 1 — forward pass through the shared layer
  z = w·x = 0.8 × 1.5 = 1.2

Step 2 — forward pass through each head
  ŷ_A = a·z = 1.2 × 1.2 = 1.44
  ŷ_B = b·z = −0.5 × 1.2 = −0.6

Step 3 — per-task error and loss
  e_A = ŷ_A − y_A = 1.44 − 2.0  = −0.56    L_A = ½(−0.56)² = 0.1568
  e_B = ŷ_B − y_B = −0.6 − 0.4  = −1.00    L_B = ½(−1.00)² = 0.5000
  L   = L_A + L_B = 0.6568

Step 4 — gradient of each task's loss w.r.t. the SHARED weight w
  chain rule: dL_t/dw = (dL_t/dŷ_t) · (dŷ_t/dz) · (dz/dw) = e_t · (head weight) · x
  dL_A/dw = e_A · a · x = (−0.56)(1.2)(1.5) = −1.008
  dL_B/dw = e_B · b · x = (−1.00)(−0.5)(1.5) = 0.750

Step 5 — combined gradient at the shared weight
  dL/dw = dL_A/dw + dL_B/dw = −1.008 + 0.750 = −0.258

Step 6 — one gradient-descent step, learning rate η = 0.1
  w_new = w − η·(dL/dw) = 0.8 − 0.1×(−0.258) = 0.8258

Read step 4 before moving to step 5. Task A's gradient on w is negative, so gradient descent increases w: raising the shared weight raises z, which raises ŷ_A toward its target of 2.0, exactly what task A wants. Task B's gradient on w is positive, so gradient descent decreases w: because b is negative, a larger z pushes ŷ_B further from its positive target of 0.4, so task B wants w to shrink. The two tasks disagree about which direction the shared weight should move, and because there is only one shared parameter here, "disagreement" reduces to the two derivatives having opposite sign, which is the one-dimensional special case of a negative cosine similarity: cos θ = −1 exactly, full conflict. Task A's gradient has the larger magnitude (1.008 against 0.750), so it wins the tug-of-war and w ends up nudged upward to 0.8258, a compromise that partially serves task A and leaves task B's error uncorrected this step.

Compare this against what would have happened with two fully independent single-task networks starting from the same w = 0.8: task A alone would update to w_A = 0.8 − 0.1×(−1.008) = 0.9008, and task B alone would update to w_B = 0.8 − 0.1×(0.750) = 0.725. Left independent, the two networks drift to visibly different weights, each fully optimised for its own objective. Shared, they are forced to a single compromise value, 0.8258, that lies between the two, closer to task A because its gradient was stronger. That compromise is precisely the mechanism by which MTL regularises: it is also precisely the mechanism by which MTL can degrade a task, if the compromise value is far enough from what that task alone would have chosen. Whether the net effect is helpful or harmful depends on how correlated the tasks' true underlying functions are, not on anything the optimiser can fix by itself, which is why real systems weight tasks (the λ_t terms) rather than trusting an unweighted sum. GradNorm (Chen et al., 2018) rescales each task's gradient magnitude to a common norm before summing; uncertainty weighting (Kendall, Gal and Cipolla, 2018) learns each λ_t from a per-task noise parameter; PCGrad (Yu et al., 2020) detects exactly the negative-cosine condition derived above and projects away the conflicting component of one task's gradient before it is added to the other's.

Architecture in code: hard parameter sharing

The three-task dispatch network described in the opening scenario is a direct scale-up of the toy example above: one shared trunk, three linear output heads, one combined loss.

# assumes: import torch; import torch.nn as nn; import torch.nn.functional as F

class DispatchMultiTaskNet(nn.Module):
    def __init__(self, input_dim, shared_dim):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, shared_dim),
            nn.ReLU(),
            nn.Linear(shared_dim, shared_dim),
            nn.ReLU(),
        )                                        # θ_shared
        self.eta_head = nn.Linear(shared_dim, 1)      # θ_A: regression, minutes
        self.cancel_head = nn.Linear(shared_dim, 1)   # θ_B: logit, cancellation
        self.reorder_head = nn.Linear(shared_dim, 1)  # θ_C: logit, reorder in 7 days

    def forward(self, x):
        z = self.encoder(x)                       # shared representation
        eta = self.eta_head(z)
        cancel_logit = self.cancel_head(z)
        reorder_logit = self.reorder_head(z)
        return eta, cancel_logit, reorder_logit


def multitask_loss(eta_pred, cancel_logit, reorder_logit,
                    eta_true, cancel_true, reorder_true,
                    lambda_eta=1.0, lambda_cancel=1.0, lambda_reorder=2.0):
    loss_eta = F.mse_loss(eta_pred.squeeze(-1), eta_true)
    loss_cancel = F.binary_cross_entropy_with_logits(cancel_logit.squeeze(-1), cancel_true)
    loss_reorder = F.binary_cross_entropy_with_logits(reorder_logit.squeeze(-1), reorder_true)
    return lambda_eta * loss_eta + lambda_cancel * loss_cancel + lambda_reorder * loss_reorder

# illustrative call only — x, eta_true, cancel_true, reorder_true are batched
# tensors assumed already loaded from the order-features pipeline, not shown here:
#   eta_pred, cancel_logit, reorder_logit = model(x)
#   loss = multitask_loss(eta_pred, cancel_logit, reorder_logit,
#                          eta_true, cancel_true, reorder_true)
#   loss.backward()

lambda_reorder is set higher than the other two, 2.0 against 1.0, precisely because the reorder task is the data-poor one: with only a few positive labels per hundred orders, its raw gradient magnitude through encoder would otherwise be drowned out by the ETA task's dense, high-magnitude signal, the same dominance effect measured directly in step 5 of the worked example above.

Architecture at a glance

The diagram below shows the same structure the code implements: one encoder producing a shared representation z, three heads consuming it, and, running the opposite direction, the three per-task gradients converging back on the branch point before they enter the shared encoder as a single summed update. That convergence point is where positive and negative transfer are decided.

The misconception this chapter has to correct

The natural first impression, especially coming straight from single-task deep learning, is that multi-task learning is a free upgrade: one network, several outputs, no reason it could ever be worse than training separately. Step 4 through step 6 of the worked example prove this false directly. When two tasks' gradients on a shared parameter have opposite sign, as they did there, the shared update is not "both improvements added together," it is a subtraction that leaves the weaker task's error only partially corrected, or in more severe real cases, actually increased. This is negative transfer, and it is not an implementation bug, it is the correct behaviour of gradient descent on a genuinely conflicting combined objective. Whether a given pair of tasks helps or hurts each other through a shared representation is an empirical property of how related their underlying functions are, and it has to be checked, typically by comparing validation performance of the multi-task model against single-task baselines for each task separately, not assumed from the fact that they share input features. A platform that shares the dispatch encoder across ETA, cancellation, and reorder prediction should expect ETA (dense, well-estimated) to be roughly stable or slightly regularised, and should specifically watch whether the sparse reorder task's validation metric improves relative to a reorder-only baseline, because that is the task most exposed to negative transfer if the loss weights are wrong.

Active recall

Work through all six before reading the answers below.

1. In hard parameter sharing, which parameters receive a gradient contribution from every task's loss, and which receive a gradient from only one task's loss?
2. Using the same toy network as the worked example, but with shared weight w = 0.5, input x = 2, head weights a = 0.9 (target y_A = 2.5) and b = 0.4 (target y_B = 0.3), compute z, both predictions, both errors, both gradients on w, the combined gradient, and the updated w after one gradient-descent step with η = 0.2.
3. Why does a rare-label task such as seven-day reorder prediction typically gain more from being folded into a shared encoder than a data-rich task such as ETA regression does?
4. Two tasks' gradients on a shared parameter vector are g_A = [0.4, −0.2] and g_B = [−0.1, 0.3]. Compute the cosine similarity between them and state whether this indicates positive or negative transfer.
5. True or false, with justification: for best results you should always share every layer of the network across all tasks (full hard sharing), never leaving any layer task-specific.
6. Two tasks sharing one encoder are weighted equally, λ_A = λ_B = 1, but task A's loss and gradient magnitudes are consistently much larger than task B's. What happens to the shared representation as a result, and name one standard fix.

Worked answers

1. The shared encoder parameters θ_shared receive a gradient contribution summed across every task's loss, since every task's prediction passes through g(x; θ_shared). Each task's own head parameters θ_t receive a gradient only from that task's own loss L_t, because no other task's loss passes through h_t.
2. z = w·x = 0.5 × 2 = 1.0. ŷ_A = a·z = 0.9 × 1.0 = 0.9, so e_A = 0.9 − 2.5 = −1.6. ŷ_B = b·z = 0.4 × 1.0 = 0.4, so e_B = 0.4 − 0.3 = 0.1. dL_A/dw = e_A·a·x = (−1.6)(0.9)(2) = −2.88. dL_B/dw = e_B·b·x = (0.1)(0.4)(2) = 0.08. Combined gradient: −2.88 + 0.08 = −2.80. Update: w_new = 0.5 − 0.2×(−2.80) = 0.5 + 0.56 = 1.06. The two per-task gradients again have opposite sign (−2.88 versus +0.08), the same conflict pattern as the main worked example, but task A's magnitude so overwhelms task B's here that the combined update is barely different from what task A alone would have produced.
3. The sparse task's own gradient signal, drawn from only a handful of positive labels, is too noisy and too limited to build a good representation on its own; a network trained on it alone tends to overfit. When it shares an encoder with a data-rich, related task, the abundant task's strong, well-estimated gradient does most of the work of shaping θ_shared into a genuinely useful feature space, and the sparse task's head only has to learn a thin mapping on top of an already-good representation rather than discover that representation itself. This is the eavesdropping effect Caruana described in the original 1997 MTL formulation.
4. Dot product: (0.4)(−0.1) + (−0.2)(0.3) = −0.04 − 0.06 = −0.10. Norms: ‖g_A‖ = √(0.16+0.04) = √0.20 ≈ 0.4472, ‖g_B‖ = √(0.01+0.09) = √0.10 ≈ 0.3162. cos θ = −0.10 / (0.4472 × 0.3162) = −0.10 / 0.1414 ≈ −0.7071, which is exactly −1/√2, corresponding to an angle of 135°. A negative cosine means the two gradients point in substantially opposing directions on the shared parameters: this is a case of gradient conflict, indicating negative-transfer risk if both are summed unmodified.
5. False. Sharing every layer assumes every task needs an identical representation all the way to the output, which is rarely true; later layers tend to be more task-specific by nature (closer to each task's own output distribution), and forcing them to be shared invites exactly the kind of gradient conflict shown in the worked example. The common, empirically motivated pattern is to share early, general-purpose layers and give each task more of its own dedicated capacity closer to the output, then validate per-task performance against single-task baselines rather than assuming maximum sharing is optimal.
6. Because the combined update on θ_shared is a raw sum of the per-task gradients, the task with the larger gradient magnitude dominates that sum and pulls the shared representation disproportionately toward its own objective, exactly as task A's gradient of magnitude 1.008 outweighed task B's 0.750 in the main worked example even before any weighting was applied. Left uncorrected, the weaker task is systematically undertrained. Standard fixes include manually raising that task's λ_t, uncertainty-based weighting that learns λ_t from a per-task noise estimate (Kendall, Gal and Cipolla, 2018), or gradient-magnitude normalisation such as GradNorm (Chen et al., 2018), which rescales each task's gradient to a common norm before the sum is taken.

Think About It

Think about this: How would you explain multi-task learning: shared representations to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where multi-task learning: shared representations is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting multi-task learning: shared representations to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind multi-task learning: shared representations, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.