Multi-Agent Systems with Large Language Models

Suppose you are designing the AI layer behind customer support for a food-delivery platform operating at the scale of a major Indian app — lakhs of orders an hour, spanning order tracking, refunds, restaurant-side complaints, and delivery-partner escalations. The obvious first attempt is one large system prompt: a single LLM given every tool it might need — an order-lookup API, a refund-policy document, a delivery-tracking API, a payment-gateway API — and told to handle whatever the customer types. In testing, this single agent starts picking the wrong tool when several are plausible, loses track of the actual complaint by turn six of a long chat, and occasionally lets text pasted into a delivery-instructions field influence how it interprets refund policy, because everything — trusted policy, untrusted user text, and tool outputs — lives in the same undifferentiated context. None of this is a model-quality problem you can fix by upgrading to a better checkpoint. It is an architecture problem: one context window is being asked to hold too many roles, too many tools, and too many trust levels at once. The fix that production systems converge on is to stop asking one agent to do everything, and instead build a small society of narrower agents, each with its own role, its own tools, its own slice of context — coordinated by an explicit protocol. That is a multi-agent LLM system, and this chapter is about how such systems are actually built, why they help, and precisely where they fail.

From a single ReAct loop to a system of agents

Start with the single-agent case, because multi-agent systems are built out of it. Yao, Zhao, Yu, Du, Shafran, Narasimhan, and Cao's ReAct: Synergizing Reasoning and Acting in Language Models (2022) formalized the now-standard agent loop: the model emits a Thought (reasoning about what to do next), an Action (a tool call — search, calculator, database query), receives an Observation (the tool's return value), and repeats until it emits a final answer. This loop is the atomic unit every agent framework you will meet — AutoGPT-style agents, LangChain agents, function-calling APIs — is built from. A single ReAct agent is remarkably capable, but it has a structural ceiling that no amount of prompt engineering removes.

The first limit is the context window itself. Even models with very large context windows do not use that context uniformly: Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, and Liang's Lost in the Middle: How Language Models Use Long Contexts (2023) showed that retrieval accuracy is highest when the relevant fact sits near the start or end of the context and degrades measurably when it is buried in the middle — a real, reproducible property of transformer attention over long sequences, not an occasional glitch. A single agent juggling five tools, a long policy document, and a growing conversation history keeps pushing the facts it actually needs for the current turn further from the edges of its own context, and its accuracy on that turn degrades accordingly.

The second limit is tool interference. As the number of tool definitions in one prompt grows, the model's accuracy at picking the correct tool for the current subtask falls, because the decision is now made against a longer, more ambiguous menu instead of a short, unambiguous one.

The third limit is persona interference. A single system prompt that must be an empathetic support voice, a strict fraud auditor, and a precise refund calculator all at once tends to blur those roles under pressure — the same weights, asked to hold three incompatible stances simultaneously, produce a compromise that serves none of them well.

A multi-agent LLM system is the direct architectural response to these three limits. Formally, it is a set of agents {A1, ..., An}, where each agent is a triple of (a role — its system prompt and persona), (a memory — private, shared, or both), and (a tool set), all wrapping calls to an underlying LLM policy; plus a communication protocol (how agents exchange information — structured messages, natural-language chat turns, or a shared read/write store) and a coordination topology (who is allowed to talk to whom, and in what order). Splitting one overloaded agent into several narrower ones is not adding intelligence — it is redistributing the same underlying capability across smaller, cleaner contexts, each closer to the regime where the model performs best.

Coordination topologies

Four topologies cover most systems built in practice, and each has become associated with a specific published system.

MetaGPT encodes standard operating procedures — the kind of role handoffs a real software team uses — directly into agent prompts: a Product Manager agent writes a requirements document, an Architect agent turns it into a design, an Engineer agent writes code against that design, and a QA agent tests it, each reading and writing a shared document pool rather than free-form chat. AutoGen instead treats coordination as a conversation: a UserProxyAgent and one or more AssistantAgents exchange turns, with a designated termination signal (an explicit token, or a maximum turn count) closing the loop. Du et al.'s debate framework has each agent answer independently, then see the other agents' answers and revise its own, for several rounds — the paper reports that this consistently improves factual accuracy and arithmetic/reasoning correctness over asking one agent to self-reflect, but — as the misconception section below makes precise — only under specific conditions on how the agents are sampled. Park et al.'s Generative Agents populated a simulated town, "Smallville," with twenty-five agents, each holding a persistent memory stream, an importance-weighted retrieval mechanism over that memory, and a periodic reflection step that distills raw memories into higher-level observations the agent can plan from — the closest thing in this list to the blackboard style, since each agent's memory is private but the simulated environment itself is the shared, mutable world all of them observe and act in.

Worked example: an orchestrator-worker trip planner, traced exactly

To make the orchestrator-worker pattern concrete, trace a small planning system exactly, with every number checkable by hand. A user asks for a 3-day, 2-night Manali trip for two people on a budget of ₹15,000. The orchestrator decomposes this into four subtasks and dispatches one to each of four worker agents. In this trace, each worker wraps a deterministic pricing lookup rather than a live LLM call, so every returned value is exact — in a production system, transport_agent() would itself run a ReAct loop calling a real fare API, but fixing the numbers here lets us check the orchestration logic without any non-determinism getting in the way.

def transport_agent():
    # tool: static fare table (deterministic, not an LLM call)
    return {"item": "Volvo AC bus, Delhi-Manali return, 2 pax", "cost": 2400}

def stay_agent():
    return {"item": "2 nights, budget hotel, double room", "cost": 3600}

def food_agent():
    return {"item": "Meals, 2 pax x 3 days", "cost": 4200}

def activity_agent():
    return {"item": "Local sightseeing cab + entry fees", "cost": 1500}

def orchestrator(total_budget):
    workers = [transport_agent, stay_agent, food_agent, activity_agent]
    itinerary = []
    running_total = 0
    for worker in workers:
        result = worker()
        itinerary.append(result)
        running_total += result["cost"]
    remaining = total_budget - running_total
    status = "within budget" if remaining >= 0 else "over budget, replanning needed"
    return {
        "itinerary": itinerary,
        "total_cost": running_total,
        "remaining": remaining,
        "status": status,
    }

plan = orchestrator(15000)
print(plan["total_cost"], plan["remaining"], plan["status"])

Trace it by hand. The loop calls each worker once and accumulates running_total: 2400 (transport), then 2400 + 3600 = 6000 (stay), then 6000 + 4200 = 10200 (food), then 10200 + 1500 = 11700 (activity). After the loop, remaining = 15000 - 11700 = 3300, which is non-negative, so status is "within budget". The print call therefore outputs exactly:

11700 3300 within budget

The 3,300-rupee gap is a 22% buffer against the original budget (3300 / 15000 = 0.22). This is the orchestrator's real job: not generating the individual cost estimates — that is delegated to workers that specialize in one pricing domain each — but composing their results into a single checkable answer and deciding, from that composition, whether the plan is acceptable or needs another round.

Where does an actual LLM call enter this picture? At the planning step, before any worker runs, the orchestrator uses the model to decide which workers are needed and what each should return — a step that is inherently non-deterministic and is not part of the exact trace above:

# Orchestrator's planning step (schematic).
# call_llm is an assumed helper, not shown: it sends one prompt scoped to
# a single agent's role and returns the model's text response.
plan_prompt = (
    "Budget: Rs 15000, 2 people, 3 days, Manali. "
    "List the worker agents needed and the JSON key each should return."
)
plan_text = call_llm(plan_prompt, agent_role="orchestrator")
# plan_text is illustrative and will vary across runs, e.g.:
# "Use transport_agent, stay_agent, food_agent, activity_agent; each
#  returns {item, cost}."

This is the actual division of labour in most production orchestrator-worker systems: the decision of what to delegate is made by an LLM call and is genuinely non-deterministic, while the individual workers may be LLM calls, deterministic tools, or — as here — a mix, and the aggregation arithmetic is ordinary code, not a model call at all.

The coordination substrate: messages and shared memory

Whatever the topology, agents need a concrete format for handing information to each other. Most orchestrator-worker and peer-to-peer systems use a structured message schema close to the one modern function-calling APIs already expose: a sender, a recipient, a message type, and a content payload. The orchestrator's request to the transport agent above would be serialized as something like:

{
  "from": "orchestrator",
  "to": "transport_agent",
  "type": "subtask_request",
  "content": {
    "task": "round-trip fare, Delhi-Manali, 2 pax, AC coach",
    "budget_ceiling": 3000
  }
}

and the worker's reply mirrors it:

{
  "from": "transport_agent",
  "to": "orchestrator",
  "type": "subtask_result",
  "content": {"item": "Volvo AC bus, Delhi-Manali return, 2 pax", "cost": 2400}
}

Structured messages like these are easy to log, replay, and validate against a schema — if transport_agent returns a payload missing the cost key, the orchestrator can detect that mechanically rather than having to re-parse free-form prose. This is precisely why production multi-agent systems favour structured message passing over letting agents just write paragraphs to each other, even though the underlying model is perfectly capable of producing prose: structure is what makes the system's internal communication auditable. The blackboard alternative — Generative Agents' shared simulated environment, or a literal shared scratchpad object every agent can read and write — trades that message-level auditability for flexibility: any agent can pick up any fact any other agent left behind, without the orchestrator having to explicitly route it, at the cost of making it harder to say afterward exactly which agent's write caused a given downstream effect.

Failure modes unique to multi-agent systems

Splitting an agent into several does not just redistribute its old failure modes — it introduces genuinely new ones that a single-agent system cannot exhibit.

Hallucination cascades. If a worker agent fabricates a fact — invents a policy clause, misreads a tool's return value — and reports it back as a structured result, the orchestrator has no way to distinguish that fabrication from a correct result unless something in the system is explicitly checking. The orchestrator then treats the fabrication as ground truth and builds the final answer on top of it, the same way a rumour compounds as it passes down a chain of people repeating what they were told rather than what they observed.

Correlated errors under debate. The debate topology is supposed to catch errors through disagreement, but disagreement requires the debating agents to actually make different mistakes. If they are near-identical in their reasoning, debate degenerates into a single voice restating itself with extra steps, and any shared error survives the "debate" untouched.

Coordination overhead. Every additional agent hop is an additional model call, and in a sequential (non-parallel) topology, an additional unit of latency; in a peer-to-peer conversation, every new turn also resends the growing conversation history as context, so cost per turn tends to rise as the conversation lengthens, not stay flat. A five-agent orchestrator-worker system with one planning call and four worker calls issues five model calls where a single-agent design issued one — five times the token cost for the same query, before any accuracy benefit is counted (the aggregation step itself is ordinary code, not a further model call).

Non-termination. A peer-to-peer conversation with no explicit stopping rule can loop — two agents endlessly clarifying a point with each other — unless the system enforces a termination condition, such as a maximum turn count or a designated stop token (AutoGen uses exactly this pattern).

Common misconception: more agents does not mean more intelligence

The misconception worth naming explicitly: that wrapping several roles around an LLM and having them talk to each other makes the system strictly smarter than one call to the same model, because it now "looks like" several minds working on the problem instead of one. It does not, automatically. No multi-agent system creates new information or new capability that was not already present in the underlying model's weights — every agent in the diagram above is a call to the same policy, just with a different prompt and a different slice of context. Splitting a task across agents is a restructuring of inference-time compute, not an increase in the model's knowledge or reasoning ability, and restructuring compute only helps when it is engineered to exploit one of three specific, checkable mechanisms:

Context isolation — each agent's window holds only what its subtask actually needs, which keeps relevant facts closer to the edges of a shorter context and away from the "lost in the middle" degradation zone a single overloaded agent would suffer.

Sampling or prompt diversity — different personas, different temperatures, or different reasoning traces produce different, decorrelated errors, which is the only thing that makes a debate or a majority vote across agents statistically better than asking one agent once. If the "different" agents are near-deterministic copies of the same base model on the same prompt, there is no diversity to exploit, and debate or voting adds cost without adding accuracy.

Explicit verification structure — a critic or auditor agent that is asked a genuinely different question ("is this claim actually supported by the retrieved policy text?") rather than the same question again ("please double-check your answer") is structurally more likely to catch an error, because it is optimizing a different objective, not just re-running the same generation process and hoping for a different roll of the dice.

If none of these three is actually built into a system, adding agents adds latency and token cost — with no offsetting gain, and, through hallucination cascades, sometimes a net loss in reliability compared to a single well-scoped agent.

Active recall

Attempt each question before reading its answer.

Q1. A developer writes one LLM call that, within a single ongoing context, first reasons about a subtask, then calls a tool, then reasons about the next subtask, then calls another tool — all in one conversation with one system prompt. Is this a multi-agent system? Why or why not?

Q2. Given the "lost in the middle" effect, explain why splitting a single 50,000-token task across five specialized agents, each working with a roughly 10,000-token context, can improve retrieval accuracy on facts relevant to each subtask — even though the total number of tokens processed across the whole system is similar to what one large-context agent would process.

Q3. In a three-agent debate system built from three copies of the same base model, why might sampling all three at temperature 0 fail to reproduce the accuracy gains reported for debate at higher temperatures?

Q4. A support-agent orchestrator gives a customer a refund-policy answer sourced from a RefundPolicyAgent, but that agent hallucinated a clause that does not exist in the real policy. Which architectural addition most directly reduces this risk, and why is it meaningfully different from just asking the same agent to double-check its own answer?

Q5. A single-agent design costs ₹1.60 per query (one model call). An orchestrator-worker design for the same task uses one planning call and four worker calls, each at the same marginal cost as the single agent's call, plus an aggregation step that is ordinary code rather than a further model call. What is the total cost per query for the multi-agent design, and by what factor has cost increased?

Q6. In the Manali orchestrator trace, suppose the activity agent's pricing table is updated so its return becomes {"item": "Local sightseeing cab + entry fees", "cost": 3200} — ₹1,700 higher than before. Recompute total_cost, remaining, and status. Do the transport, stay, or food agents' return values change as a result? At what activity-agent cost would status flip to "over budget, replanning needed"?

Answers

A1. No. A multi-agent system requires separated contexts and roles that communicate through an explicit protocol — distinct agents, each with its own scoped context, exchanging messages. A single LLM interleaving reasoning and tool calls within one continuous context is the ReAct pattern applied by one agent; there is only one role, one context, and no inter-agent communication to speak of, even though the sequence of thought-action-observation steps looks superficially similar to what an orchestrator and a worker might do together.

A2. The total token count being similar is beside the point — what changes is how close the relevant fact sits to the edges of whatever context the model is reading at the moment it needs that fact. In the single 50,000-token agent, a fact needed for subtask 3 might be buried deep in the middle of an enormous mixed context, in the degraded-attention zone Liu et al. (2023) documented. In the five-agent split, the fact relevant to subtask 3 lives in a ~10,000-token context built specifically for that subtask, so it is proportionally much closer to that context's edges. Splitting the task redistributes the same information into several smaller "always near an edge" contexts instead of one large context where most facts, by definition, cannot all be near an edge simultaneously.

A3. Temperature 0 makes each agent's output nearly deterministic given the same prompt and model. Since all three agents share the same weights and the same prompt content, near-zero temperature means their independent answers — and their independent errors — are highly likely to be nearly identical. Debate only produces a correction signal when the agents' mistakes are decorrelated enough that at least one agent is likely to have gotten it right where another got it wrong; at temperature 0 with a shared model, that decorrelation mostly disappears, so the "debate" reduces to one agent agreeing with itself twice, with no error-correction possible.

A4. A dedicated verifier or critic agent with its own access to the actual grounding source — a retrieval tool over the real policy document — tasked specifically with adversarially checking the RefundPolicyAgent's claim against that source, most directly reduces the risk. This differs fundamentally from re-prompting the same agent to "double-check," because a same-agent re-check reruns essentially the same generation process that produced the hallucination in the first place and is prone to confidently repeating it; a verifier with a different objective (audit this claim against a source) and its own tool access can actually catch a mismatch the original agent's generation process was blind to.

A5. Total cost = ₹1.60 × 5 calls = ₹8.00 per query. The cost increase factor is 8.00 / 1.60 = 5, i.e. a 5x increase. For this design to be worth deploying, the accuracy or reliability gain from decomposition has to be large enough in business terms to justify roughly five times the inference spend per query — a real tradeoff a production team has to make explicitly, not something a multi-agent architecture gets for free.

A6. New running_total = 2400 + 3600 + 4200 + 3200 = 13400. New remaining = 15000 − 13400 = 1600, still non-negative, so status stays "within budget" — but the buffer has shrunk from ₹3,300 (22%) to ₹1,600 (about 10.7%), a drop of exactly ₹1,700, matching the activity agent's cost increase rupee-for-rupee, since total_budget is fixed and every other worker's output is unchanged. The transport, stay, and food agents' return values do not change: each is a pure function taking no arguments and reading no shared state, so none of them observes or reacts to the activity agent's new cost — this is exactly the isolation benefit of independent workers, but it cuts both ways, since none of them will automatically compensate (say, by suggesting a cheaper hotel tier) without the orchestrator explicitly triggering a new planning round. The threshold at which status flips is where remaining turns negative: 15000 − (2400 + 3600 + 4200) − activity_cost < 0, i.e. activity_cost > 4800. Any activity-agent cost above ₹4,800 would push the plan over budget and require replanning.

Think About It

Think about this: How would you explain multi-agent systems with large language models to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind multi-agent systems with large language models, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.