August 8, 2026 · 9 min read

From Single-Prompt to Multi-Agent: Orchestrating Local LLMs with a Centralized Controller

How to build an orchestrator-worker multi-agent system on local LLMs with Ollama: structured delegation, hard loop limits, and when not to use multiple agents.

By Husain Abbas, Enterprise Architect & former CTO

The Problem

The simplest LLM integration is a single prompt and a single response. It works well for bounded questions. It breaks down when a task needs several distinct kinds of reasoning, a check on intermediate results, or more context than one prompt can hold reliably.

Take a typical enterprise request: “Research the competitive landscape of edge computing platforms, write a technical brief, and make sure it is accurate.” A single prompt asks one model call to be the researcher, the writer and the fact-checker at once. The instructions compete with each other. Nothing checks the claims. Nothing can run in parallel. When the output is wrong, you can’t tell which “role” failed.

The usual fix is to move to a bigger model. That helps, but it is not the only lever. Anthropic reported that its multi-agent research system, with a Claude Opus 4 lead agent and Claude Sonnet 4 subagents, outperformed single-agent Claude Opus 4 by 90.2% on its internal research evaluation. The same report is just as clear about the cost: multi-agent runs used about 15× more tokens than chat interactions. Structure can beat raw model size on the right kind of task, and you pay for it in tokens and latency.

The Architectural Solution

The orchestrator-worker pattern splits the system into a coordination layer and a set of narrow, stateless workers. Anthropic’s Building Effective Agents describes it as a central LLM that “dynamically breaks down tasks” and delegates them. It is best suited to tasks where you can’t predict the subtasks in advance.

The design has three components:

Orchestrator (coordinator). It owns the user’s intent and decides which worker to call next and with what inputs. It does not produce the deliverable itself. Its only “tool” is delegation.
Workers (specialists). Each worker has a tightly scoped system prompt. It receives a task plus only the context it needs, returns one result and keeps no memory between calls.
Blackboard (shared state). The harness, not the model, stores worker outputs. The orchestrator sees short previews and refers to outputs by name. Full artifacts go to workers only when they are needed. This keeps the orchestrator’s context small, which matters a great deal on local models with limited context windows.

Orchestrator-worker architecture: the user request goes to a central orchestrator, which reads and writes a shared blackboard and delegates research, drafting and review subtasks to stateless worker agents, then returns the approved draft

The control loop looks like this:

Orchestrator receives the user request
  -> delegate(researcher, task)                    -> blackboard["researcher"]
  -> delegate(writer, task, context=[researcher])  -> blackboard["writer"]
  -> delegate(reviewer, context=[researcher, writer])
       verdict == "pass"   -> return blackboard["writer"]
       verdict == "revise" -> writer again with the reviewer's feedback (bounded)

Why Not One Larger Model?

Prompt isolation. A researcher told to “list facts and mark uncertainty” and a writer told to “use only the facts supplied” each have one job. Putting both instructions in one prompt dilutes both.
Context hygiene. Each worker starts with a clean context window and sees only what it needs. Anthropic’s context engineering guidance recommends sub-agents for exactly this reason: the lead agent receives condensed results, not every intermediate token. For the single-agent version of the problem, see Harness Engineering.
A separate evaluation step. Asking a model to check its own answer in the same context has limited value. Huang et al. (2023) found that LLMs struggle to self-correct reasoning without external feedback. Panickssery et al. (2024) showed that LLM evaluators recognise and favour their own generations. A separate reviewer with its own prompt and context is an improvement, but it is not fully independent if it runs on the same weights. For real independence, use a different model family, or ground the review in something external such as source documents, tests or retrieval.
Different models for different roles. Narrow worker tasks can often run on small models, such as Llama 3.2 at 1B or 3B. Planning and judgement are where a larger model earns its cost.
Parallelism. Independent subtasks, such as researching two competitors, can run concurrently. Locally, this only helps if the server allows it. Ollama processes one request per model at a time by default (OLLAMA_NUM_PARALLEL=1), and parallel slots multiply context memory (Ollama FAQ).

Core Implementation

The reference implementation below uses the Ollama Python client against a local Ollama server. It relies on Ollama’s structured outputs, which constrain generation to a JSON schema produced by a Pydantic model. That is much stronger than the older format="json" mode, which guarantees valid JSON but not the right fields.

Prerequisites:

pip install ollama pydantic
ollama pull llama3.2:3b      # worker model (3B, ~2 GB)
ollama pull llama3.3:70b     # orchestrator (70B, ~43 GB download); use llama3.1:8b on smaller hardware

Agent Definitions and Stateless Workers

from dataclasses import dataclass
from typing import Literal

import httpx
import ollama
from pydantic import BaseModel, ValidationError

WORKER_MODEL = "llama3.2:3b"
ORCHESTRATOR_MODEL = "llama3.3:70b"
client = ollama.Client(timeout=300)  # seconds; forwarded to the underlying httpx client


class Review(BaseModel):
    verdict: Literal["pass", "revise"]
    feedback: str


@dataclass(frozen=True)
class AgentConfig:
    name: str
    system_prompt: str
    model: str = WORKER_MODEL
    output_schema: type[BaseModel] | None = None


AGENTS = {a.name: a for a in [
    AgentConfig("researcher", "You are a research specialist. List the facts relevant to the "
                "task as bullet points and mark anything uncertain as UNVERIFIED. Do not write prose."),
    AgentConfig("writer", "You are a technical writer. Turn the supplied research into a "
                "structured brief. Use only facts present in the research; never add new ones."),
    AgentConfig("reviewer", "You are a strict reviewer. Compare the draft with the research. Flag "
                "unsupported claims, gaps and unclear passages with specific, actionable feedback.",
                output_schema=Review),
]}


@dataclass
class WorkerResult:
    agent: str
    ok: bool
    output: str


def spawn_worker(cfg: AgentConfig, task: str, context: str = "") -> WorkerResult:
    """One-shot call: the worker sees only its prompt, its task and the context it is given."""
    user_msg = f"TASK:\n{task}\n\nCONTEXT:\n{context}" if context else f"TASK:\n{task}"
    try:
        resp = client.chat(
            model=cfg.model,
            messages=[{"role": "system", "content": cfg.system_prompt},
                      {"role": "user", "content": user_msg}],
            format=cfg.output_schema.model_json_schema() if cfg.output_schema else None,
            options={"num_ctx": 16384, "temperature": 0.2},
        )
        output = resp.message.content
        if cfg.output_schema:
            cfg.output_schema.model_validate_json(output)  # fail fast on schema drift
        return WorkerResult(cfg.name, True, output)
    except (ollama.ResponseError, httpx.HTTPError, ValidationError) as exc:
        return WorkerResult(cfg.name, False, f"ERROR {type(exc).__name__}: {exc}")

Two details matter on local hardware. First, num_ctx is set explicitly. Ollama’s default context length depends on available VRAM and is only 4K tokens below 24 GiB. Without the override, long research context gets cut off silently. Second, worker failures come back as data, not exceptions, so the orchestrator can retry or reroute.

The Orchestration Loop

class Decision(BaseModel):
    action: Literal["delegate", "finish"]
    agent: str = ""
    task: str = ""
    context_from: list[str] = []  # names of blackboard entries the worker needs


ORCHESTRATOR_PROMPT = """You coordinate specialist agents and never do the work yourself.
Agents: researcher, writer, reviewer. Reply with JSON matching the schema.
To delegate, set agent, task and context_from (names of earlier outputs the agent needs).
Typical flow: researcher -> writer [researcher] -> reviewer [researcher, writer].
If the reviewer asks for revisions, call writer with [researcher, writer, reviewer]."""

MAX_TURNS, MAX_WRITER_CALLS, PREVIEW_CHARS = 10, 3, 600


def orchestrate(request: str) -> str:
    board: dict[str, str] = {}  # blackboard owned by the harness, not the model
    writer_calls = 0
    messages = [{"role": "system", "content": ORCHESTRATOR_PROMPT},
                {"role": "user", "content": request}]

    for _ in range(MAX_TURNS):
        resp = client.chat(model=ORCHESTRATOR_MODEL, messages=messages,
                           format=Decision.model_json_schema(), options={"temperature": 0})
        raw = resp.message.content
        messages.append({"role": "assistant", "content": raw})
        try:
            d = Decision.model_validate_json(raw)
        except ValidationError as exc:
            messages.append({"role": "user", "content": f"Invalid decision: {exc}"})
            continue

        if d.action == "finish":
            return board.get("writer", "Finished without producing a draft.")
        if d.agent not in AGENTS:
            messages.append({"role": "user", "content": f"Unknown agent {d.agent!r}; use {list(AGENTS)}"})
            continue
        if d.agent == "writer":
            writer_calls += 1
            if writer_calls > MAX_WRITER_CALLS:  # hard cap enforced in code, not in the prompt
                return board.get("writer", "Revision budget exhausted.")

        ctx = "\n\n".join(f"## {k}\n{board[k]}" for k in d.context_from if k in board)
        result = spawn_worker(AGENTS[d.agent], d.task, ctx)
        if result.ok:
            board[d.agent] = result.output
            # Deterministic exit: don't ask the model to decide something code can decide.
            if d.agent == "reviewer" and Review.model_validate_json(result.output).verdict == "pass":
                return board["writer"]

        status = "OK" if result.ok else "FAILED"
        messages.append({"role": "user",
                         "content": f"[{result.agent} {status}] preview:\n{result.output[:PREVIEW_CHARS]}"})

    return board.get("writer", "Stopped: turn budget exhausted without a draft.")

The design choices worth copying are these:

Schema-constrained decisions. Ollama constrains the orchestrator to the Decision schema, and Pydantic validates the result again. A validation error goes back to the model as feedback rather than crashing the loop.
Budgets live in code. MAX_TURNS and MAX_WRITER_CALLS are enforced by the harness. A small model told to “revise at most twice” in its prompt will sometimes ignore the rule.
Previews, not payloads. The orchestrator sees 600-character previews. Full artifacts move between workers through the blackboard. This keeps the orchestrator’s context roughly constant as the workflow grows.

Production Hardening

Observability. Log one structured event per turn: the decision, the target agent, the context keys passed, output size, latency and model. Without traces, delegation failures are close to impossible to diagnose. Cemri et al. (2025) analysed more than 1,600 traces across seven multi-agent frameworks and grouped the failures into system design issues, inter-agent misalignment and task verification. Most of them are only visible in traces.
Model residency. Mixing a 70B orchestrator with 3B workers on one GPU can trigger model swaps. Check OLLAMA_MAX_LOADED_MODELS and keep_alive, and measure load time separately from inference time.
Grounded review. Wherever you can, give the reviewer something to check against: the research artifact, retrieved sources or executable tests. “Is this good?” is a weak instruction for a 3B model.
Security. Worker outputs are untrusted input to the next prompt. If workers browse the web or read documents, treat their outputs as a prompt-injection channel. For deterministic guardrails around agent tool execution, see Engineering Enterprise Guardrails for Agentic AI.

Edge Cases & Limitations

This exact workflow does not need an LLM orchestrator. Researcher → writer → reviewer is a fixed sequence. A plain Python pipeline (prompt chaining plus an evaluator-optimizer loop) is cheaper, faster and fully deterministic. An LLM orchestrator earns its cost only when the subtasks depend on the input and you can’t enumerate them in advance. Anthropic’s advice is to find “the simplest solution possible, and only increasing complexity when needed”.

Prefer a single agent when:

The task fits in one well-structured prompt with tools.
Latency or cost per request is the main constraint, since every delegation is a full inference.
The subtasks are tightly coupled and need shared understanding. Cognition’s Don’t Build Multi-Agents argues that parallel agents working from partial context make conflicting implicit decisions. This applies especially to coding.

Consider multi-agent orchestration when:

The work is open-ended and parallelisable, as in breadth-first research.
A step needs a separate evaluator with different instructions, a different model or external evidence.
The total working context exceeds what one context window handles reliably.
You need an auditable record of who did what.

Known failure modes: orchestrators that loop between two workers, workers that drop information when summarising for the next step, reviewers that rubber-stamp, and orchestrators that copy worker output into their own answer (defeating the blackboard). Token use multiplies as well. Anthropic found that token usage alone explained 80% of the performance variance on its BrowseComp evaluation. That is good news for quality and bad news for budgets.

Framework Landscape

If you need more than a hand-rolled loop, all of the following can work with local models through Ollama:

LangGraph / LangChain. Graph-based, stateful orchestration with documented subagent, handoff, router and custom-workflow patterns. A good fit when you need durable state and explicit control flow.
Microsoft Agent Framework. Microsoft’s successor to AutoGen and Semantic Kernel. AutoGen itself is now in maintenance mode. AG2 is a separate, community-led continuation of the AutoGen lineage.
LlamaIndex Workflows. Event-driven workflows, strongest when retrieval over your own data is central.
CrewAI. A high-level, role-based abstraction. Fast to prototype, but you have less control over routing.

My advice: build the raw loop once so you understand the mechanics and failure modes, then adopt a framework for its persistence, tracing and human-in-the-loop support.

Key Strategic Takeaways

Start with a pipeline, earn the orchestrator. Use an LLM orchestrator only when the subtasks can’t be known in advance. Fixed sequences belong in code.
The harness owns state and budgets. Keep artifacts on a blackboard, give the orchestrator previews, and enforce turn and revision limits in code rather than in prompts.
Separate is not the same as independent. A reviewer on the same model reduces self-confirmation but does not eliminate it. Ground reviews in evidence, or use a different model.
Budget for tokens and latency. Multi-agent systems trade cost for quality. Measure that trade on your own tasks before committing to the architecture.