August 16, 2026 · 8 min read

Harness Engineering: Why the Wrapper Around Your LLM Matters More Than the Model

Agent harness engineering in practice: context budgets, staged compaction, goal pinning and schema-validated tool calls, with evidence, code and failure modes.

By Husain Abbas, Enterprise Architect & former CTO

The Problem

When an agent produces poor output, the instinctive fix is a bigger model. That is often the most expensive lever and not always the most effective one. The software around the model decides what the model sees, which tools it can call and how it recovers from errors. That software has a large, measurable effect on results.

The evidence comes from three directions.

Position matters. Lost in the Middle (Liu et al., Stanford, UC Berkeley and Samaya AI; published in TACL) showed a U-shaped performance curve. Models use information at the start or end of the context much better than information in the middle. In the paper’s multi-document QA experiments, GPT-3.5-Turbo’s accuracy dropped by more than 20% when the answer sat mid-context. In the worst case it fell below its closed-book score, meaning that adding the documents made the answer worse. The effect also appeared in models built for long contexts.

Length matters, even on easy tasks. Chroma’s Context Rot report (July 2025) tested 18 models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3. Performance degraded consistently as input length grew, even on simple retrieval and text-replication tasks. Distractors made it worse. In practice, a 200K-token window does not give you 200K tokens of equally useful attention.

The harness changes the score. LangChain moved its coding agent from 52.8% to 66.5% on Terminal-Bench 2.0 with the model held fixed (gpt-5.2-codex). It changed only the harness: verification-oriented prompting, a pre-completion checklist, loop detection and time-budget warnings.

Agentic workloads make all three problems worse. Over dozens of tool calls, logs, stack traces and stale intermediate reasoning pile up. Quality drops gradually and without any error. The agent repeats finished steps, revisits errors it already fixed or loses track of the original objective. Practitioners call the last of these goal drift.

In June 2025, Andrej Karpathy described context engineering as “the delicate art and science of filling the context window with just the right information for the next step.” Harness engineering is the broader discipline of building the system that does this reliably. OpenAI’s Harness engineering post (February 2026) helped popularise the term.

The Architectural Solution

An agent harness is the software that turns a stateless text generator into a dependable agent. If the model is the CPU, the harness is the operating system. It schedules work, manages memory, mediates I/O and enforces permissions.

A production harness rests on three pillars.

Execution loop and tool registry. The loop prompts the model, dispatches tool calls through a registry and feeds results back. Arguments are validated against each tool’s JSON Schema before anything runs. Every call, whether it succeeds or fails, produces a result the model can see and act on. Prefer provider-side constrained decoding where it exists, such as OpenAI’s Structured Outputs with strict schemas. Validate on your side anyway.
Context budget manager. The manager measures token usage before every model call and compares it with a budget you set, which is lower than the advertised window. No research supports a single universal threshold. Degradation is gradual and depends on the task. Treat the trigger point (40% of the window in the code below) as a tunable starting value, and calibrate it with your own evals.
Compaction engine. This applies the cheapest effective reduction first:
- Offload. Move bulky tool output to external storage and leave a pointer plus a short excerpt.
- Summarise. Condense older turns into a state record that keeps decisions, constraints, open tasks and identifiers.
- Pin. Re-insert the system prompt and the original goal verbatim after every compaction, so they are never summarised away.

Anthropic’s Effective Context Engineering for AI Agents describes the same toolkit: compaction, tool-result clearing, structured note-taking outside the context window and sub-agents with clean contexts.

Agent harness architecture: the user task enters an execution loop that calls the LLM and routes tool calls through schema validation before execution, while a context budget monitor triggers a compaction engine that offloads large outputs, summarises older turns and re-pins the goal

Core Implementation

The TypeScript below is a minimal harness with staged compaction. The model client, artifact store and token counter are declared as interfaces so the harness logic stays visible. In production, use your provider’s token counting rather than a characters-per-token estimate.

Staged Compaction

type Role = 'system' | 'user' | 'assistant' | 'tool';
interface ToolCall { id: string; name: string; args: unknown }
interface Message { role: Role; content: string; toolCalls?: ToolCall[]; toolCallId?: string }

// Supplied by your model SDK, artifact store and tokenizer.
declare function callLLM(messages: Message[]): Promise<{ content: string; toolCalls: ToolCall[] }>;
declare function storeArtifact(content: string): Promise<string>; // returns a URI
declare function countTokens(messages: Message[]): number;

const CONTEXT_WINDOW = 128_000;
const COMPACT_AT = 0.4;     // starting heuristic: calibrate with evals per model and task
const KEEP_RECENT = 6;
const OFFLOAD_CHARS = 2_000;

const pin = (systemPrompt: string, goal: string) =>
  `${systemPrompt}\n\nPINNED GOAL (verbatim, never summarise):\n${goal}`;

async function offloadToolOutputs(history: Message[]): Promise<Message[]> {
  return Promise.all(history.map(async (m) => {
    if (m.role !== 'tool' || m.content.length <= OFFLOAD_CHARS || m.content.startsWith('[offloaded')) return m;
    const uri = await storeArtifact(m.content);
    return { ...m, content: `[offloaded to ${uri}; excerpt]\n${m.content.slice(0, 300)}` };
  }));
}

// Never split a tool result from the assistant turn that requested it.
function recentStart(history: Message[]): number {
  let i = Math.max(1, history.length - KEEP_RECENT);
  while (i > 1 && history[i].role === 'tool') i--;
  return i;
}

async function compactIfNeeded(history: Message[], systemPrompt: string, goal: string): Promise<Message[]> {
  const budget = CONTEXT_WINDOW * COMPACT_AT;
  if (countTokens(history) <= budget) return history;

  // Stage 1: cheap and recoverable. Move bulky tool output out of the prompt.
  const h = await offloadToolOutputs(history);
  if (countTokens(h) <= budget) return h;

  // Stage 2: lossy. Summarise everything between the system prompt and the recent tail.
  const cut = recentStart(h);
  const older = h.slice(1, cut);
  if (older.length === 0) return h; // nothing compressible; caller must escalate or fail

  const summary = await callLLM([
    { role: 'system', content: 'You compress agent history into a terse state record.' },
    { role: 'user', content:
      'KEEP: user constraints, decisions and reasons, open tasks, unresolved errors, ' +
      'file paths, IDs, artifact URIs. DROP: resolved errors, verbose logs, finished steps.\n\n' +
      older.map((m) => `[${m.role}] ${m.content}`).join('\n') },
  ]);

  // Stage 3: rebuild from the ORIGINAL system prompt and goal, so repeated
  // compactions cannot erode or duplicate them.
  return [
    { role: 'system', content: pin(systemPrompt, goal) },
    { role: 'user', content: `[Summary of earlier work]\n${summary.content}` },
    ...h.slice(cut),
  ];
}

The Execution Loop

interface Tool {
  name: string;
  schema: object;                          // JSON Schema advertised to the model
  validate: (args: unknown) => boolean;    // e.g. a compiled Ajv validator for `schema`
  execute: (args: unknown) => Promise<string>;
}

const MAX_STEPS = 25;

async function runAgent(goal: string, systemPrompt: string, tools: Map<string, Tool>): Promise<string> {
  let history: Message[] = [
    { role: 'system', content: pin(systemPrompt, goal) },
    { role: 'user', content: goal },
  ];

  for (let step = 0; step < MAX_STEPS; step++) {
    history = await compactIfNeeded(history, systemPrompt, goal);
    const res = await callLLM(history);
    history.push({ role: 'assistant', content: res.content, toolCalls: res.toolCalls });
    if (res.toolCalls.length === 0) return res.content; // no tool requested: task complete

    for (const call of res.toolCalls) {
      const tool = tools.get(call.name);
      let result: string;
      if (!tool) {
        result = `ERROR: unknown tool "${call.name}". Available: ${[...tools.keys()].join(', ')}`;
      } else if (!tool.validate(call.args)) {
        result = 'ERROR: arguments failed schema validation. Correct them and retry.';
      } else {
        try { result = await tool.execute(call.args); }
        catch (e) { result = `ERROR: ${e instanceof Error ? e.message : String(e)}`; }
      }
      // Every tool call needs exactly one result linked by id; most chat APIs
      // reject the next request otherwise.
      history.push({ role: 'tool', toolCallId: call.id, content: result });
    }
  }
  throw new Error(`Step budget (${MAX_STEPS}) exhausted before completion`);
}

Three design decisions do most of the work here:

Errors are observations, not exceptions. Unknown tools, schema failures and tool exceptions all go back to the model as tool results. The model can usually correct itself, and the transcript stays valid for the API.
The cut point respects tool-call pairing. Splitting an assistant tool call from its result is a common cause of hard API errors after compaction.
Pinning uses the original strings. Rebuilding from systemPrompt and goal, rather than from the current history[0], stops repeated compactions from appending the goal over and over or slowly paraphrasing it.

Edge Cases & Limitations

Summaries lose information. A constraint the user stated 15 turns ago can vanish in a summary. The agent then carries on confidently. Mitigations: write durable facts (constraints, decisions, IDs) to a structured state file or task list outside the transcript, and re-inject them. Don’t rely on the summariser to keep them.
Compounding summary errors. Each compaction summarises a summary. An early hallucination becomes “history” that every later step trusts. Keep the raw transcript and offloaded artifacts in storage so a human or a recovery routine can audit and rebuild state.
Latency, cost and cache invalidation. Summarisation is an extra model call. Rewriting earlier messages also changes the prompt prefix, which invalidates provider-side prompt caching for everything after the change. Compact in larger, less frequent steps rather than trimming a little on every turn.
Threshold calibration. The 40% trigger is a heuristic. Compacting too early throws away useful recent detail and pays for extra summarisation calls. Compacting too late lets quality degrade before the trigger fires. Calibrate per model and task with an eval set that includes long-horizon cases.
Compaction is not enough on its own for very long tasks. Anthropic’s Effective Harnesses for Long-Running Agents states plainly that compaction isn’t sufficient for multi-session work. It recommends an initializer step, a progress file, git history and an explicit feature list, so each new session can rebuild state from artifacts rather than from a summary.
When not to build your own. If a mature harness already fits your use case, such as a coding agent CLI, a framework’s built-in summarisation (for example LangChain’s short-term memory middleware) or a provider’s native context-editing features, start there. Build a custom harness when you need control over data residency, tool permissions or domain-specific state.

For splitting long tasks across agents with separate context windows, see Orchestrating Local LLMs with a Centralized Controller.

Key Strategic Takeaways

Treat context as a budget, not a buffer. Measure it on every turn and compact before quality degrades. Set the threshold from your own evals, not from the advertised window.
Pin the goal verbatim and keep durable state outside the transcript. Summaries are for narrative. Constraints, decisions and identifiers belong in structured state that is re-injected every time.
Make every failure visible to the model. Validated tool arguments, errors returned as observations and hard step budgets turn crashes into recoverable steps.
Invest in the harness before the model. With the model held fixed, harness changes alone have produced double-digit benchmark gains. Change the harness first, then decide whether you still need a bigger model.