Harness Engineering: Why the Wrapper Around Your LLM Matters More Than the Model

The Problem

The industry’s default approach to improving AI-driven workflows has been to chase larger, more expensive frontier models. When an agent produces poor output, the instinct is to upgrade the model. However, research consistently demonstrates that the surrounding infrastructure—not the model itself—is the dominant factor in production response quality.

The foundational paper “Lost in the Middle: How Language Models Use Long Contexts” (Liu et al., Stanford & UC Berkeley, 2023) proved that LLMs exhibit a U-shaped attention curve: they recall information placed at the beginning or end of the context window with high accuracy, but performance drops by over 20% for information buried in the middle. This phenomenon persists even in models explicitly marketed as “long-context.” A 200K-token context window does not mean 200K tokens of useful context.

In agentic workflows—where an AI agent executes multi-step tasks over dozens of tool calls—context naturally accumulates. Logs, tool outputs, intermediate reasoning, and resolved errors pile up in the conversation history. Without active management, the context window fills with low-signal noise, triggering what practitioners call “context rot”: a gradual, silent degradation of response quality as the model’s effective attention becomes diluted. The agent begins hallucinating previously resolved errors, repeating completed steps, or losing sight of the original goal entirely—a failure mode known as “goal drift.”

As Andrej Karpathy defined in June 2025: “Context engineering is the delicate art and science of filling the context window with just the right information for the next step.” The discipline of building the infrastructure that achieves this is now called Harness Engineering.

The Architectural Solution

An Agent Harness is the software scaffolding that wraps around an LLM to transform it from a stateless text generator into a reliable, autonomous agent. If the LLM is the CPU, the harness is the operating system. It governs what the agent can see, what it can do, and how it manages memory across long-running tasks.

Controlled ablation studies have demonstrated that the same underlying LLM can exhibit up to a 6x performance variation on identical tasks simply by changing the surrounding harness. This gap frequently exceeds the performance difference between successive generations of frontier models, confirming that harness quality is the primary lever for production reliability.

A production-grade harness is built on three pillars:

1. Execution Loop & Tool Registry: The orchestration core that cycles through receiving input, prompting the model, dispatching tool calls via a centralized registry, and feeding results back. Tools are registered with strict JSON schemas and validated before execution, preventing hallucinated parameters from reaching backend services.
2. Context Window Manager: A budget-aware controller that continuously monitors token utilization. Research and industry benchmarks indicate that model fidelity begins to degrade significantly when context utilization exceeds approximately 40% of the window capacity (regardless of the model’s advertised maximum). The manager enforces this threshold by triggering compaction before the window becomes overloaded.
3. Compaction Engine: The mechanism that reduces context size when utilization approaches the threshold. It employs a staged strategy: first, offloading large tool outputs to external storage (leaving only reference pointers in the prompt); then summarizing older conversation turns while preserving live state (variable values, open tasks, hard constraints); and critically, pinning the original goal and system instructions so they are never compressed or dropped.

Core Implementation

The following TypeScript implementation demonstrates a minimal but functional agent harness with context-aware compaction. The key insight is the compactIfNeeded function: it monitors token utilization and proactively compresses history before quality degradation begins. This pattern allows even a smaller, less expensive model to produce high-quality responses by ensuring it always operates within its effective attention range.

// Agent Harness with Context-Aware Compaction
interface Message { role: 'system' | 'user' | 'assistant' | 'tool'; content: string; }
interface Tool { name: string; schema: object; execute: (args: any) => Promise<string>; }

const MAX_CONTEXT_TOKENS = 128_000;
const COMPACTION_THRESHOLD = 0.40; // Trigger compaction at 40% utilization
const MAX_STEPS = 25;

// The pinned goal survives every compaction cycle.
// This prevents "goal drift" — the #1 failure mode in long-running agents.
let pinnedGoal: string = '';

function estimateTokens(messages: Message[]): number {
  return messages.reduce((sum, m) => sum + Math.ceil(m.content.length / 4), 0);
}

async function compactIfNeeded(history: Message[]): Promise<Message[]> {
  const currentTokens = estimateTokens(history);
  const utilization = currentTokens / MAX_CONTEXT_TOKENS;

  // If utilization is below threshold, no action needed.
  if (utilization <= COMPACTION_THRESHOLD) return history;

  // Stage 1: Offload large tool outputs to external storage.
  // Replace verbose tool responses with compact reference pointers.
  const offloaded = history.map(msg => {
    if (msg.role === 'tool' && msg.content.length > 2000) {
      const ref = writeToExternalStore(msg.content);
      return { ...msg, content: `[Output stored: ${ref}. Key result: ${msg.content.slice(0, 200)}...]` };
    }
    return msg;
  });

  // Stage 2: Summarize older turns, but preserve the most recent 5.
  // The summarizer condenses resolved errors, completed sub-tasks,
  // and intermediate reasoning into a compact historical record.
  const recent = offloaded.slice(-5);
  const older = offloaded.slice(1, -5); // Skip index 0 (system prompt)
  const summary = await callLLM([{
    role: 'user',
    content: `Summarize these agent interactions. Preserve: variable values, 
              open tasks, unresolved errors, and key decisions. 
              Discard: resolved errors, verbose logs, and completed steps.\n\n
              ${older.map(m => `[${m.role}]: ${m.content}`).join('\n')}`
  }]);

  // Stage 3: Reconstruct context with the pinned goal always at the top.
  // The goal is injected into the system prompt so it occupies the
  // "primacy" position where LLM attention is strongest.
  return [
    { role: 'system', content: `${offloaded[0].content}\n\nACTIVE GOAL: ${pinnedGoal}` },
    { role: 'assistant', content: `[Compacted History]\n${summary}` },
    ...recent,
  ];
}

async function runAgent(task: string, tools: Map<string, Tool>): Promise<string> {
  pinnedGoal = task;
  let history: Message[] = [
    { role: 'system', content: SYSTEM_PROMPT },
    { role: 'user', content: task },
  ];

  for (let step = 0; step < MAX_STEPS; step++) {
    // Compact before every LLM call to maintain the quality threshold.
    history = await compactIfNeeded(history);

    const response = await callLLM(history);
    history.push({ role: 'assistant', content: response.content });

    if (response.toolCall) {
      const tool = tools.get(response.toolCall.name);
      if (!tool) throw new Error(`Unknown tool: ${response.toolCall.name}`);

      // Validate parameters against the registered JSON schema
      // before executing. This prevents hallucinated arguments
      // from reaching production microservices.
      if (!validateSchema(tool.schema, response.toolCall.args)) {
        history.push({ role: 'tool', content: 'REJECTED: Schema validation failed.' });
        continue;
      }

      const result = await tool.execute(response.toolCall.args);
      history.push({ role: 'tool', content: result });
    } else {
      return response.content; // Task complete
    }
  }
  return 'Max steps reached without completion.';
}

Edge Cases & Limitations

While harness engineering dramatically improves agent reliability, it introduces its own category of risks:

1. Compaction Information Loss: Summarization is inherently lossy. If the summarizer model itself is weak or the summary prompt is poorly designed, critical context (such as a user-specified constraint mentioned 15 turns ago) can be silently dropped. The agent then proceeds confidently with incomplete information, producing plausible but incorrect output.
2. Compaction Latency Overhead: Each compaction cycle requires an additional LLM call (to generate the summary), adding latency and cost to the agent loop. In latency-sensitive applications, this overhead may be unacceptable. Caching strategies and incremental summarization can mitigate this, but they add significant engineering complexity.
3. Threshold Calibration: The 40% utilization threshold is a practical heuristic, not a universal constant. Different models, task types, and context compositions may require different thresholds. Over-aggressive compaction (e.g., at 20%) risks discarding useful recent context prematurely, while under-aggressive thresholds (e.g., at 70%) may allow quality degradation before compaction kicks in.
4. Recursive Failure: If the LLM used for summarization during compaction itself hallucinates or produces a poor summary, the error compounds on every subsequent compaction cycle. The harness effectively “poisons its own memory,” leading to cascading failures that are extremely difficult to debug.

Key Strategic Takeaways

• Context is Finite Attention: Treat context window capacity as a precious execution budget rather than a dumping ground. Enforce proactive compaction at the 40% threshold.
• Pin the North Star: Goal drift is the primary failure mode of long-running autonomous workflows. Pin original objectives into system instructions across all compaction cycles.
• Harness Over Model Scale: Upgrading model parameter size yields diminishing returns compared to building deterministic schema validation, output offloading, and structured memory management.

---

Recommended Industry Reading & References

For further exploration of harness engineering, context management, and agent architecture, consider these authentic resources:

1. Liu et al. (2023): Lost in the Middle: How Language Models Use Long Contexts - The foundational Stanford/UC Berkeley paper proving the U-shaped attention curve and positional bias in LLMs.
2. Anthropic: Building Effective Agents - Production-grade guidance on agent harness design, context management, and compaction best practices.
3. OpenAI: Function Calling & Structured Outputs - The gold standard for deterministic tool-calling enforcement and JSON schema validation.
4. LangChain: Agent Memory & Context Management - Comprehensive documentation on sliding window, summarization, and tool output offloading patterns.
5. Andrej Karpathy: Context Engineering Definition (June 2025) - The widely-cited definition that crystallized the shift from prompt engineering to context engineering.