Harness Engineering: Evaluation-Driven Testbeds for Frontier AI Agents

Frontier models now ship every few weeks. The durable asset is the harness around them: fixtures, sandboxes, graders and a regression gate that tells you — on your own tasks — whether Opus 5.5, Fable 5.2, Gemini 4 Neon, GPT-6.2 Sol or Grok 4.7 is the right choice.

Gigantic Work EngineeringSystems architecture team • 20 Oct 2026 • Updated 22 Nov 2026

In the ten days before we refreshed this note, three labs shipped new frontier models: Anthropic released Claude Fable 5.2 (9 November 2026), OpenAI released GPT-6.2 Sol and Astra (17 November), and Google released Gemini 4 Neon (18 November). Two months earlier came Claude Opus 5.5; in October, SpaceXAI shipped Grok 4.7 to its API, a few months after launching Grok Bot, its always-on agents with their own cloud computers.

Every launch arrived with a benchmark chart showing it ahead. Google’s chart puts Gemini 4 Neon above Opus 5.5 and GPT-6.2 Sol on one composite index; Anthropic says Fable 5.2 at high effort matches Opus 5.5 at max effort on its coding benchmark for about half the cost per task. Both claims may be true — and neither tells you which model will answer your students’ questions, triage your grant applications or fix your codebase best.

That is the job of a harness.

The first law of harness engineering

A capability you cannot measure with an automated, calibrated harness does not exist. Every model swap, prompt edit, tool change or effort setting passes through the same evaluation gate before it reaches users.


1. Two meanings of “harness” — and why they merged

The word is used for two things, and in 2026 they have become inseparable:

  • The agent harness is the runtime around the model: the loop, the tools it can call, how context is assembled and compacted, how errors and retries are handled. OpenAI’s Codex team popularised harness engineering in early 2026 to describe this work — instructions, architectural constraints, linters and clean-up loops that keep an agent productive.
  • The evaluation harness is the testbed that runs tasks through that agent and grades the results.

They merged because you can no longer evaluate a model in isolation. Anthropic’s guidance on evaluating AI agents makes the point directly: what you are testing is the model and its harness together. Public leaderboards keep showing the same model scoring very differently under different harnesses — which also means a good harness is a competitive advantage you own, while the model is something you rent.


2. Anatomy of an evaluation harness (2026 edition)

A production harness has six decoupled layers. Each can change without rewriting the others.

Layer What it holds Typical failure it catches
Fixtures Versioned tasks: a golden set from real usage, an adversarial set, and fresh samples from production Regressions on the questions users actually ask
Environment A sandbox with resettable state: files, a seeded database, mocked or recorded APIs Agents that only work on the developer’s laptop
Runner Executes each fixture k times per configuration (model × effort × prompt version) Flaky behaviour hidden by a single lucky run
Graders Code checks first, then model-based judges, then sampled human review Wrong answers, unsafe tool calls, missing citations
Transcript store Every trajectory: messages, tool calls, tokens, latency, cost “It passed, but it deleted the staging table on the way”
Gate Statistical comparison against the current baseline with budgets for cost and latency Shipping a change that is 3% better and 40% more expensive

Try it: a simulated evaluation gate

Choose a proposed change and run it through the gate. The numbers are illustrative, but the mechanics — cheap checks first, judges only where needed, a statistical decision at the end — are the ones we use on client work.

Evaluation gate Baseline: Claude Opus 5.5 · medium · 120 fixtures × 5 runs
SIMULATION · illustrative numbers

3. Grade outcomes, then check the path

For agents, grade the end state first: is the file fixed, is the record updated, does the answer cite the right handbook section? Agents often find valid routes you did not anticipate, so grading the exact sequence of steps punishes creativity.

But inspect the trajectory for things that must never happen, whatever the outcome: a write to a production system, a tool call outside the allowed list, personal data sent to an external service.

Cheap deterministic checks go first, because they are fast, free and never disagree with themselves:

import ast
from pydantic import BaseModel

FORBIDDEN_IMPORTS = {"os", "sys", "subprocess", "socket"}

class CheckResult(BaseModel):
    fixture_id: str
    passed: bool
    violation: str | None = None

def check_generated_patch(fixture_id: str, code: str) -> CheckResult:
    """Layer 1: syntax and policy checks before any model-based grading."""
    try:
        tree = ast.parse(code)
    except SyntaxError:
        return CheckResult(fixture_id=fixture_id, passed=False, violation="SYNTAX_INVALID")

    for node in ast.walk(tree):
        names = []
        if isinstance(node, ast.Import):
            names = [a.name.split(".")[0] for a in node.names]
        elif isinstance(node, ast.ImportFrom) and node.module:
            names = [node.module.split(".")[0]]
        if FORBIDDEN_IMPORTS.intersection(names):
            return CheckResult(fixture_id=fixture_id, passed=False, violation="SANDBOX_POLICY")

    return CheckResult(fixture_id=fixture_id, passed=True)

def check_trajectory(tool_calls: list[dict], allowed: set[str]) -> str | None:
    """Layer 1b: the path must stay inside the allowed tool list, whatever the outcome."""
    for call in tool_calls:
        if call["name"] not in allowed:
            return f"TOOL_NOT_ALLOWED:{call['name']}"
    return None

Only fixtures that pass these checks are sent to a model-based judge, so you pay for semantic grading only where it can change the verdict.


4. Calibrating the LLM-as-a-judge

Judges are indispensable for open-ended outputs — tone, completeness, whether an explanation would make sense to a first-year student — but they carry well-known biases:

  • Position bias: preferring whichever answer appears first in a pairwise comparison.
  • Verbosity bias: rewarding longer answers regardless of substance.
  • Self-preference: rating outputs from their own model family higher.
  • Score compression: clustering everything at 7–8 out of 10.

Mitigations that work in practice:

  1. Swap and agree. Run every pairwise comparison twice with the order reversed; accept a verdict only if both runs agree.
  2. Judge from a different family. If the candidate is Claude, judge with Gemini or GPT, and vice versa. With five frontier families available, there is no excuse for a model marking its own homework.
  3. Rubrics over scores. Ask for specific pass/fail criteria (“cites the section”, “states the deadline”, “does not invent a policy”) instead of a 1–10 number.
  4. Calibrate against people. Have staff label 50–100 examples and measure judge–human agreement before trusting the judge on its own. Re-check whenever you change the judge model.
async def pairwise_verdict(fixture, answer_a, answer_b, judge) -> str:
    first = await judge.compare(fixture, first=answer_a, second=answer_b)
    swapped = await judge.compare(fixture, first=answer_b, second=answer_a)

    if first.winner == "first" and swapped.winner == "second":
        return "A"
    if first.winner == "second" and swapped.winner == "first":
        return "B"
    return "INCONCLUSIVE"  # position bias detected: escalate to rubric grading or a person

5. Consistency: pass@k is not enough for agents

A coding assistant that solves a task in one of five attempts is useful — a developer picks the good one. An agent that updates student records correctly in four of five runs is a liability. Measure both:

  • pass@k — at least one of k runs succeeds. Right for “generate options, a human chooses”.
  • pass^k — all k runs succeed. Right for anything that acts on its own.

Run each fixture several times per configuration and report a confidence interval, not a single number. A three-point gain on 40 fixtures is usually noise.


6. Treat model, effort and price as one search space

Frontier models now expose effort or reasoning-budget settings, and labs ship fast and slow variants of the same generation. The question is no longer “which model?” but “which model at which effort, for which step?”. A sensible harness sweeps the grid and plots quality against cost and latency:

# .github/workflows/eval-gate.yml
name: Evaluation gate

on:
  pull_request:
    paths: ['prompts/**', 'tools/**', 'agent/**', 'eval/fixtures/**']

jobs:
  eval:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        model: [claude-opus-5-5, claude-fable-5-2, gemini-4-neon, gpt-6.2-sol, grok-4.7]
        effort: [medium, high]
    steps:
      - uses: actions/checkout@v4
      - name: Run harness
        run: |
          python -m harness.run \
            --fixtures eval/fixtures/golden_v7.jsonl \
            --model ${{ matrix.model }} --effort ${{ matrix.effort }} \
            --repeats 5 --out results/${{ matrix.model }}-${{ matrix.effort }}.json
      - uses: actions/upload-artifact@v4
        with:
          name: results-${{ matrix.model }}-${{ matrix.effort }}
          path: results/

  gate:
    needs: eval
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with: { path: results, merge-multiple: true }
      - name: Compare against baseline
        run: |
          python -m harness.gate \
            --baseline eval/baseline.json --results results/ \
            --min-pass-hat-k 0.95 --max-cost-delta 0.10 --max-p95-latency-delta 0.20

The gate fails the pull request if consistency drops, cost rises more than 10%, or p95 latency rises more than 20% against the current production configuration. The model IDs above are the API names at the time of writing; keep them in config, not code.

Routing usually beats picking one winner. In our projects, a common pattern is a stronger model for planning and judging, and a faster, cheaper tier for high-volume steps such as retrieval rewriting or classification — chosen per step from harness results, and re-checked when a new release lands.


7. Keep the fixtures honest

  • Build them from real usage. For a course assistant: real student questions (anonymised), the handbook version they were asked against, and the correct section.
  • Include questions it must refuse. “What grade will I get?” or “What is my classmate’s student number?” are test cases too.
  • Assume public benchmarks leak. Popular benchmark tasks end up in training data. Keep a private set the vendors have never seen.
  • Refresh every term. Sample new production questions each semester and retire fixtures that every configuration passes.

8. Summary

Harness engineering turns model choice from a matter of opinion into a measured decision. With:

  1. versioned fixtures drawn from your own users,
  2. sandboxed, resettable environments,
  3. deterministic checks before calibrated judges,
  4. consistency metrics and confidence intervals,
  5. a CI gate with quality, cost and latency budgets,

you can adopt the next frontier release in days rather than months — or decide, with evidence, that you don’t need to.


9. Working with Gigantic Work

Every AI assistant we deliver for a university, NGO or SME ships with its evaluation harness: a fixture set built with your staff, the graders, and the gate wired into deployment. You keep all of it, so you can test the next model yourself.

Model names, release dates and vendor claims are as published by Anthropic, Google, OpenAI and SpaceXAI in September–November 2026. Vendor benchmark results are self-reported.

WORK WITH US

Building an AI agent or assistant?

We design agent architectures, evaluation harnesses and secure tool access for universities, NGOs and SMEs — model-agnostic across Claude, Gemini, GPT, Grok and open-weight models, hosted where your data policy requires.