Harness Study Notes

The concept of a harness (a test bench or control system) comes from traditional software testing, where tests run in a controlled environment with standardized inputs, outputs, and measurable evaluation criteria. An AI Agent Harness applies the same idea to an AI agent, providing:

• Controllability - Constrain the Agent’s behavioral boundaries
• Observability - Track the Agent’s thinking and actions
• Evaluability - Quantify the quality of the Agent’s output

Why Harness

Prompting an AI directly creates a black box: you cannot reliably predict what it will do, how it will do it, or how well it performed. A harness is like putting reins on an agent, keeping it on a track you designed and constraining its behavior.

graph LR
    A[用户请求] --> B[Prompt Only<br/>黑盒模式]
    A --> C[Harness Framework<br/>白盒模式]
    B --> D[不可预测输出]
    C --> E[可追踪轨迹]
    C --> F[可量化评估]
    C --> G[可干预控制]

Factor	Description	Key Question
Action Space	The set of actions the Agent can execute	Do the actions cover all necessary behaviors?
Observation	The information the Agent can perceive	Is the output sufficiently parseable?
Recovery	Error handling and recovery capability	Can it self-correct after failure?
Context Budget	Context resource management	Are tokens used effectively?

Action Space Design

Action Space is the set of all possible actions an Agent can take. Each action represents one way the Agent interacts with the world.

// Action Space 示例
interface Action {
  name: string;           // 稳定、显式的动作名称
  description: string;     // 清晰的意图描述
  inputSchema: z.ZodType;  // 输入参数校验
  outputSchema: z.ZodType; // 输出格式定义
}

Design Principles

1. Exhaustiveness. Cover all behaviors the Agent may need to perform, with no omissions.

2. Mutual Exclusiveness. Avoid semantic overlap between actions to reduce the Agent’s choice confusion.

3. Composability. Simple actions can be composed into complex behaviors.

// ❌ 反模式：语义重叠
const actions_bad = [
  { name: "delete_file", description: "删除文件" },
  { name: "remove_file", description: "移除文件" },  // 与上面重复！
  { name: "rm_file", description: "删除文件" },      // 又一个重复！
];

// ✅ 正确：互斥且清晰
const actions_good = [
  { name: "delete_file", description: "永久删除文件" },
  { name: "move_file", description: "移动文件到指定位置" },
  { name: "read_file", description: "读取文件内容" },
];

Granularity Rules.

Choose the appropriate action granularity based on task risk and frequency:

Granularity	Applicable Scenario	Example
Micro-Tools	High-risk operations (deployment, permissions, migration)	execute_deployment, grant_permission
Medium-Tools	Common edit/read/search loops	read_file, search_code, edit_block
Macro-Tools	When round-trip overhead is the primary cost	refactor_component, write_tests

Tool Definition

Standard Interface Design: Every Tool should have a clear interface definition

import { z } from "zod";

// Tool 定义示例
const ReadFileTool = {
  name: "read_file",
  description: "读取指定路径的文件内容",
  inputSchema: z.object({
    path: z.string().describe("文件路径"),
    offset: z.number().optional().describe("读取偏移量"),
    limit: z.number().optional().describe("读取字节数"),
  }),
  outputSchema: z.object({
    content: z.string(),
    bytesRead: z.number(),
    status: z.enum(["success", "partial", "error"]),
  }),
};

Observation Design: Every Tool response should include standardized fields to help the Agent understand the result and decide the next step

interface ToolResponse<T> {
  status: "success" | "warning" | "error";
  summary: string;           // 一句话结果总结
  data: T;                   // 实际数据
  artifacts?: string[];       // 生成的文件路径
  next_actions?: string[];    // 建议的后续动作
  recovery_hint?: string;     // 错误时的恢复提示
}

Error Recovery Contract: For every error path, recovery information must be provided, for example

// ❌ 反模式：只返回错误
{
  "error": "File not found"
}

// ✅ 正确：包含恢复信息
{
  "status": "error",
  "summary": "文件不存在",
  "error": {
    "code": "ENOENT",
    "path": "/tmp/nonexistent.txt",
    "root_cause": "路径不存在或无权访问",
    "recovery_hint": "请检查路径是否正确，或使用 search_files 搜索文件位置",
    "safe_retry": true,
    "stop_condition": "重试3次后仍失败则停止"
  }
}

Evaluation Metrics

Difficulties in Evaluating Agents

• Multi-path nature - The same goal has multiple ways to be achieved
• Subjectivity - Some output quality is hard to quantify
• Long dependencies - Early decisions affect later outcomes
• Cost - Every evaluation can incur high API fees

Key Metrics

Metric	Description	Calculation
Completion Rate	Task completion rate	Successful tasks / Total tasks
Retries Per Task	Average retry count	Total retries / Number of tasks
Pass@1	First-pass rate	First-attempt successes / Total tasks
Pass@3	Pass rate within three attempts	Successes within three attempts / Total tasks
Cost Per Success	Cost per success	Total cost / Successful tasks

Benchmark Design

interface Benchmark {
  name: string;
  tasks: Task[];
  metrics: Metric[];
  constraints: {
    maxTurns: number;        // 最大交互轮次
    maxTokens: number;        // 最大 token 消耗
    timeoutMs: number;        // 超时时间
  };
}

interface Task {
  id: string;
  prompt: string;
  expectedOutcome: string;
  evaluationCriteria: EvaluationCriterion[];
}

Building a Mini Agent Harness Demo

Architecture Overview

graph TD
    subgraph "Agent Core"
        P[Planner<br/>ReAct Loop]
        A[Action Selector]
    end

    subgraph "Harness Layer"
        AS[Action Space]
        TR[Tool Registry]
        ER[Evaluation Runner]
    end

    subgraph "Execution Env"
        FS[File System]
        WS[Web Search]
        SE[Shell]
    end

    P --> A
    A --> AS
    AS --> TR
    TR --> FS
    TR --> WS
    TR --> SE

    ER --> AS
    ER --> TR

Core Code

// 1. 定义 Action Space
const FileActionSpace = {
  actions: [
    {
      name: "read_file",
      description: "读取文件内容",
      inputSchema: z.object({
        path: z.string(),
        encoding: z.enum(["utf-8", "base64"]).default("utf-8"),
      }),
    },
    {
      name: "write_file",
      description: "写入文件内容",
      inputSchema: z.object({
        path: z.string(),
        content: z.string(),
        mode: z.enum(["overwrite", "append"]).default("overwrite"),
      }),
    },
    {
      name: "list_directory",
      description: "列出目录内容",
      inputSchema: z.object({
        path: z.string(),
        recursive: z.boolean().default(false),
      }),
    },
  ],

  // 验证动作空间完整性
  validate(): boolean {
    const names = this.actions.map((a) => a.name);
    return new Set(names).size === names.length; // 检查互斥性
  },
};

// 2. Tool Registry
class ToolRegistry {
  private tools: Map<string, Tool> = new Map();

  register(tool: Tool): void {
    if (this.tools.has(tool.name)) {
      throw new Error(`Tool ${tool.name} already registered`);
    }
    this.tools.set(tool.name, tool);
  }

  get(name: string): Tool | undefined {
    return this.tools.get(name);
  }

  list(): Tool[] {
    return Array.from(this.tools.values());
  }
}

// 3. Evaluation Runner
class EvaluationRunner {
  constructor(
    private registry: ToolRegistry,
    private actionSpace: ActionSpace
  ) {}

  async run(task: Task): Promise<EvaluationResult> {
    const startTime = Date.now();
    const trace: Action[] = [];
    let turns = 0;

    while (turns < task.constraints.maxTurns) {
      const context = await this.buildContext(trace);
      const action = await this.selectAction(task, context);

      if (!action) break; // 无可用动作

      const result = await this.executeAction(action);
      trace.push({ action, result, timestamp: Date.now() });

      if (this.isComplete(task, trace)) {
        return this.evaluate(trace, task, startTime);
      }

      turns++;
    }

    return this.evaluate(trace, task, startTime, { reason: "max_turns" });
  }
}

Run Example

// 创建 Harness
const registry = new ToolRegistry();
registry.register(ReadFileTool);
registry.register(WriteFileTool);
registry.register(ListDirectoryTool);

const harness = new EvaluationRunner(registry, FileActionSpace);

// 定义任务
const task: Task = {
  id: "read-config-01",
  prompt: "读取 /app/config.json 并告诉我数据库连接字符串",
  expectedOutcome: "返回 config.json 中的 db.connection 字段值",
  constraints: {
    maxTurns: 5,
    maxTokens: 4000,
    timeoutMs: 30000,
  },
  evaluationCriteria: [
    { type: "contains", value: "connection" },
    { type: "no_sensitive_data", pattern: /password|secret|key/i },
  ],
};

// 运行评估
const result = await harness.run(task);
console.log(result);
// {
//   passed: true,
//   metrics: {
//     completionRate: 1,
//     turns: 1,
//     tokensUsed: 512,
//     latencyMs: 234,
//   },
//   trace: [...]
// }

Architecture Choice

Pattern	Applicable Scenario	Pros	Cons
ReAct	Exploratory tasks with uncertain paths	Flexible, interpretable	More rounds, higher cost
Function Calling	Structured deterministic processes	Precise, controllable	Low flexibility
Hybrid (recommended)	ReAct planning + typed execution	Balanced	Medium implementation complexity

Common Anti-patterns

Anti-pattern	Problem	Solution
Overlapping action semantics	Agent confusion, choice difficulty	Mutual exclusivity check, refactoring
Opaque tool output	Cannot recover, cannot trace	Standardized response format
Returning only errors	Agent doesn’t know how to fix	Provide recovery_hint
Context overload	Irrelevant references drown out key information	Streamline system prompt, reference files instead of inlining

Summary

Core points of an AI Agent Harness:

1. Action Space: Must be exhaustive, mutually exclusive, and composable
2. Tool Definition: Must be schema-first, output must include next_actions
3. Error Recovery: Every error path must have a recovery_hint
4. Evaluation: Quantify four key metrics: completion rate, retry count, pass rate, cost

References

• ChatGPT

• Claude Code Agent SDK documentation

• ReAct: Synergizing Reasoning and Acting in Language Models