← Back to all products

Agent Tool Eval Harness

$39

Dependency-light evaluation harness for agent tools and MCP servers: golden cases, matchers, a runner with latency metrics, and console + JSON reporters for CI regression gating.

8 filesv1.0.0 (updated 2026-08-02)
MarkdownTypeScriptJSONAirflow

File Structure 8 files

agent-eval-harness/ ├── LICENSE ├── README.md ├── examples/ │ └── run.ts ├── package.json └── src/ ├── harness.ts ├── matchers.ts └── report.ts

Documentation Preview README excerpt

Agent Tool Eval Harness

A tiny, dependency-light evaluation harness for agent tools and MCP servers.

Define golden cases (input → expected), run them against your tool, and get a

pass/fail report with latency and optional cost — so you catch regressions before

your users (and their agents) do.

Why eval your tools

Prompt and model changes silently break tool behavior. Without evals you find out

in production. This harness gives you a fast, deterministic feedback loop: a set of

cases you can run in CI on every change.

Features

  • Golden cases — declare { name, input, expect } where expect is a value,

a predicate, or a partial-match object.

  • Runner — executes cases against any async (input) => output function

(wrap your MCP tools/call, HTTP endpoint, or plain function).

  • Metrics — pass rate, per-case latency (p50/p95), and an optional cost hook.
  • Matchers — exact, partial (subset match), and predicate (custom fn).
  • Reporters — a console table and a machine-readable JSON summary for CI.

Requirements

  • Node.js 18+ (uses only the standard library + your tool under test)

Quick start


npm install
npx tsx examples/run.ts     # runs the sample suite against a demo tool

import { runSuite } from './src/harness.js';
import { partial } from './src/matchers.js';

const report = await runSuite('search', mySearchTool, [
  { name: 'finds by keyword', input: { query: 'airflow' }, expect: partial({ total: (n: number) => n > 0 }) },
  { name: 'empty query is safe', input: { query: '' }, expect: (out) => Array.isArray(out.results) },
]);

process.exit(report.failed === 0 ? 0 : 1);

File tree


src/
  harness.ts    # runSuite(): executes cases, collects latency, builds the report
  matchers.ts   # exact / partial / predicate matchers
  report.ts     # console table + JSON summary
examples/
  run.ts        # a runnable sample suite against a demo tool

Using it in CI

Exit non-zero when any case fails; the JSON summary can gate a deploy:

... continues with setup instructions, usage examples, and more.

Code Sample .ts preview

src/harness.ts /** * harness.ts — run a suite of golden cases against an async tool function. */ import { evaluate } from './matchers.js'; export interface Case<I = any, O = any> { name: string; input: I; expect: unknown; // value | predicate | partial(shape) timeoutMs?: number; } export interface CaseResult { name: string; passed: boolean; latencyMs: number; error?: string; } export interface Report { suite: string; total: number; passed: number; failed: number; passRate: number; latency: { p50: number; p95: number; max: number }; cases: CaseResult[]; } type ToolFn<I, O> = (input: I) => Promise<O> | O; async function withTimeout<T>(p: Promise<T> | T, ms?: number): Promise<T> { if (!ms) return await p; return await Promise.race([ Promise.resolve(p), new Promise<T>((_, rej) => setTimeout(() => rej(new Error(`timeout after ${ms}ms`)), ms)), ]); } function percentile(sorted: number[], pct: number): number { if (sorted.length === 0) return 0; const idx = Math.min(sorted.length - 1, Math.floor((pct / 100) * sorted.length)); return sorted[idx]!; } export async function runSuite<I, O>( suite: string, tool: ToolFn<I, O>, cases: Case<I, O>[], ): Promise<Report> { # ... 29 more lines ...