Agent Tool Eval Harness
Dependency-light evaluation harness for agent tools and MCP servers: golden cases, matchers, a runner with latency metrics, and console + JSON reporters for CI regression gating.
File Structure 8 files
Documentation Preview README excerpt
Agent Tool Eval Harness
A tiny, dependency-light evaluation harness for agent tools and MCP servers.
Define golden cases (input → expected), run them against your tool, and get a
pass/fail report with latency and optional cost — so you catch regressions before
your users (and their agents) do.
Why eval your tools
Prompt and model changes silently break tool behavior. Without evals you find out
in production. This harness gives you a fast, deterministic feedback loop: a set of
cases you can run in CI on every change.
Features
- Golden cases — declare
{ name, input, expect }whereexpectis a value,
a predicate, or a partial-match object.
- Runner — executes cases against any async
(input) => outputfunction
(wrap your MCP tools/call, HTTP endpoint, or plain function).
- Metrics — pass rate, per-case latency (p50/p95), and an optional cost hook.
- Matchers — exact,
partial(subset match), andpredicate(custom fn). - Reporters — a console table and a machine-readable JSON summary for CI.
Requirements
- Node.js 18+ (uses only the standard library + your tool under test)
Quick start
npm install
npx tsx examples/run.ts # runs the sample suite against a demo tool
import { runSuite } from './src/harness.js';
import { partial } from './src/matchers.js';
const report = await runSuite('search', mySearchTool, [
{ name: 'finds by keyword', input: { query: 'airflow' }, expect: partial({ total: (n: number) => n > 0 }) },
{ name: 'empty query is safe', input: { query: '' }, expect: (out) => Array.isArray(out.results) },
]);
process.exit(report.failed === 0 ? 0 : 1);
File tree
src/
harness.ts # runSuite(): executes cases, collects latency, builds the report
matchers.ts # exact / partial / predicate matchers
report.ts # console table + JSON summary
examples/
run.ts # a runnable sample suite against a demo tool
Using it in CI
Exit non-zero when any case fails; the JSON summary can gate a deploy:
... continues with setup instructions, usage examples, and more.