LLM Evaluation Framework
Automated evaluation harnesses, custom metrics, human feedback collection, regression testing, and quality monitoring dashboards.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 33 files
Documentation Preview README excerpt
LLM Evaluation Framework
A comprehensive toolkit for evaluating large language model outputs with automated
metrics, regression testing, LLM-as-judge scoring, and human feedback collection.
Whether you're comparing prompt templates, testing a model version upgrade, or
building a continuous evaluation pipeline, this framework gives you the metrics
library, harness infrastructure, and reporting tools to measure quality
systematically instead of eyeballing responses.
Features
- 13+ built-in metrics covering lexical (BLEU, ROUGE, F1), semantic (embedding
similarity), structural (JSON validity, format compliance), and LLM-as-judge
evaluation
- Evaluation harness that loads JSONL/JSON/CSV datasets, runs model inference
with retry logic, computes metrics, and saves structured results
- Regression tester that compares two evaluation runs, computes per-metric and
per-category deltas, and fires alerts when scores drop below configurable
thresholds
- LLM-as-judge with pre-built rubrics (helpfulness, safety, code quality,
summarization) and pairwise comparison support
- Human feedback collection with structured annotation schemas, JSONL storage,
and inter-annotator agreement metrics (Cohen's kappa)
- Report generator producing self-contained HTML dashboards and Markdown
reports with metric visualizations, category breakdowns, and worst-sample
analysis
- Dataset management with validation, stratified sampling, category splitting,
and data contamination detection
Quick Start
1. Run the metrics library standalone
from src.metrics import compute_all_metrics
prediction = "The capital of France is Paris."
reference = "Paris is the capital of France."
results = compute_all_metrics(prediction, reference)
for metric, score in results.items():
print(f" {metric}: {score:.4f}")
2. Run a full evaluation
from src.eval_harness import EvalHarness
def my_model(instruction: str, input_text: str = "") -> str:
# Replace with your actual model call
return "model response"
harness = EvalHarness(
model_fn=my_model,
metrics=["exact_match", "token_f1", "bleu", "rouge_l"],
model_name="my-model-v1",
)
*... continues with setup instructions, usage examples, and more.*