Evaluation Guide — Computer Vision Toolkit

Understanding mAP, IoU, precision-recall, and segmentation metrics — with practical guidance on when each metric matters and how to interpret the numbers.

Metrics at a Glance

IoU — Intersection over Union

IoU is the foundation of all detection and segmentation metrics. It measures how well a predicted region overlaps with the ground truth.

IoU = Area(Prediction ∩ Ground Truth) / Area(Prediction ∪ Ground Truth)

Bounding Box IoU

from cv_toolkit.evaluation import compute_iou

# Perfect overlap → IoU = 1.0
iou = compute_iou([0, 0, 100, 100], [0, 0, 100, 100])

# Partial overlap → 0 < IoU < 1
iou = compute_iou([0, 0, 100, 100], [50, 50, 150, 150])

# No overlap → IoU = 0.0
iou = compute_iou([0, 0, 50, 50], [100, 100, 200, 200])

IoU Thresholds and What They Mean

Practical impact: At IoU=0.5, a detection that covers 50% of the object counts as correct. At IoU=0.75, the box must be much tighter. COCO's mAP@[.5:.95] averages across thresholds, rewarding models that produce tight boxes.

Segmentation IoU (Per-Pixel)

For segmentation, IoU is computed per-class at the pixel level:

from cv_toolkit.evaluation import compute_mask_iou, compute_mean_iou

# Predicted and ground truth masks (each pixel = class ID)
pred = [[0, 0, 1, 1],
        [0, 0, 1, 1],
        [2, 2, 2, 2],
        [2, 2, 2, 2]]

gt   = [[0, 0, 1, 1],
        [0, 1, 1, 1],
        [2, 2, 2, 2],
        [2, 2, 0, 2]]

# Per-class IoU
class_ious = compute_mask_iou(pred, gt, num_classes=3)
# {0: 0.6, 1: 0.8, 2: 0.857}

# Mean IoU across all classes
miou = compute_mean_iou(pred, gt, num_classes=3)

Average Precision (AP)

AP summarizes the precision-recall tradeoff for a detector at a specific IoU threshold. It answers: *"Across all confidence thresholds, how well does the model balance finding objects (recall) and avoiding false alarms (precision)?"*

How AP Is Computed

1. Sort all detections by confidence (highest first)

2. Walk through the sorted list. For each detection:

• If it matches a ground truth box (IoU ≥ threshold) → True Positive (TP)
• If it matches a GT box already matched → False Positive (FP)
• If no matching GT box → False Positive (FP)

3. At each step, compute:

• Precision = TP / (TP + FP) — "of my detections, how many are real?"
• Recall = TP / Total GT — "of all real objects, how many did I find?"

4. Plot the precision-recall curve

5. Compute the area under this curve = AP

Interpolation Methods

All-point interpolation (COCO style): compute the area under the smoothed PR curve using every unique recall value. More precise, the current standard.

11-point interpolation (Pascal VOC 2007): sample precision at 11 recall levels (0.0, 0.1, ..., 1.0) and average. Simpler but less accurate.

from cv_toolkit.evaluation import compute_ap_per_class, DetectionAnnotation

# Define ground truth and predictions
gt = [
    DetectionAnnotation("img1", class_id=0, box=[10, 10, 50, 50]),
    DetectionAnnotation("img1", class_id=0, box=[60, 60, 100, 100]),
]
preds = [
    DetectionAnnotation("img1", class_id=0, box=[12, 8, 48, 52], confidence=0.95),
    DetectionAnnotation("img1", class_id=0, box=[62, 58, 98, 102], confidence=0.80),
    DetectionAnnotation("img1", class_id=0, box=[200, 200, 250, 250], confidence=0.30),
]

ap, recalls, precisions = compute_ap_per_class(
    preds, gt, class_id=0, iou_threshold=0.5, method="all_points"
)
print(f"[email protected] for class 0: {ap:.4f}")

mAP — Mean Average Precision

mAP averages AP across all object classes. It is the single number that summarizes a detector's overall performance.

from cv_toolkit.evaluation import compute_map, compute_coco_map

# mAP at a single IoU threshold
result = compute_map(predictions, ground_truths, iou_threshold=0.5)
print(f"[email protected]: {result['mAP']}")
print(f"Per-class AP: {result['per_class_ap']}")

# COCO-style mAP across 10 thresholds (0.50, 0.55, ..., 0.95)
coco = compute_coco_map(predictions, ground_truths)
print(f"mAP@[.5:.95]: {coco['mAP@[.5:.95]']}")
print(f"[email protected]:       {coco['[email protected]']}")
print(f"[email protected]:      {coco['[email protected]']}")

Interpreting mAP Numbers

Important: mAP varies dramatically across datasets. An [email protected] of 50% on COCO (80 classes, small objects, occlusion) is much harder to achieve than 90% on a custom dataset with 5 large, well-separated classes.

Classification Metrics

Accuracy

Accuracy = Correct Predictions / Total Predictions

When accuracy lies: With imbalanced classes (99% negative, 1% positive), a model that always predicts "negative" gets 99% accuracy while being completely useless. Use F1 or precision/recall instead.

Precision, Recall, F1

Precision = TP / (TP + FP)   — "When I say positive, am I right?"
Recall    = TP / (TP + FN)   — "Did I find all the positives?"
F1        = 2 × P × R / (P + R) — harmonic mean of precision and recall

from cv_toolkit.evaluation import compute_classification_metrics

metrics = compute_classification_metrics(
    y_true=[0, 0, 1, 1, 2, 2, 2],
    y_pred=[0, 1, 1, 1, 2, 0, 2],
    num_classes=3,
)

print(f"Overall accuracy: {metrics['accuracy']:.4f}")
for cls_id, cls_metrics in metrics['per_class'].items():
    print(f"  Class {cls_id}: P={cls_metrics['precision']:.3f} "
          f"R={cls_metrics['recall']:.3f} F1={cls_metrics['f1']:.3f}")

Confusion Matrix

The confusion matrix shows exactly where the model confuses classes:

             Predicted
             Cat  Dog  Bird
Actual Cat  [ 45   3    2 ]
       Dog  [  5  38    7 ]
       Bird [  1   4   45 ]

Read row-by-row: "Of 50 actual cats, the model predicted 45 as cat, 3 as dog, and 2 as bird."

High off-diagonal values reveal which class pairs are confused — this tells you where to focus data collection or model improvement.

Decision Guide: Which Metric to Report

Classification

• Balanced dataset: accuracy is fine as the primary metric
• Imbalanced dataset: report macro-F1 (averaged across classes) and per-class recall
• Binary classification: report precision, recall, and F1 for the positive class, plus the ROC-AUC if you have probability outputs
• Multi-label: report per-label AP and mean AP

Detection

• Academic paper / benchmark: report mAP@[.5:.95] (COCO standard)
• Production deployment: report [email protected] (practical), plus per-class AP for your critical classes
• Safety-critical (autonomous driving, medical): report recall at high precision — missing a real object is worse than a false alarm

Segmentation

• Semantic segmentation: report mIoU and per-class IoU
• Instance segmentation: report mask mAP (like detection mAP but using mask IoU instead of box IoU)
• Panoptic segmentation: report Panoptic Quality (PQ)

Common Evaluation Mistakes

Mistake 1: Evaluating on Training Data

Never report metrics on data the model was trained on. Always hold out a validation set (10-20% of data) and ideally a test set that you only evaluate once.

Mistake 2: Ignoring Class Imbalance

If 95% of your images are "background" and 5% are "defect", a model that always predicts "background" gets 95% accuracy. Report per-class metrics and look at the minority class recall.

Mistake 3: Tuning on the Test Set

If you use test set metrics to choose hyperparameters, you're effectively training on the test set. Use a validation set for hyperparameter tuning and reserve the test set for final evaluation.

Mistake 4: Not Reporting Confidence Intervals

A single accuracy number is meaningless without context. Report the standard deviation across multiple runs (different seeds) or compute confidence intervals via bootstrapping.

Mistake 5: Wrong IoU Threshold for Your Application

Using [email protected] when your application needs precise localization (surgery, manufacturing inspection) gives a misleadingly optimistic picture. Use a stricter threshold that matches your deployment requirements.