Batching and Scaling

Dynamic micro-batching is the single highest-leverage optimization for online

model serving. This guide explains how the batcher works, how to tune it, and

how to scale the service horizontally once a single replica is saturated.

Why batching wins

Frameworks predict on a *matrix* far more efficiently than on one row at a time.

Two overheads get amortized across the batch:

• Python / dispatch overhead. Each predict() call crosses into numpy / C /

BLAS once. A batch of 32 rows pays that crossing once; 32 single-row calls pay

it 32 times.

• Accelerator overhead. On a GPU/TPU, every call is a kernel launch plus a

host↔device copy. Tiny batches leave the device idle waiting on Python. Larger

batches keep it busy and can be 10–50× more throughput-efficient.

The catch: a REST endpoint receives requests one at a time. The batcher's job is

to *reassemble* a matrix from many small requests without making any single

caller wait too long.

The flush policy

A batch is dispatched as soon as either trigger fires:

flush when:  len(batch) >= max_batch_size        # the "size" trigger
        or   elapsed     >= max_batch_latency_ms  # the "latency" trigger

• The size trigger bounds how large a batch can get (memory, and the point

of diminishing returns on your hardware).

• The latency trigger bounds how long the *first* instance in a batch waits.

It is the key to keeping tail latency predictable under light load: if only

one request arrives, it waits at most max_batch_latency_ms and then goes.

In batching.py the loop grabs the first queued item, then keeps pulling more

with a shrinking asyncio.wait_for timeout until the deadline passes or the

size cap is hit. The Flask ThreadedBatcher does the same with a

queue.Queue(timeout=...).

Choosing max_batch_size and max_latency_ms

These two knobs are a throughput-vs-latency dial. Tune them empirically:

1. Measure single-row latency t1 and the per-row marginal cost in a

batch. Benchmark predict at sizes 1, 2, 4, 8, 16, 32, 64 and plot

rows/second. Throughput climbs, then plateaus — set max_batch_size near the

start of the plateau (often 16–64 for CPU sklearn, 64–512 for GPU deep nets).

2. Pick a latency budget. If your SLA is "p99 < 50 ms" and the model itself

takes ~20 ms on a full batch, you can afford max_batch_latency_ms of

roughly 10–15 ms of queueing. Smaller windows = lower latency, fewer co-batched

requests; larger windows = bigger batches, higher throughput, higher tail.

3. Validate under realistic load. Use examples/client_request.py's

concurrency burst, then read inference_batch_size from /metrics. If the

histogram still piles up at 1 under load, your latency window is too short (or

the load is genuinely too low to batch — which is fine).

Rules of thumb:

A little queueing theory

Little's Law (L = λ · W) is enough to sanity-check capacity. With arrival rate

λ (req/s) and mean time-in-system W (s), the average number of in-flight

requests is L. If your effective service rate (batches/s × batch_size) drops

below λ, the queue grows without bound and latency runs away. Two consequences:

• Headroom matters. Target sustained utilization well under 100% (say

≤ 70%). A service running at 95% utilization has no slack to absorb bursts and

will see latency explode at the first traffic spike.

• Watch queue depth, not just CPU. GET /healthz reports the batcher's

queue_depth. A steadily rising queue depth is the earliest, clearest signal

that you are under-provisioned — earlier than CPU saturation.

Scaling out

Once one replica is saturated, scale horizontally:

• More replicas. Each replica loads its own model copy and runs its own

batcher. Put a load balancer in front; round-robin is fine because the

batching happens *inside* each replica. This scales throughput linearly until

a shared bottleneck (database, feature store) appears.

• Workers vs replicas. On CPU, uvicorn --workers N gives N processes per

container, each with a full model copy — watch memory (N × model size). On GPU,

prefer one process per GPU and scale with more replicas/pods; multiple

workers contending for one device usually *hurts* throughput.

• Autoscale on the right signal. CPU utilization is a lagging, noisy signal

for batched inference. Prefer a custom metric: scale on queue_depth,

requests-per-second, or p95 of http_request_duration_seconds. In Kubernetes

this is an HPA on a Prometheus-adapter custom metric scraped from /metrics.

Memory, warmup, and cold starts

• Warm up the model. The first prediction often triggers lazy allocation

(BLAS threadpools, CUDA context, JIT). Send a synthetic batch during startup

so the first *real* request does not eat that cost. (The lifespan is the place

to add a warmup call.)

• Right-size memory limits. Account for: model size × workers, plus the peak

batch's intermediate tensors, plus framework overhead. Set container limits

with headroom; an OOM-kill mid-batch fails every co-batched caller at once.

• Pin thread counts on CPU. Oversubscribed BLAS threads (OMP_NUM_THREADS,

OPENBLAS_NUM_THREADS) across many workers cause thrashing. A common recipe is

one BLAS thread per process and concurrency from replicas instead.

Observability checklist

Everything you need to run this in production is already exported on /metrics:

• http_requests_total — request volume and error rate (by status).
• http_request_duration_seconds — latency histogram; compute p50/p95/p99 with

histogram_quantile() in PromQL.

• inference_batch_size — confirms batching is actually happening; if it stays

at 1 under load, revisit the latency window.

• inference_errors_total — model/inference failures, distinct from HTTP 4xx.
• queue_depth (via /healthz) — the leading indicator for autoscaling.

Alert on rising error rate, p99 latency breaching your SLA, and sustained queue

depth growth — in that order of urgency.