Hyperparameter Optimization: A Practical Guide

This guide is the opinionated "how to actually run a tuning job" companion to the

kit. It assumes you can already train a model and measure a validation metric;

the question is how to search the configuration space efficiently and trust the

result. Every code reference maps to a real symbol in src/hpo/.

The workflow in one picture

define metric  ->  define search space  ->  pick sampler + pruner/scheduler
      ->  set a budget  ->  run study  ->  inspect best + importance
      ->  widen/narrow space  ->  repeat until improvements plateau

Tuning is iterative. The first run is a reconnaissance pass with wide ranges; the

second run narrows toward the promising region. Two or three short rounds almost

always beat one giant run, because you spend later budget where it matters.

Step 1 — Nail down the objective

Before sampling a single configuration, decide:

• The metric. Use the metric you are actually judged on (e.g. balanced

accuracy, AUC, RMSE), not a proxy. The kit ships common scorers in

hpo.objective (accuracy, mse, rmse, mae, r2, neg_mse); pass any

callable (y_true, y_pred) -> float for anything custom.

• The direction. "maximize" for scores, "minimize" for losses. Keep it

consistent across the space, the pruner, and the study. The neg_* scorers

exist so a loss can be maximized uniformly when that is more convenient.

• The estimator of generalization. A single train/validation split is noisy

and easy to overfit through hundreds of trials. Prefer K-fold

cross-validation. hpo.objective.Objective wraps K-fold CV for you and returns

the mean fold score.

Pitfall — tuning on the test set. Never let the test set influence the

search. Tune against CV folds or a dedicated validation split, and evaluate the

single chosen configuration on the held-out test set exactly once.

Step 2 — Choose a sampler

The sampler decides *which configuration to try next*. The kit exposes the Optuna

samplers through StudyConfig(sampler=...):

Rule of thumb: start with random (or qmc) for one round to map the

terrain, then switch to tpe to exploit it. If a smart sampler cannot beat random

search by a meaningful margin, your space or metric is the problem, not the

sampler. The bundled examples/random_search_baseline.py gives you that baseline

with zero dependencies.

Step 3 — Prune the losers

Most configurations are mediocre and reveal themselves early. Pruning stops their

training once the intermediate learning curve makes the outcome obvious, which can

multiply throughput several-fold. Pruning needs two things:

1. an objective that reports intermediate values (per epoch, per boosting

round, or — as in hpo.objective.Objective — per CV fold); and

2. a pruner that compares the running trial against completed ones.

The kit implements the decision logic in pure Python so you can reason about it:

• MedianPruner — prune a trial whose best-so-far value is worse than the

median of completed trials at the same step. The sensible default.

• PercentilePruner(percentile=p) — keep the best p%, prune the rest.

MedianPruner is exactly PercentilePruner(50). Lower p is more aggressive.

• ThresholdPruner(lower=..., upper=...) — a guard rail: abort trials that

cross an absolute bound (e.g. a diverged loss), independent of other trials.

• EarlyStopping(patience=...) — classic in-loop stopping for a single trial;

no cross-trial baseline required.

Two parameters control pruning aggressiveness:

• n_startup_trials — do not prune until this many trials have *completed*, so the

baseline distribution is meaningful. Too low and you prune against noise.

• n_warmup_steps — do not prune any trial before this step, so slow-starting but

ultimately-strong configurations are not killed prematurely. Models with warm-up

schedules or heavy regularization need a larger value here.

Convert any of these to the real Optuna pruner with to_optuna_pruner(...), or

just hand the kit pruner to StudyConfig(pruner=...) and run_study does it.

Step 4 — Schedule trials (Ray Tune)

Ray Tune reframes pruning as *scheduling* across many parallel trials. The kit's

TuneConfig(scheduler=...) selects:

• ASHA (asha) — Async Successive Halving. Runs many trials for a short

grace_period, then promotes the top 1/reduction_factor to train longer,

repeating up to max_t. The workhorse for parallel HPO; the Ray analogue of

median/percentile pruning.

• PBT (pbt) — Population Based Training. Maintains a population, periodically

copying weights from strong trials and perturbing their hyperparameters. Powerful

for schedules that should *change during training* (e.g. learning rate). Requires

scheduler_kwargs["hyperparam_mutations"].

• Median stopping (median) — the simplest comparative rule, a good first try.

grace_period, reduction_factor, and max_t trade exploration for speed:

larger grace_period is safer but slower; larger reduction_factor is faster but

riskier. See examples/ray_tune_torch.py for a complete ASHA run.

Step 5 — Set a budget (and persist it)

• Trial count. A practical default is 20–30 trials per tunable dimension, then

inspect convergence. The optimization_history helper returns the running best,

which is the convergence curve — if it has been flat for many trials, stop.

• Parallelism. random/qmc/ASHA parallelize cleanly; tpe is partly

sequential (later trials depend on earlier results) but still benefits from

modest parallelism.

• Persistence and resume. Set `StudyConfig(storage="sqlite:///hpo.db",

study_name=..., load_if_exists=True)`. You can stop and resume, or point several

worker processes at the same database to fan trials out. examples/optuna_sklearn.py

does exactly this — re-run it and it continues the previous study.

Step 6 — Multi-objective optimization

When you care about more than one metric (accuracy *and* latency, recall *and*

precision), there is no single "best" — there is a Pareto front of

non-dominated trade-offs. To run it:

• pass a list of scorers and matching directions: `Objective(..., scorer=["accuracy",

my_latency_scorer]) with directions=["maximize", "minimize"]` on the study;

• the objective returns a tuple, and Optuna records study.best_trials (the front);
• pareto_front(records, directions) recomputes the front from backend-agnostic

records, and you pick the trade-off that fits your product constraints.

Do not collapse multiple objectives into a hand-weighted sum unless the weights are

genuinely known — the weighting hides the trade-off you are trying to see.

Step 7 — Read the results

After a study finishes, from_optuna_study(study) gives you TrialRecords that

feed every analysis helper:

• best_trial / top_k — the winners and the runners-up (inspect the top few:

if they cluster, you have found a stable region).

• param_importance — a fast variance-reduction proxy telling you which knobs

moved the metric. Use it to redesign the next round: widen important params,

freeze unimportant ones at a good value to cut dimensionality.

• optimization_history — the convergence curve.
• export_csv / export_json — dump everything for a notebook or a report.

Common failure modes

• Best value sits on a range boundary. The optimum is probably outside your

range — widen it and re-run. See search-space-design.md.

• Pruning too aggressively. If completed-trial counts are tiny and many trials

are pruned, raise n_startup_trials and n_warmup_steps.

• Noisy single-split scoring. Move to K-fold CV; the variance between splits is

often larger than the differences you are chasing.

• Over-tuning. Hundreds of trials against one validation split will find a

configuration that is lucky, not good. Cross-validate and keep an untouched test

set.

• Ignoring seeds. Set seed on the sampler and CV so results are reproducible

and comparable across rounds.

The short version: baseline with random search, exploit with TPE/ASHA, prune the

losers, cross-validate, inspect importance, narrow the space, repeat.