← Back to all products

GPU Training Toolkit

$39

Multi-GPU training configs, mixed precision training, distributed training, and cloud GPU setup guides.

16 filesv1.0.0
PythonMarkdownJSON

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 16 files

gpu-training-toolkit/ ├── LICENSE ├── README.md ├── examples/ │ ├── train_amp.py │ └── train_ddp.py ├── guides/ │ ├── distributed-training-guide.md │ └── mixed-precision.md ├── src/ │ └── gpu_training/ │ ├── __init__.py │ ├── amp.py │ ├── checkpoint.py │ ├── config.py │ ├── ddp.py │ ├── launcher.py │ └── profiling.py └── tests/ ├── test_config.py └── test_profiling.py

Documentation Preview README excerpt

GPU Training Toolkit

Battle-tested PyTorch patterns for multi-GPU, mixed-precision, and

distributed training — packaged as a small, importable library you drop into

your project. It encodes the parts people get subtly wrong (effective batch math,

fp16 loss scaling, rank-0 checkpointing, OOM-safe batch sizing) so your scale-out

is correct the first time.

Every PyTorch import is lazy, so the configuration/profiling math and the full

test suite run on the Python standard library alone — no GPU and no torch

required to explore, plan, or test.

Features

  • Distributed Data Parallel (DDP) — process-group init/teardown, per-rank

device binding, DistributedSampler sharding, cross-rank metric reduction, and

a main_process_first helper for race-free one-time work.

  • Mixed precision done right — autocast + GradScaler with the correct

fp16-vs-bf16 behavior (a scaler is enabled only for fp16) and

unscale-before-clip gradient clipping.

  • torchrun launch generation — turn a typed config into the exact

single-node, multi-node, or SLURM srun command.

  • Profiling & OOM-safe batch finder — GPU memory snapshots, a throughput

meter (samples/sec, tokens/sec), and a logarithmic batch-size search that finds

the largest batch that fits before you crash.

  • Distributed-safe checkpoints — rank-0-only, atomic writes with automatic

rotation and DDP-wrapper unwrapping.

  • Typed TrainingConfig — one object that computes effective batch size,

optimizer-step counts, and a linear-warmup + cosine learning-rate schedule.

Requirements

  • Python 3.10+
  • PyTorch 2.x and CUDA 11.8+ to run training (one or more NVIDIA GPUs)
  • Nothing but the standard library to run the config/profiling math and tests

The package is pure-stdlib at import time; torch is imported lazily inside the

functions that actually touch the GPU.

Quick Start

1. Explore the math with no GPU


python3 -m gpu_training.config      # effective batch + step-count demo
python3 -m gpu_training.profiling   # OOM-safe batch finder demo
python3 -m gpu_training.launcher    # generate torchrun commands

import sys; sys.path.insert(0, "src")
from gpu_training.config import TrainingConfig, gradient_accumulation_for_target

# Hit a target effective batch of 512 with 8/GPU across 4 GPUs:
accum = gradient_accumulation_for_target(512, per_device_batch_size=8, world_size=4)
cfg = TrainingConfig(per_device_batch_size=8, gradient_accumulation_steps=accum, world_size=4)
print(cfg.effective_batch_size)          # 512
print(cfg.describe(num_samples=1_000_000))

... continues with setup instructions, usage examples, and more.

Code Sample .py preview

src/gpu_training/amp.py """Mixed-precision training: autocast, GradScaler, and the accumulation loop. Mixed precision is two independent ideas that people conflate: 1. **autocast** runs the forward pass in a low-precision dtype (``float16`` or ``bfloat16``) for speed and memory, while keeping a master copy of weights in ``float32``. 2. **loss scaling** (``GradScaler``) is only needed for ``float16``, whose narrow exponent range underflows small gradients to zero. ``bfloat16`` has the same exponent range as ``float32`` and therefore needs **no scaler at all**. Getting (2) wrong is the most common mixed-precision bug: people wrap a bf16 run in a scaler (harmless but pointless) or, worse, run fp16 *without* a scaler and silently train on zeroed gradients. :class:`MixedPrecisionTrainer` encodes the correct behavior — a scaler is enabled iff ``precision == "fp16"``. The pure-Python helpers (:func:`should_step`, :func:`precision_to_dtype_name`) are unit-tested without torch; the trainer imports torch lazily on first use. """ from __future__ import annotations from contextlib import nullcontext from typing import Any, Iterable, Optional # Maps our precision strings to the torch dtype names they select. _DTYPE_NAMES: dict[str, str] = { "fp32": "float32", "fp16": "float16", "bf16": "bfloat16", } # Precisions that engage autocast (i.e. anything other than full fp32). _AUTOCAST_PRECISIONS: frozenset[str] = frozenset({"fp16", "bf16"}) def precision_to_dtype_name(precision: str) -> str: """Return the torch dtype *name* for a precision string (pure Python).""" try: return _DTYPE_NAMES[precision] except KeyError: raise ValueError( f"Unknown precision {precision!r}; expected one of {sorted(_DTYPE_NAMES)}" ) from None def is_autocast_precision(precision: str) -> bool: """True when ``precision`` should run under autocast (fp16/bf16).""" if precision not in _DTYPE_NAMES: raise ValueError(f"Unknown precision {precision!r}") # ... 248 more lines ...