GPU Training Toolkit
Multi-GPU training configs, mixed precision training, distributed training, and cloud GPU setup guides.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 16 files
Documentation Preview README excerpt
GPU Training Toolkit
Battle-tested PyTorch patterns for multi-GPU, mixed-precision, and
distributed training — packaged as a small, importable library you drop into
your project. It encodes the parts people get subtly wrong (effective batch math,
fp16 loss scaling, rank-0 checkpointing, OOM-safe batch sizing) so your scale-out
is correct the first time.
Every PyTorch import is lazy, so the configuration/profiling math and the full
test suite run on the Python standard library alone — no GPU and no torch
required to explore, plan, or test.
Features
- Distributed Data Parallel (DDP) — process-group init/teardown, per-rank
device binding, DistributedSampler sharding, cross-rank metric reduction, and
a main_process_first helper for race-free one-time work.
- Mixed precision done right — autocast +
GradScalerwith the correct
fp16-vs-bf16 behavior (a scaler is enabled only for fp16) and
unscale-before-clip gradient clipping.
torchrunlaunch generation — turn a typed config into the exact
single-node, multi-node, or SLURM srun command.
- Profiling & OOM-safe batch finder — GPU memory snapshots, a throughput
meter (samples/sec, tokens/sec), and a logarithmic batch-size search that finds
the largest batch that fits before you crash.
- Distributed-safe checkpoints — rank-0-only, atomic writes with automatic
rotation and DDP-wrapper unwrapping.
- Typed
TrainingConfig— one object that computes effective batch size,
optimizer-step counts, and a linear-warmup + cosine learning-rate schedule.
Requirements
- Python 3.10+
- PyTorch 2.x and CUDA 11.8+ to run training (one or more NVIDIA GPUs)
- Nothing but the standard library to run the config/profiling math and tests
The package is pure-stdlib at import time; torch is imported lazily inside the
functions that actually touch the GPU.
Quick Start
1. Explore the math with no GPU
python3 -m gpu_training.config # effective batch + step-count demo
python3 -m gpu_training.profiling # OOM-safe batch finder demo
python3 -m gpu_training.launcher # generate torchrun commands
import sys; sys.path.insert(0, "src")
from gpu_training.config import TrainingConfig, gradient_accumulation_for_target
# Hit a target effective batch of 512 with 8/GPU across 4 GPUs:
accum = gradient_accumulation_for_target(512, per_device_batch_size=8, world_size=4)
cfg = TrainingConfig(per_device_batch_size=8, gradient_accumulation_steps=accum, world_size=4)
print(cfg.effective_batch_size) # 512
print(cfg.describe(num_samples=1_000_000))
... continues with setup instructions, usage examples, and more.