← Back to all products
$39
GPU Optimization Guide
CUDA memory management, mixed precision training, distributed training configs, and GPU utilization monitoring.
PythonYAMLMarkdownJSON
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 38 files
gpu-optimization-guide/
├── LICENSE
├── README.md
├── benchmarks/
│ ├── __pycache__/
│ │ └── memory_benchmark.cpython-312.pyc
│ └── memory_benchmark.py
├── configs/
│ ├── ddp_config.yaml
│ ├── deepspeed_config.json
│ └── fsdp_config.yaml
├── free-sample.zip
├── guide/
│ ├── 01-distributed-training.md
│ ├── 02-memory-management.md
│ ├── 03-mixed-precision.md
│ └── 04-profiling.md
├── guides/
│ ├── distributed-training.md
│ ├── memory-management.md
│ ├── mixed-precision.md
│ └── profiling.md
├── index.html
├── interactive.html
├── requirements.txt
├── src/
│ ├── __init__.py
│ ├── __pycache__/
│ │ ├── __init__.cpython-312.pyc
│ │ ├── amp_trainer.cpython-312.pyc
│ │ ├── gpu_monitor.cpython-312.pyc
│ │ ├── memory_manager.cpython-312.pyc
│ │ └── profiler.cpython-312.pyc
│ ├── amp_trainer.py
│ ├── gpu_monitor.py
│ ├── memory_manager.py
│ └── profiler.py
└── tests/
├── __pycache__/
│ ├── conftest.cpython-312-pytest-9.0.2.pyc
│ ├── test_gpu_monitor.cpython-312-pytest-9.0.2.pyc
│ ├── test_memory_manager.cpython-312-pytest-9.0.2.pyc
│ └── test_profiler_and_amp.cpython-312-pytest-9.0.2.pyc
├── conftest.py
├── test_gpu_monitor.py
├── test_memory_manager.py
└── test_profiler_and_amp.py
Documentation Preview README excerpt
GPU Optimization Guide
A practical toolkit for squeezing maximum performance from NVIDIA GPUs during deep learning training. Covers CUDA memory management, mixed-precision training, gradient accumulation, distributed training (DDP/FSDP/DeepSpeed), profiling, OOM debugging, and GPU monitoring.
Features
- Memory Manager — CUDA memory snapshots, fragmentation analysis, leak detection, OOM diagnostics with actionable fix suggestions
- AMP Trainer — Drop-in mixed-precision training loop with gradient accumulation, GradScaler, and gradient checkpointing support
- GPU Monitor — Parse nvidia-smi output into structured Python objects, detect anomalies (idle GPUs, thermal throttling, memory pressure)
- Profiler — stdlib SectionTimer for quick bottleneck detection + PyTorch profiler wrapper for detailed Chrome-trace analysis
- Memory Benchmark — Automated comparison of fp32 vs AMP vs checkpointing across batch sizes
- Distributed Configs — Production-ready DDP, FSDP, and DeepSpeed ZeRO configurations with extensive inline comments
File Structure
gpu-optimization-guide/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│ ├── __init__.py # Package init
│ ├── memory_manager.py # CUDA memory tracking, OOM diagnostics
│ ├── amp_trainer.py # Mixed-precision training loop
│ ├── gpu_monitor.py # nvidia-smi parser + anomaly detection
│ └── profiler.py # SectionTimer + torch.profiler wrapper
├── configs/
│ ├── ddp_config.yaml # DistributedDataParallel configuration
│ ├── fsdp_config.yaml # FullyShardedDataParallel configuration
│ └── deepspeed_config.json # DeepSpeed ZeRO Stage 2 configuration
├── guides/
│ ├── memory-management.md # OOM debugging flowchart + optimization ladder
│ ├── mixed-precision.md # AMP deep dive with troubleshooting
│ ├── distributed-training.md # DDP vs FSDP vs DeepSpeed decision tree
│ └── profiling.md # Bottleneck identification workflow
└── benchmarks/
└── memory_benchmark.py # Automated memory/speed comparison
Quick Start
pip install -r requirements.txt
# Monitor GPU utilization
python src/gpu_monitor.py
# Run memory benchmark (requires CUDA GPU)
python benchmarks/memory_benchmark.py
Usage
Memory Tracking
from src.memory_manager import memory_tracking, format_bytes
with memory_tracking(label="training_step") as tracker:
loss = model(batch)
*... continues with setup instructions, usage examples, and more.*
Code Sample .py preview
src/amp_trainer.py
"""
amp_trainer — Mixed-precision (AMP) training loop with gradient accumulation.
Automatic Mixed Precision uses float16 for forward/backward passes and
float32 for the optimizer step. This typically:
- Cuts GPU memory usage by 30–50 % (tensors are half the size).
- Speeds up training by 1.5–3× on Ampere+ GPUs (which have TF32 cores).
- Requires NO changes to your model architecture.
The GradScaler handles loss scaling automatically to prevent float16
underflow in gradients.
This module provides a drop-in training function and a configurable
TrainerConfig dataclass.
"""
from __future__ import annotations
import logging
import time
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, List, Optional, Tuple
logger = logging.getLogger(__name__)
@dataclass
class TrainerConfig:
"""Configuration for the AMP training loop.
Attributes:
use_amp: Enable automatic mixed precision.
gradient_accumulation_steps:
Number of micro-batches to accumulate before an optimizer step.
Effective batch size = batch_size × accumulation_steps.
Set > 1 when the desired batch size doesn't fit in GPU memory.
max_grad_norm: Clip gradients to this L2 norm (0 = no clipping).
use_gradient_checkpointing:
Trade compute for memory: recompute activations during backward
pass instead of storing them. Reduces memory by ~60 % at the
cost of ~30 % slower training.
log_every_n_steps: How often to log training metrics.
amp_dtype: "float16" or "bfloat16". bfloat16 doesn't need
loss scaling but requires Ampere+ hardware.
"""
use_amp: bool = True
gradient_accumulation_steps: int = 1
max_grad_norm: float = 1.0
use_gradient_checkpointing: bool = False
log_every_n_steps: int = 10
# ... 197 more lines ...