← Back to all products

GPU Optimization Guide

$39

CUDA memory management, mixed precision training, distributed training configs, and GPU utilization monitoring.

38 files
PythonYAMLMarkdownJSON

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 38 files

gpu-optimization-guide/ ├── LICENSE ├── README.md ├── benchmarks/ │ ├── __pycache__/ │ │ └── memory_benchmark.cpython-312.pyc │ └── memory_benchmark.py ├── configs/ │ ├── ddp_config.yaml │ ├── deepspeed_config.json │ └── fsdp_config.yaml ├── free-sample.zip ├── guide/ │ ├── 01-distributed-training.md │ ├── 02-memory-management.md │ ├── 03-mixed-precision.md │ └── 04-profiling.md ├── guides/ │ ├── distributed-training.md │ ├── memory-management.md │ ├── mixed-precision.md │ └── profiling.md ├── index.html ├── interactive.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── amp_trainer.cpython-312.pyc │ │ ├── gpu_monitor.cpython-312.pyc │ │ ├── memory_manager.cpython-312.pyc │ │ └── profiler.cpython-312.pyc │ ├── amp_trainer.py │ ├── gpu_monitor.py │ ├── memory_manager.py │ └── profiler.py └── tests/ ├── __pycache__/ │ ├── conftest.cpython-312-pytest-9.0.2.pyc │ ├── test_gpu_monitor.cpython-312-pytest-9.0.2.pyc │ ├── test_memory_manager.cpython-312-pytest-9.0.2.pyc │ └── test_profiler_and_amp.cpython-312-pytest-9.0.2.pyc ├── conftest.py ├── test_gpu_monitor.py ├── test_memory_manager.py └── test_profiler_and_amp.py

Documentation Preview README excerpt

GPU Optimization Guide

A practical toolkit for squeezing maximum performance from NVIDIA GPUs during deep learning training. Covers CUDA memory management, mixed-precision training, gradient accumulation, distributed training (DDP/FSDP/DeepSpeed), profiling, OOM debugging, and GPU monitoring.

Features

  • Memory Manager — CUDA memory snapshots, fragmentation analysis, leak detection, OOM diagnostics with actionable fix suggestions
  • AMP Trainer — Drop-in mixed-precision training loop with gradient accumulation, GradScaler, and gradient checkpointing support
  • GPU Monitor — Parse nvidia-smi output into structured Python objects, detect anomalies (idle GPUs, thermal throttling, memory pressure)
  • Profiler — stdlib SectionTimer for quick bottleneck detection + PyTorch profiler wrapper for detailed Chrome-trace analysis
  • Memory Benchmark — Automated comparison of fp32 vs AMP vs checkpointing across batch sizes
  • Distributed Configs — Production-ready DDP, FSDP, and DeepSpeed ZeRO configurations with extensive inline comments

File Structure


gpu-optimization-guide/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│   ├── __init__.py              # Package init
│   ├── memory_manager.py        # CUDA memory tracking, OOM diagnostics
│   ├── amp_trainer.py           # Mixed-precision training loop
│   ├── gpu_monitor.py           # nvidia-smi parser + anomaly detection
│   └── profiler.py              # SectionTimer + torch.profiler wrapper
├── configs/
│   ├── ddp_config.yaml          # DistributedDataParallel configuration
│   ├── fsdp_config.yaml         # FullyShardedDataParallel configuration
│   └── deepspeed_config.json    # DeepSpeed ZeRO Stage 2 configuration
├── guides/
│   ├── memory-management.md     # OOM debugging flowchart + optimization ladder
│   ├── mixed-precision.md       # AMP deep dive with troubleshooting
│   ├── distributed-training.md  # DDP vs FSDP vs DeepSpeed decision tree
│   └── profiling.md             # Bottleneck identification workflow
└── benchmarks/
    └── memory_benchmark.py      # Automated memory/speed comparison

Quick Start


pip install -r requirements.txt

# Monitor GPU utilization
python src/gpu_monitor.py

# Run memory benchmark (requires CUDA GPU)
python benchmarks/memory_benchmark.py

Usage

Memory Tracking


from src.memory_manager import memory_tracking, format_bytes

with memory_tracking(label="training_step") as tracker:
    loss = model(batch)

*... continues with setup instructions, usage examples, and more.*

Code Sample .py preview

src/amp_trainer.py """ amp_trainer — Mixed-precision (AMP) training loop with gradient accumulation. Automatic Mixed Precision uses float16 for forward/backward passes and float32 for the optimizer step. This typically: - Cuts GPU memory usage by 30–50 % (tensors are half the size). - Speeds up training by 1.5–3× on Ampere+ GPUs (which have TF32 cores). - Requires NO changes to your model architecture. The GradScaler handles loss scaling automatically to prevent float16 underflow in gradients. This module provides a drop-in training function and a configurable TrainerConfig dataclass. """ from __future__ import annotations import logging import time from dataclasses import dataclass, field from typing import Any, Callable, Dict, List, Optional, Tuple logger = logging.getLogger(__name__) @dataclass class TrainerConfig: """Configuration for the AMP training loop. Attributes: use_amp: Enable automatic mixed precision. gradient_accumulation_steps: Number of micro-batches to accumulate before an optimizer step. Effective batch size = batch_size × accumulation_steps. Set > 1 when the desired batch size doesn't fit in GPU memory. max_grad_norm: Clip gradients to this L2 norm (0 = no clipping). use_gradient_checkpointing: Trade compute for memory: recompute activations during backward pass instead of storing them. Reduces memory by ~60 % at the cost of ~30 % slower training. log_every_n_steps: How often to log training metrics. amp_dtype: "float16" or "bfloat16". bfloat16 doesn't need loss scaling but requires Ampere+ hardware. """ use_amp: bool = True gradient_accumulation_steps: int = 1 max_grad_norm: float = 1.0 use_gradient_checkpointing: bool = False log_every_n_steps: int = 10 # ... 197 more lines ...