Python Performance Guide

A practical guide to finding and fixing performance bottlenecks in Python applications.

The Profiling Workflow

Performance optimization should always follow this order:

1. Measure — Profile before changing anything

2. Identify — Find the actual bottleneck (not the suspected one)

3. Optimize — Apply targeted fixes

4. Verify — Benchmark before and after

Step 1: CPU Profiling

Start with cProfile to find which functions consume the most time:

from src.profilers.cpu_profiler import CPUProfiler

profiler = CPUProfiler()
with profiler.profile():
    run_your_application()

profiler.print_stats(top_n=20, sort_by="cumulative")

Look for functions with high cumulative_time — they're the bottleneck entry points.

Step 2: Memory Profiling

If your application uses too much memory or leaks:

from src.profilers.memory_profiler import MemoryProfiler

profiler = MemoryProfiler()
with profiler.track():
    process_data()

profiler.print_summary(top_n=10)
report = profiler.detect_leaks()

Step 3: Line-Level Analysis

Once you've identified the slow function, profile it line by line:

from src.profilers.line_profiler import LineProfiler

lp = LineProfiler()

@lp.profile
def process_records(records):
    ...

process_records(data)
lp.print_stats()

Common Bottlenecks and Fixes

1. String Concatenation in Loops

Slow: result += string in a loop (O(n^2) total)

Fast: "".join(parts) (O(n) total)

2. N+1 Database Queries

Slow: Querying related objects one at a time

Fast: Batch fetch with WHERE id IN (...) or ORM eager loading

3. Repeated Computation

Slow: Recomputing the same value in every function call

Fast: Use @ttl_cache or @memoize from src/optimizers/caching.py

4. Blocking I/O in Loops

Slow: Sequential HTTP requests to multiple APIs

Fast: asyncio.gather() or parallel_map() from src/optimizers/batch_processing.py

5. Large Data Copies

Slow: Creating new lists/dicts when filtering

Fast: Use generators, itertools, or views

asyncio vs threading vs multiprocessing

Rule of thumb:

• Waiting for network/disk? Use asyncio or threading
• Crunching numbers? Use multiprocessing
• Mixed workload? Use asyncio for I/O + ProcessPoolExecutor for CPU

Algorithmic Optimization

Before optimizing code, check your algorithm's time complexity:

*append is amortized O(1), insert at index is O(n)

Key Optimizations

• Replace if x in list with if x in set for large collections
• Use collections.Counter instead of manual counting loops
• Use heapq.nlargest() instead of sorting when you need top-K
• Use bisect.insort() for maintaining sorted order (see SortedList)

C Extensions and Alternatives

When pure Python isn't fast enough:

1. NumPy — Vectorized array operations (100x faster for math)

2. Cython — Compile Python to C with type declarations

3. PyPy — Alternative interpreter with JIT compilation

4. Rust/C extensions — Write critical paths in compiled languages

By Datanest Digital | Python Performance Toolkit v1.0.0