Python Logging Guide

Why Structured Logging?

Plain-text logs are hard to search, filter, and alert on at scale. Structured

logging (JSON) makes every log line a queryable record — essential for

production observability.

Configuration Strategy

Use environment-specific YAML configs:

configs/
├── logging_dev.yaml   # Colorized console, DEBUG level
├── logging_prod.yaml  # JSON to stdout + file, INFO level
└── logging_test.yaml  # WARNING only, minimal output

Load with one call:

from src.setup import configure_logging
configure_logging("prod")

Request Context Propagation

Every log line should identify which request it belongs to. This toolkit

uses contextvars (works with asyncio, threads, and sync code):

from src.context import set_context, generate_request_id

# In middleware (automatic):
set_context(request_id=generate_request_id())

# In your code:
set_context(user_id="usr-42", tenant="acme")

# Every log line now includes these fields automatically

Middleware Integration

FastAPI / Starlette (ASGI)

from src.middleware import ASGILoggingMiddleware
app.add_middleware(ASGILoggingMiddleware)

Flask / Django (WSGI)

from src.middleware import WSGILoggingMiddleware
app.wsgi_app = WSGILoggingMiddleware(app.wsgi_app)

Both middleware:

1. Generate or extract request_id from headers

2. Set context variables

3. Log request start/finish with duration

4. Clean up context after the request

Filtering Noise

Suppress third-party loggers

from src.filters import SuppressLoggerFilter
handler.addFilter(SuppressLoggerFilter(["urllib3", "botocore", "asyncio"]))

Rate-limit repeated messages

from src.filters import RateLimitFilter
handler.addFilter(RateLimitFilter(period_seconds=60))

Performance Tips

1. Use QueueHandler in production — create_async_handler() offloads

I/O to a background thread so logging never blocks your request.

2. Set appropriate levels — Don't log DEBUG in production. Every log

line has a cost (CPU, I/O, storage).

3. Use lazy formatting — logger.info("User %s", user_id) is faster

than logger.info(f"User {user_id}") because formatting is skipped

if the level is disabled.

4. Rotate logs — Use RotatingFileHandler or TimedRotatingFileHandler

to prevent disk exhaustion.

Common Patterns

Add context to exceptions

try:
    process_order(order_id)
except Exception:
    logger.exception("Failed to process order", extra={"order_id": order_id})

Structured extra fields

logger.info(
    "Payment processed",
    extra={"amount": 49.99, "currency": "USD", "order_id": "ORD-123"},
)

Conditional logging

if logger.isEnabledFor(logging.DEBUG):
    # Expensive serialization only when DEBUG is active
    logger.debug("Full payload: %s", json.dumps(payload, indent=2))

Anti-Patterns

1. print() instead of logging — No levels, no formatting, no routing.

2. Logging sensitive data — Never log passwords, tokens, or PII.

3. Catching and logging without re-raising — Swallows the error.

4. String concatenation in log calls — Use %s placeholders instead.

5. One logger for everything — Use logging.getLogger(__name__) per module.