AI API Gateway

Chapter 1

Deployment Guide — AI API Gateway

A practical guide to deploying the gateway in different environments, from local development to production.

Architecture Overview

  Your Application
        │
        ▼
┌───────────────────────────────────────────────┐
│              GatewayClient                    │
│                                               │
│  ┌──────────┐  ┌──────────┐  ┌────────────┐  │
│  │  Cache    │  │  Rate    │  │  Request   │  │
│  │  Layer    │  │  Limiter │  │  Logger    │  │
│  └──────────┘  └──────────┘  └────────────┘  │
│                                               │
│  ┌────────────────────────────────────────┐   │
│  │          Retry Router                  │   │
│  │  ┌──────┐  ┌──────────┐  ┌──────┐     │   │
│  │  │OpenAI│→ │Anthropic │→ │Local │     │   │
│  │  └──────┘  └──────────┘  └──────┘     │   │
│  └────────────────────────────────────────┘   │
│                                               │
│  ┌────────────────────────────────────────┐   │
│  │          Usage Analytics               │   │
│  └────────────────────────────────────────┘   │
└───────────────────────────────────────────────┘

Local Development

Minimal Setup (One Provider)

python

from src.client import GatewayClient
from src.providers.openai_provider import OpenAIProvider

gateway = GatewayClient(
    providers=[OpenAIProvider(api_key="YOUR_API_KEY_HERE")],
    enable_logging=False,  # Less noise during development
)

answer = gateway.chat("What is the meaning of life?")
print(answer)

With a Local LLM Server

If you're running a local model (Ollama, vLLM, llama.cpp), use the LocalProvider:

python

from src.providers.local_provider import LocalProvider

# Ollama
gateway = GatewayClient(
    providers=[LocalProvider(base_url="http://localhost:11434/v1", default_model="llama3")],
    enable_rate_limiting=False,  # Local server, no API quota
)

# vLLM
gateway = GatewayClient(
    providers=[LocalProvider(base_url="http://localhost:8000/v1")],
)

Production Deployment

Multi-Provider with Fallback

python

from src.client import GatewayClient
from src.providers.openai_provider import OpenAIProvider
from src.providers.anthropic_provider import AnthropicProvider
from src.providers.local_provider import LocalProvider

gateway = GatewayClient(
    providers=[
        # Primary — fastest and cheapest for most tasks
        OpenAIProvider(
            api_key=os.environ["OPENAI_API_KEY"],
            default_model="gpt-4o-mini",
        ),
        # Fallback 1 — used when OpenAI is down or rate-limited
        AnthropicProvider(
            api_key=os.environ["ANTHROPIC_API_KEY"],
            default_model="claude-sonnet-4-20250514",
        ),
        # Fallback 2 — local model for total API outage
        LocalProvider(
            base_url="http://internal-llm.docs.example.com:8000/v1",
        ),
    ],
    enable_cache=True,
    cache_path="/var/lib/gateway/cache.json",
    log_path="/var/log/gateway/requests.jsonl",
    max_retries=3,
)

Environment Variables

Never hardcode API keys. Use environment variables:

bash

export OPENAI_API_KEY="sk-YOUR_KEY_HERE"
export ANTHROPIC_API_KEY="sk-ant-YOUR_KEY_HERE"

Rate Limiting Configuration

Set per-provider rate limits to stay within your API quota:

python

gateway = GatewayClient(providers=[])

# OpenAI Tier 2: ~500 RPM → ~8 RPS
gateway.add_provider(
    OpenAIProvider(api_key=os.environ["OPENAI_API_KEY"]),
    rate_limit_capacity=30,   # Burst of 30
    rate_limit_refill=8.0,    # 8 req/sec sustained
)

# Anthropic: ~1000 RPM → ~16 RPS
gateway.add_provider(
    AnthropicProvider(api_key=os.environ["ANTHROPIC_API_KEY"]),
    rate_limit_capacity=50,
    rate_limit_refill=16.0,
)

Monitoring

Log Analysis

The gateway writes structured JSONL logs. Analyse them with standard tools:

bash

# Count requests by provider
jq -r '.provider' gateway_requests.jsonl | sort | uniq -c | sort -rn

# Find slow requests (>2s)
jq 'select(.latency_ms > 2000)' gateway_requests.jsonl

# Count errors
jq 'select(.error != "")' gateway_requests.jsonl | wc -l

# Cost estimate (rough)
jq '[.prompt_tokens + .completion_tokens] | add' gateway_requests.jsonl

Analytics Reports

Generate analytics reports programmatically:

python

from src.analytics import UsageAnalytics

analytics = UsageAnalytics()
analytics.load_from_file("gateway_requests.jsonl")

# Per-provider breakdown
for stat in analytics.by_provider():
    print(f"{stat.provider}: {stat.total_requests} reqs, "
          f"{stat.error_rate:.1%} errors, "
          f"{stat.avg_latency_ms:.0f}ms avg")

# Full Markdown report
report = analytics.summary_markdown()
with open("analytics_report.md", "w") as f:
    f.write(report)

Health Checks

Use in a health check endpoint or monitoring script:

python

health = gateway.health()
# {'openai': True, 'anthropic': True, 'local': False}

provider_health = gateway.provider_health
# {'openai': {'total_requests': 1024, 'total_failures': 3, 'error_rate': 0.003, ...}}

Circuit Breaker Behaviour

The retry router includes a circuit breaker that protects against cascading failures:

1. Closed (normal): Requests flow through normally.

2. Open (tripped): After N consecutive failures (default: 5), the circuit opens. No requests are sent to the provider for a cooldown period (default: 60s).

3. Half-open (testing): After the cooldown, the next request is allowed through. If it succeeds, the circuit closes. If it fails, the circuit re-opens.

Normal → 5 failures → Open (60s) → Half-open → Success → Normal
                                              → Failure → Open (60s)

Scaling Considerations

Concern	Solution
Cache size	Set `max_entries` based on memory budget. 5K entries ≈ 50-100MB.
Log file size	Rotate `gateway_requests.jsonl` with logrotate or a cron job.
Thread safety	The rate limiter is thread-safe. The cache and logger use append-only writes.
Multiple processes	Each process gets its own cache and rate limiter state. For shared state, use Redis (not included).

Troubleshooting

"Rate limit timeout" errors

Increase rate_limit_capacity or rate_limit_refill for the provider.
Check your API tier and upgrade if needed.

All providers failing

Run gateway.health() to check provider connectivity.
Verify API keys are set correctly.
Check the JSONL log for error details.

Cache not helping

Check gateway.cache_stats for hit rate.
If hit rate is low, your queries may be too unique (each prompt is different).
Try setting cache_temperature_zero_only=True and using temperature=0 for deterministic queries.

Chapter 2

🔒 Available in full product

Deployment Guide — AI API Gateway

You’ve reached the end of the free preview

Get the full AI API Gateway and unlock everything.

All Chapters

Get the complete guide with every chapter unlocked, including code samples, diagrams, and best practices.

Full Tool Suite

Access all interactive tools with complete data, all workload profiles, and the full scenario library.

Source Files

Downloadable source code, configuration files, and working examples from every chapter.

Lifetime Updates

Free updates for life. Every new chapter, tool, and improvement included.

Buy Now — $39 →
📦 Free sample included — download another copy or visit the store for the full product.

AI API Gateway v1.0.0 — Free Preview

Contents

Deployment Guide — AI API Gateway

Architecture Overview

Local Development

Minimal Setup (One Provider)

With a Local LLM Server

Production Deployment

Multi-Provider with Fallback

Environment Variables

Rate Limiting Configuration

Monitoring

Log Analysis

Analytics Reports

Health Checks

Circuit Breaker Behaviour

Scaling Considerations

Troubleshooting

"Rate limit timeout" errors

All providers failing

Cache not helping

Deployment Guide — AI API Gateway

You’ve reached the end of the free preview

All Chapters

Full Tool Suite

Source Files

Lifetime Updates