Benchmarks¶

The benchmark suite exercises Traffik the way it actually runs in production: as a real server process, listening on a real socket, handling real HTTP and WebSocket connections, no just as a Python function called directly in the same process. This page documents how the suite works, how to run it, and how to read its output.

How It Works¶

Every benchmark run:

1. Spawns the target app as its own OS process. uvicorn for a single worker, or gunicorn (with --preload and the fork start method) for multiple workers.
2. Waits for the process to answer a health check before sending any traffic.
3. Drives real traffic against it: httpx2.AsyncClient over a TCP connection for HTTP/middleware scenarios, and the websockets library for WebSocket connections.
4. Resets the throttle backend's state via an admin endpoint between every iteration, so later iterations aren't skewed by counters left over from earlier ones.
5. Tears the process down before moving to the next scenario.

One process is spun up per scenario and reused across that scenario's warmup and timed iterations, then shut down. It isn't restarted for every single request, and it isn't shared across different scenarios.

Because the app runs as a real process, gunicorn's forked workers behave exactly as they would in a real deployment: MultiProcessInMemoryBackend.start() runs once in the master process before fork(), and every worker shares that state for real, rather than the benchmark merely asserting that it should.

Installation¶

The benchmark suite has its own dependency group so it doesn't bloat a normal install:

uv sync --group benchmark --inexact
# or
pip install "traffik[benchmark]"

This pulls in click, rich, fastapi, uvicorn, gunicorn (POSIX only), websockets, and the backend client libraries. gunicorn isn't available on Windows as anything requiring more than one worker process needs a POSIX system (Linux or macOS).

If you want to benchmark against Redis or Memcached instead of the default in-memory backend, start real instances first:

docker compose up -d redis memcached

Any backend other than inmemory or multiprocess needs a real, reachable server. The suite does not stub these out.

Running Benchmarks¶

The suite is a click-based CLI with four commands, one per integration pattern:

python -m benchmarks http
python -m benchmarks middleware
python -m benchmarks websocket
python -m benchmarks multiprocess

Or via the Makefile shortcut, which forwards any arguments after bench:

make bench "http --scenarios below_limit,over_limit"

Each command accepts --help for the full option reference.

Common Options¶

These options are shared across all four commands:

Commands and Scenarios¶

http - Dependency-Based Throttling¶

Benchmarks the most common integration pattern: a throttle injected via Depends(throttle) on a single endpoint.

middleware - Middleware-Based Throttling¶

Benchmarks ThrottleMiddleware with a MiddlewareThrottle entry, applied without touching route handlers. Includes the same nine scenarios as http, plus:

websocket - WebSocket Throttling¶

Benchmarks a single throttled /ws endpoint over real WebSocket connections.

multiprocess - Real Multi-Worker State Sharing¶

Benchmarks MultiProcessInMemoryBackend across real, forked gunicorn workers (POSIX only). This command forces --backend multiprocess regardless of what --backend is passed, and reuses the same Depends-based endpoint as http. It includes the same nine scenarios as http, plus two that specifically stress the shared-memory backend:

Backends¶

Set with --backend / -b:

Strategies¶

Set with --strategy / -s:

fixed_window, sliding_window_counter, sliding_window_log, token_bucket, token_bucket_debt, leaky_bucket, leaky_bucket_queue, gcra.

See Strategies for what each one does and when to reach for it.

Reading the Output¶

Table Output (default)¶

A rich-rendered table, one row per scenario, aggregated across all timed iterations (warmup iterations are discarded and never shown). Columns:

JSON Output¶

Pass --output json for machine-readable results - useful for feeding into your own reporting or tracking regressions over time in CI. The structure is:

{
  "meta": {
    "backend": "...",
    "strategy": "...",
    "iterations": 3,
    "warmup_iterations": 1,
    "workers": 1,
    "timestamp": "...",
    "platform": "...",
    "python_version": "..."
  },
  "results": [
    {
      "scenario_name": "...",
      "backend_kind": "...",
      "strategy_kind": "...",
      "iterations": 3,
      "total_requests": 0,
      "mean_rps": 0.0,
      "p50_ms": 0.0,
      "p95_ms": 0.0,
      "p99_ms": 0.0,
      "mean_ms": 0.0,
      "success_rate": 0.0,
      "throttle_rate": 0.0,
      "error_rate": 0.0,
      "rps_stddev": 0.0
    }
  ]
}

What To Expect¶

A few things are worth understanding before you interpret a run, so you don't mistake expected behaviour for a bug.

Success/throttle percentages should match the configured rate. For a scenario sending N requests against a rate that permits M of them, expect roughly M/N × 100 success and the rest throttled (barring the "many distinct keys" scenarios, where each key individually stays under its own limit and everything should succeed). If these don't line up, something's wrong with the run, not with your expectations.

Numbers reflect real network and process overhead - by design. Because this suite drives real HTTP/WebSocket traffic against a real server process, every request pays for a real TCP round trip, real HTTP/1.1 framing, and real ASGI request handling. That overhead did not exist in earlier versions of this suite (which called the ASGI app directly in-process) and won't disappear here - it's an accurate reflection of what a deployed instance actually costs per request, not a regression.

Concurrency and --workers only pay off with real CPU cores. asyncio concurrency helps most when there's real I/O wait time to overlap; on loopback that wait time is minimal, so a single worker process is largely CPU-bound on request parsing and routing. Multiple gunicorn workers only run in true parallel if there are separate physical cores for them to run on, i.e, on a single-core machine, --workers 4 will look barely different from --workers 1, because there's only one core for either to use. If you want to see --workers make a real difference, set it based on how many cores your machine actually has and compare against a --workers 1 run of the same scenario.

window_boundary scenarios can show some run-to-run variance. These scenarios time bursts around fixed-window edges, and real request latency plus real sleep timing can shift exactly where a burst lands relative to a window boundary. This is a genuine property of testing against real wall-clock timing, not a flaw in the scenario.

Warmup iterations are discarded on purpose. The first iteration against a freshly-started process can be slower (import caches warming, initial connection setup); warmup iterations exist to absorb that before timed iterations begin. Increase --warmup if you still see a slow first timed iteration.

Troubleshooting¶

ERROR: Could not start server for <scenario>: the spawned uvicorn/gunicorn process failed its health check within the startup timeout. The error includes a tail of the process's stderr; check it first. Common causes: a missing dependency for the selected backend, or a backend that requires a running external server (Redis/Memcached) that isn't reachable.

--workers greater than 1 fails outright: this requires a POSIX system. gunicorn's worker model relies on the fork start method, which Windows does not support. The multiprocess command is unavailable on Windows entirely for the same reason.

Connection refused for aioredis/coredis/aiomcache/emcache: start the relevant service first (docker compose up -d redis memcached), or point --redis-url / --memcached-host / --memcached-port at a server that's actually running.

A scenario reports a nonzero error rate: this means requests failed for a reason other than throttling (connection errors, timeouts, unexpected responses). It shouldn't happen in a healthy run; check the scenario's stderr output and the backend you selected.

Extending the Suite¶

Scenarios are declarative specs, not hand-written functions. See benchmarks/scenarios.py. Adding a new scenario to an existing command means adding an entry to the relevant registry (HTTP_SCENARIOS, MIDDLEWARE_SCENARIOS, WEBSOCKET_SCENARIOS, or MULTIPROCESS_SCENARIOS) with a rate, request count, and traffic pattern (sequential, concurrent, waves, unique_keys_batched, unique_keys_split, or mixed_paths for HTTP-like scenarios). The actual traffic-generation logic lives in benchmarks/live/runners.py and is shared across every scenario of that shape. You shouldn't need to touch it to add a new scenario, only to add a genuinely new traffic pattern.