Benchmarks¶

The benchmark suite exercises Traffik the way it actually runs in production: as a real server process, listening on a real socket, handling real HTTP and WebSocket connections, no just as a Python function called directly in the same process. This page documents how the suite works, how to run it, and how to read its output.

How It Works¶

Every benchmark run:

1. Spawns the target app as its own OS process. uvicorn for a single worker, or gunicorn (with --preload and the fork start method) for multiple workers.
2. Waits for the process to answer a health check before sending any traffic.
3. Drives real traffic against it: httpx2.AsyncClient over a TCP connection for HTTP/middleware scenarios, and the websockets library for WebSocket connections.
4. Resets the throttle backend's state via an admin endpoint between every iteration, so later iterations aren't skewed by counters left over from earlier ones.
5. Tears the process down before moving to the next scenario.

One process is spun up per scenario and reused across that scenario's warmup and timed iterations, then shut down. It isn't restarted for every single request, and it isn't shared across different scenarios.

Because the app runs as a real process, gunicorn's forked workers behave exactly as they would in a real deployment: MultiProcessInMemoryBackend.start() runs once in the master process before fork(), and every worker shares that state for real, rather than the benchmark merely asserting that it should.

Installation¶

The benchmark suite has its own dependency group so it doesn't bloat a normal install:

uv sync --group benchmark --inexact
# or
pip install "traffik[benchmark]"

This pulls in click, rich, fastapi, uvicorn, gunicorn (POSIX only), websockets, psutil (memory sampling, for scale), slowapi (for compare), and the backend client libraries. gunicorn isn't available on Windows as anything requiring more than one worker process needs a POSIX system (Linux or macOS).

If you want to benchmark against Redis or Memcached instead of the default in-memory backend, start real instances first:

docker compose up -d redis memcached

Any backend other than inmemory or multiprocess needs a real, reachable server. The suite does not stub these out.

Running Benchmarks¶

The suite is a click-based CLI with six commands: four benchmark one integration pattern each, and two (compare, scale) answer a different kind of question - how traffik stacks up against SlowAPI, and how it behaves as key cardinality grows.

python -m benchmarks http
python -m benchmarks middleware
python -m benchmarks websocket
python -m benchmarks multiprocess
python -m benchmarks compare
python -m benchmarks scale

Or via the Makefile shortcut, which forwards any arguments after bench:

make bench "http --scenarios below_limit,over_limit"

Each command accepts --help for the full option reference.

Common Options¶

These options are shared across http, middleware, websocket, and multiprocess. compare and scale overlap heavily but not exactly - see their own sections below (compare adds --mode and --endpoint; scale drops --iterations/--warmup/--scenarios and adds --checkpoints/--mp-max-keys).

Commands and Scenarios¶

http - Dependency-Based Throttling¶

Benchmarks the most common integration pattern: a throttle injected via Depends(throttle) on a single endpoint.

middleware - Middleware-Based Throttling¶

Benchmarks ThrottleMiddleware with a MiddlewareThrottle entry, applied without touching route handlers. Includes the same nine scenarios as http, plus:

websocket - WebSocket Throttling¶

Benchmarks a single throttled /ws endpoint over real WebSocket connections.

multiprocess - Real Multi-Worker State Sharing¶

Benchmarks MultiProcessInMemoryBackend across real, forked gunicorn workers (POSIX only). This command forces --backend multiprocess regardless of what --backend is passed, and reuses the same Depends-based endpoint as http. It includes the same nine scenarios as http, plus two that specifically stress the shared-memory backend:

compare - traffik vs SlowAPI, Under Matched Conditions¶

Runs each selected scenario against both a traffik app and a SlowAPI app, one after the other, and reports both side by side.

What's held identical between the two apps for a given run:

python -m benchmarks compare --backend inmemory --scenarios below_limit,hot_key,many_keys
python -m benchmarks compare --backend inmemory --strategy sliding_window_counter
python -m benchmarks compare --backend inmemory --mode middleware
python -m benchmarks compare --backend aioredis --redis-url redis://localhost:6379/0 --endpoint sync

scale - Memory Pressure and Scalability¶

Starts one traffik app server and keeps it running for the entire command - unlike every other command here, which restarts the server before each iteration. Growing the backend's distinct-key count while it keeps running is the point: a fresh process per measurement would hide exactly the memory accumulation and latency drift this command exists to show.

For each --checkpoints value (cumulative, ascending distinct-key counts), it sends that many new requests - each carrying a brand-new identity - concurrently (--concurrency in flight at once), then samples:

• The target process's resident memory (RSS), via psutil.
• For aioredis/coredis, also the Redis server's own INFO memory used_memory (best-effort; the app process's RSS mostly reflects connection overhead for these backends, not the key data itself).
• That batch's mean req/s, P50, and P99 latency - watch these for drift as the backend fills up, not just the memory columns. Real concurrent access at scale, not just raw key count, is half of what this command is for.

# Get a feel for it first - the default checkpoints go up to 1,000,000 and can take a while.
python -m benchmarks scale --backend inmemory --checkpoints 1000,10000

# Then something closer to production scale:
python -m benchmarks scale --backend inmemory --checkpoints 10000,100000,1000000 --concurrency 200
python -m benchmarks scale --backend multiprocess --checkpoints 10000,100000 --shards 64

--mp-max-keys (for --backend multiprocess) defaults to the highest --checkpoints value: that backend's shared-memory table is fixed-size, sized once at startup, so it can't grow past what it was told to expect. Sizing it to your largest checkpoint means the very first checkpoint's memory jump reflects the whole table's allocation up front, not a per-key cost - a real, useful thing to know about that backend's memory profile, not a benchmark artifact worth hiding.

The Bytes/key column is cumulative Δ RSS ÷ cumulative keys - a rough running average, not a precise per-key allocation figure (it also carries whatever fixed overhead the backend paid once at startup, amortized over however many keys exist by that checkpoint).

Backends¶

Set with --backend / -b:

Strategies¶

Set with --strategy / -s:

fixed_window, sliding_window_counter, sliding_window_log, token_bucket, token_bucket_debt, leaky_bucket, leaky_bucket_queue, gcra.

See Strategies for what each one does and when to reach for it.

Reading the Output¶

Table Output (default)¶

A rich-rendered table, one row per scenario, aggregated across all timed iterations (warmup iterations are discarded and never shown). Columns:

JSON Output¶

Pass --output json for machine-readable results - useful for feeding into your own reporting or tracking regressions over time in CI. The structure is:

{
  "meta": {
    "backend": "...",
    "strategy": "...",
    "iterations": 3,
    "warmup_iterations": 1,
    "workers": 1,
    "timestamp": "...",
    "platform": "...",
    "python_version": "..."
  },
  "results": [
    {
      "scenario_name": "...",
      "backend_kind": "...",
      "strategy_kind": "...",
      "iterations": 3,
      "total_requests": 0,
      "mean_rps": 0.0,
      "p50_ms": 0.0,
      "p95_ms": 0.0,
      "p99_ms": 0.0,
      "mean_ms": 0.0,
      "success_rate": 0.0,
      "throttle_rate": 0.0,
      "error_rate": 0.0,
      "rps_stddev": 0.0
    }
  ]
}

What To Expect¶

A few things are worth understanding before you interpret a run, so you don't mistake expected behaviour for a bug.

Success/throttle percentages should match the configured rate. For a scenario sending N requests against a rate that permits M of them, expect roughly M/N × 100 success and the rest throttled (barring the "many distinct keys" scenarios, where each key individually stays under its own limit and everything should succeed). If these don't line up, something's wrong with the run, not with your expectations.

Numbers reflect real network and process overhead - by design. Because this suite drives real HTTP/WebSocket traffic against a real server process, every request pays for a real TCP round trip, real HTTP/1.1 framing, and real ASGI request handling. That overhead did not exist in earlier versions of this suite (which called the ASGI app directly in-process) and won't disappear here - it's an accurate reflection of what a deployed instance actually costs per request, not a regression.

Concurrency and --workers only pay off with real CPU cores. asyncio concurrency helps most when there's real I/O wait time to overlap; on loopback that wait time is minimal, so a single worker process is largely CPU-bound on request parsing and routing. Multiple gunicorn workers only run in true parallel if there are separate physical cores for them to run on, i.e, on a single-core machine, --workers 4 will look barely different from --workers 1, because there's only one core for either to use. If you want to see --workers make a real difference, set it based on how many cores your machine actually has and compare against a --workers 1 run of the same scenario.

window_boundary scenarios can show some run-to-run variance. These scenarios time bursts around fixed-window edges, and real request latency plus real sleep timing can shift exactly where a burst lands relative to a window boundary. This is a genuine property of testing against real wall-clock timing, not a flaw in the scenario.

Warmup iterations are discarded on purpose. The first iteration against a freshly-started process can be slower (import caches warming, initial connection setup); warmup iterations exist to absorb that before timed iterations begin. Increase --warmup if you still see a slow first timed iteration.

Troubleshooting¶

ERROR: Could not start server for <scenario>: the spawned uvicorn/gunicorn process failed its health check within the startup timeout. The error includes a tail of the process's stderr; check it first. Common causes: a missing dependency for the selected backend, or a backend that requires a running external server (Redis/Memcached) that isn't reachable.

--workers greater than 1 fails outright: this requires a POSIX system. gunicorn's worker model relies on the fork start method, which Windows does not support. The multiprocess command is unavailable on Windows entirely for the same reason.

Connection refused for aioredis/coredis/aiomcache/emcache: start the relevant service first (docker compose up -d redis memcached), or point --redis-url / --memcached-host / --memcached-port at a server that's actually running.

A scenario reports a nonzero error rate: this means requests failed for a reason other than throttling (connection errors, timeouts, unexpected responses). It shouldn't happen in a healthy run; check the scenario's stderr output and the backend you selected.

compare/scale fail on import with a missing slowapi/psutil: these two are benchmark-only dependencies (see Installation) not needed by the other four commands; a plain uv sync --group benchmark or pip install psutil slowapi picks them up.

Extending the Suite¶

Scenarios are declarative specs, not hand-written functions. See benchmarks/scenarios.py. Adding a new scenario to an existing command means adding an entry to the relevant registry (HTTP_SCENARIOS, MIDDLEWARE_SCENARIOS, WEBSOCKET_SCENARIOS, or MULTIPROCESS_SCENARIOS) with a rate, request count, and traffic pattern (sequential, concurrent, waves, unique_keys_batched, unique_keys_split, or mixed_paths for HTTP-like scenarios). The actual traffic-generation logic lives in benchmarks/live/runners.py and is shared across every scenario of that shape. You shouldn't need to touch it to add a new scenario, only to add a genuinely new traffic pattern.