Skip to content

Performance & scaling

This page describes where the performance ceiling is and why. Numbers below are indicative (a developer laptop, loopback) — reproduce your own with the harness.

The wire path is Rust

The wire and TLS paths are axum / hyper / rustls, and the per-request Rust work this crate adds — path-param extraction, query parsing, percent-decode, config parse — is covered by the crate's benches (benches/parse.rs, cargo bench). Indicative single-core numbers:

Path Time
path-param extraction (/users/{id}/orders/{order}) ~135 ns
query parsing (4 params) ~255 ns
percent-decode ~35 ns
http.yaml parse (boot / hot-reload) ~5.5 µs

A counting-allocator leak check (./scripts/mem_leak_test.sh) hammers these and asserts live bytes stay flat (Δ 0 over 200k cycles). Against the harness's in-process mock, the driver + loopback sustain ~270k req/s at sub-100 µs p50 — the driver ceiling, not a real server. The Rust request path is not the throughput limit.

The limit is Python handler dispatch

Under load, aggregate throughput is bounded by the per-request Python handler dispatch, which runs in CPython. On a standard (GIL) interpreter it serialises to roughly one core, regardless of how many connections or cores you have.

Two things follow:

  • Keep per-request handler work minimal. Push heavy lifting into Rust; do as little as possible in the Python handler on the hot path.
  • Run against free-threaded CPython — this is the real unlock. On a free-threaded interpreter (3.13t / 3.14t) handlers run on every core and aggregate throughput scales with cores.

This is the same scaling characteristic as any Python-in-the-loop siphon addon — nothing HTTP-specific.

Free-threaded CPython is experimental

Free-threaded CPython support is still stabilising; treat it as experimental.

One executor job per request

A request runs through its middlewares and route handler as one job on siphon's Python executor: the Request object is built, every stage runs and the Response is read back inside a single interpreter attach. The tokio task that accepted the connection never attaches. On free-threaded CPython a thread that attaches takes part in every garbage-collector stop-the-world pause, so keeping the listener tasks off the interpreter means a pause cannot stall them.

An async def stage that never awaits runs inline in the same job. One that awaits moves, with the stages after it, to siphon's asyncio pool as one coroutine.

benches/dispatch.rs measures it against the previous dispatcher, which ran every stage as its own executor job and attached on the tokio worker around each one. Per request, at concurrency 1 (latency) and 64 (throughput), two runs each:

Route Previous One job
3 middlewares, c1 26–29 µs 10.6–11.8 µs
3 middlewares, c64 6.2 µs 2.4 µs
no middleware, c1 10.5–10.9 µs 10.3–12.7 µs
no middleware, c64 2.24 µs 2.44 µs
async def that never awaits, c64 2.5 µs
async def that awaits, c64 7.6–7.8 µs

A route behind middleware gets about 2.5× faster. A bare route pays about 9 % at saturation: building the Request object moved from the tokio worker into the job, which is what keeps the worker off the interpreter. A handler that awaits pays for its asyncio task; write a handler that does not await as def or async def, the cost is the same.

Request timing metrics

siphon-http times every request it answers on siphon's metrics registry, next to siphon's own metrics. Labels are listener (the servers[] name), method (the nine standard methods, anything else other) and status_class (2xx, 5xx, ...); never the path or anything from the request.

Metric From To
http_server_request_seconds request head reaches the router response body done (written, or dropped when the client went away)
http_server_handler_wait_seconds request head reaches the router the Python executor starts the middlewares and handler
http_server_handler_seconds first middleware (or the handler) called the reply produced
http_server_async_handoff_wait_seconds an async def stage that awaits is handed to the asyncio pool its coroutine starts on a pool loop
http_server_requests_in_flight (gauge, listener only) request head reaches the router response body done

request_seconds covers every answer: the 404 of an unknown path, the 503 of a limits entry or the admission guard, the 504 of the request timeout. The handler metrics are only recorded when a handler ran. So request − handler_wait − handler is the time spent writing the response, handler_wait is the executor queue, and async_handoff_wait is the asyncio pool's queue, which handler_seconds includes.

The executor job and the asyncio hand-off only read the clock; every metric update happens on the tokio task, never while attached to Python. The cost is about 1 µs per request at saturation (single_job_timed in benches/dispatch.rs, against single_job). siphon caps each metric at 128 label sets, which is about five listeners using five methods each; once a metric is over the cap, further label sets are dropped and the first failure is logged.

Scaling out vs. scaling up

Lever How Effect
Scale out Add replicas (a Kubernetes HPA on CPU does this — see Deployment). The Rust path isn't the limit, so more pods means more aggregate throughput.
Scale up in a pod Build siphon against free-threaded CPython. Handlers run on every core inside one process.

Reproducing it

The harness/ reproduces both the GIL-bound and the free-threaded behaviour. It ships a standalone load driver (http-load) and a mock server, so you can smoke-test the driver on loopback and then drive real load at a siphon-http server.

Self-test (no siphon — measures the driver + loopback, not a real server):

./run.sh self-test

Real load against a siphon-http server (build the HTTP-enabled binary, run it, then drive):

./run.sh drive --host 127.0.0.1 --port 8080 --count 1000000 --concurrency 128

To exercise the free-threaded path, build siphon against a free-threaded interpreter and drive the same load; handlers then run across cores instead of serialising to one. See the harness README for the interpreter/LD_LIBRARY_PATH details.