Performance & scaling¶
This page describes where the performance ceiling is and why. Numbers below are indicative (a developer laptop, loopback) — reproduce your own with the harness.
The wire path is Rust¶
The wire and TLS paths are axum / hyper / rustls, and the per-request Rust work
this crate adds — path-param extraction, query parsing, percent-decode, config
parse — is covered by the crate's benches
(benches/parse.rs,
cargo bench). Indicative single-core numbers:
| Path | Time |
|---|---|
path-param extraction (/users/{id}/orders/{order}) |
~135 ns |
| query parsing (4 params) | ~255 ns |
| percent-decode | ~35 ns |
http.yaml parse (boot / hot-reload) |
~5.5 µs |
A counting-allocator
leak check
(./scripts/mem_leak_test.sh) hammers these and asserts live bytes stay flat
(Δ 0 over 200k cycles). Against the harness's in-process mock, the driver +
loopback sustain ~270k req/s at sub-100 µs p50 — the driver ceiling, not a
real server. The Rust request path is not the throughput limit.
The limit is Python handler dispatch¶
Under load, aggregate throughput is bounded by the per-request Python handler dispatch, which runs in CPython. On a standard (GIL) interpreter it serialises to roughly one core, regardless of how many connections or cores you have.
Two things follow:
- Keep per-request handler work minimal. Push heavy lifting into Rust; do as little as possible in the Python handler on the hot path.
- Run against free-threaded CPython — this is the real unlock. On a free-threaded interpreter (3.13t / 3.14t) handlers run on every core and aggregate throughput scales with cores.
This is the same scaling characteristic as any Python-in-the-loop siphon addon — nothing HTTP-specific.
Free-threaded CPython is experimental
Free-threaded CPython support is still stabilising; treat it as experimental.
One executor job per request¶
A request runs through its middlewares and route handler as one job on
siphon's Python executor: the Request object is built, every stage runs and
the Response is read back inside a single interpreter attach. The tokio task
that accepted the connection never attaches. On free-threaded CPython a thread
that attaches takes part in every garbage-collector stop-the-world pause, so
keeping the listener tasks off the interpreter means a pause cannot stall them.
An async def stage that never awaits runs inline in the same job. One that
awaits moves, with the stages after it, to siphon's asyncio pool as one
coroutine.
benches/dispatch.rs
measures it against the previous dispatcher, which ran every stage as its own
executor job and attached on the tokio worker around each one. Per request, at
concurrency 1 (latency) and 64 (throughput), two runs each:
| Route | Previous | One job |
|---|---|---|
| 3 middlewares, c1 | 26–29 µs | 10.6–11.8 µs |
| 3 middlewares, c64 | 6.2 µs | 2.4 µs |
| no middleware, c1 | 10.5–10.9 µs | 10.3–12.7 µs |
| no middleware, c64 | 2.24 µs | 2.44 µs |
async def that never awaits, c64 |
2.5 µs | |
async def that awaits, c64 |
7.6–7.8 µs |
A route behind middleware gets about 2.5× faster. A bare route pays about 9 %
at saturation: building the Request object moved from the tokio worker into
the job, which is what keeps the worker off the interpreter. A handler that
awaits pays for its asyncio task; write a handler that does not await as def
or async def, the cost is the same.
Request timing metrics¶
siphon-http times every request it answers on siphon's metrics registry, next
to siphon's own metrics. Labels are listener (the servers[] name),
method (the nine standard methods, anything else other) and
status_class (2xx, 5xx, ...); never the path or anything from the
request.
| Metric | From | To |
|---|---|---|
http_server_request_seconds |
request head reaches the router | response body done (written, or dropped when the client went away) |
http_server_handler_wait_seconds |
request head reaches the router | the Python executor starts the middlewares and handler |
http_server_handler_seconds |
first middleware (or the handler) called | the reply produced |
http_server_async_handoff_wait_seconds |
an async def stage that awaits is handed to the asyncio pool |
its coroutine starts on a pool loop |
http_server_requests_in_flight (gauge, listener only) |
request head reaches the router | response body done |
request_seconds covers every answer: the 404 of an unknown path, the 503 of
a limits entry or the admission guard, the 504 of the request timeout. The
handler metrics are only recorded when a handler ran. So
request − handler_wait − handler is the time spent writing the response,
handler_wait is the executor queue, and async_handoff_wait is the asyncio
pool's queue, which handler_seconds includes.
The executor job and the asyncio hand-off only read the clock; every metric
update happens on the tokio task, never while attached to Python. The cost is
about 1 µs per request at saturation (single_job_timed in
benches/dispatch.rs, against single_job). siphon caps each metric at 128
label sets, which is about five listeners using five methods each; once a
metric is over the cap, further label sets are dropped and the first failure
is logged.
Scaling out vs. scaling up¶
| Lever | How | Effect |
|---|---|---|
| Scale out | Add replicas (a Kubernetes HPA on CPU does this — see Deployment). | The Rust path isn't the limit, so more pods means more aggregate throughput. |
| Scale up in a pod | Build siphon against free-threaded CPython. | Handlers run on every core inside one process. |
Reproducing it¶
The
harness/
reproduces both the GIL-bound and the free-threaded behaviour. It ships a
standalone load driver (http-load) and a mock server, so you can smoke-test the
driver on loopback and then drive real load at a siphon-http server.
Self-test (no siphon — measures the driver + loopback, not a real server):
Real load against a siphon-http server (build the HTTP-enabled binary, run it, then drive):
To exercise the free-threaded path, build siphon against a free-threaded
interpreter and drive the same load; handlers then run across cores instead of
serialising to one. See the harness README for the interpreter/LD_LIBRARY_PATH
details.