Benchmark method

The ladder

bench/apps.py defines ASGI apps where each rung adds one layer; all return a small JSON body.

rung	app	request
l0_raw	bare ASGI callable, fixed bytes	GET /
l1_starlette	one Starlette Route, JSONResponse	GET /
l1b_starlette_params	Starlette /items/{item_id:int} reading path and query	GET /items/42?q=hello
l1c_starlette_50routes	10 param plus 40 static Starlette routes	GET /r39, the last
l2_fastapi_dict	FastAPI route returning an annotated dict	GET /
l2b_fastapi_untyped	same, no return annotation	GET /
l2c_fastapi_included	same route via include_router	GET /
l5_fastapi_50routes	50 routes via add_api_route	GET /r39
l5b_fastapi_50routes_included	50 routes on an APIRouter, included	GET /r39
l3_fastapi_params	item_id: int path, optional q: str query	GET /items/42?q=hello
l4_fastapi_model	pydantic body with response_model	POST /items

A rung's µs/request minus l0_raw is the cost of everything above the server; bench.run prints it as vs l0.

Load

bench/run.py starts the server for one rung, warms it for 2 s, then runs oha three times:

oha -z 5s -c 64 --no-tui --output-format json http://127.0.0.1:8123/

A success rate below 100% aborts. It records requests/s, p50 and p99 per repeat; µs/request is 1,000,000 over the median requests/s. uvicorn runs with --loop uvloop --http httptools --no-access-log --log-level warning, granian with --interface asgi --workers 1 --loop uvloop --no-log; other server setups are tagged, as in l2_fastapi_dict[granian].

uv run python -m bench.run
BENCH_ONLY=l2_fastapi_dict,l3_fastapi_params uv run python -m bench.run

BENCH_REPEATS, BENCH_DURATION and BENCH_CONCURRENCY override 3, 5s and 64. oha must be on PATH; granian comes from the bench dependency group.

Profiles

BENCH_SAMPLE=1 runs macOS sample(1) on the server for 8 s under a 9 s load; bench/analyze_sample.py buckets top-of-stack frames by library (kernel, malloc, uvloop, httptools, pydantic_core, interpreter, other). BENCH_PYINSTRUMENT=1 wraps the app in pyinstrument (0.5 ms interval, async mode disabled) and writes HTML and text at shutdown. Output goes to bench/out/.

Before and after

Each change was measured in one window against the previous commit, stashed, at a load average low enough that repeats agree; later changes alternate after/before/after. Runs during a system stall (indexing, a backup, load average of 7 and above) were discarded and the commit says so; one commit's window was too noisy to measure at all. A gc.freeze() control left medians unchanged with p99 spikes present: the tail is sporadic stalls, not garbage collection.

A rung's same-window figure and its full-ladder figure differ by a few percent: the plain route is 17.1 µs in the last same-window run and 18.3 µs in results_ladder_v3.json. The per-change before/after figures on What changed are same-window runs; every table (landing page, benchmarks page, docs overview, README) quotes the full ladder files.

The single-machine ceiling

Multi-worker runs put the load generator and the server on the same cores. Six granian workers measured 140,000 requests/s with oha and about 160,000 with wrk; six uvicorn workers 153,000 and 175,000; twelve granian workers were slower than six, with p99 above 30 ms. These numbers describe the machine, not the framework. Cross-core scaling needs a separate client machine.

Raw files

Under bench/baseline/: results_ladder_v1.json (upstream FastAPI 0.141.1, day one), results_ladder_v3.json (current full ladder; v2 is older), results_fixN_before.json and results_fixN_after.json (one pair per change), results_servers_v1.json (uvicorn default and tuned, granian, granian --task-impl rust), results_workers_sweep_v1.json and wrk_vs_oha_w6_v1.txt (multi-worker), results_controls_v1.json and results_gc_freeze_check.json. Compare two files rung by rung:

uv run python bench/compare.py bench/baseline/results_fix1_before.json bench/baseline/results_fix1_after.json

Ladder v5

results_ladder_v5.json (2026-09-03 evening, AC power, master with all 41 changes) is assembled from three runs taken 15 minutes apart while the machine was in use: a full run, then two re-runs of the rungs a load burst had hit, accepted only when both raw ASGI floors matched the full run within 5 percent (13.5 to 13.9 µs on uvicorn, 7.9 on granian). Which run a rung comes from is recorded in the session notes; every row is a median of three 4 s oha runs at 64 connections.

Ladder v6

results_ladder_v6.json (2026-09-11 morning, AC power, master with all 43 changes) is one full run with BENCH_MAX_LOAD=6: before each rung the runner waited until the 1-minute load average was below 6, so no rung ran during a load burst. Every row is a median of three 5 s oha runs at 64 connections. Against v5 the uvicorn floor moved from 13.9 to 13.2 µs, so rows should be compared to their floor: the three-dependency route went from 9.5 to 8.9 µs above the uvicorn floor and from 5.2 to 3.2 µs above the granian floor (the compiled solver plan), and 50 routes via include_router from 3.0 to 2.0 µs above the uvicorn floor (direct dispatch of included routes).