Cross-Framework Performance Benchmarks

TL;DR. One harness drives nine participants — Lauren, FastAPI, Starlette, Litestar, BlackSheep, Sanic, Falcon, Quart and a hand-written ASGI floor — through five workloads, five repeats each, in-process through httpx.ASGITransport. Lauren lands mid-pack and stays there: 3rd to 6th of the eight frameworks measured on every workload, with a five-repeat range that overlaps Litestar's and Sanic's on all five workloads, FastAPI's on four and BlackSheep's and Falcon's on four, and is cleanly separated only from Quart. Starlette, Falcon and BlackSheep take the podium (in the top three on four or five workloads each); Quart is last on four of five. The hand-written floor — no framework in the request path at all — runs only 1.13x to 1.24x faster than the best measured framework, a margin narrower than the spread of the cells it compares: read that as "the whole framework layer costs a few tens of percent on this machine", not as a fixed factor the floor wins by. Cold start is where Lauren pays for its startup validation: 707 ms to a served request (median of five, spread 28%), against 552 ms for the floor, with 318 ms of that being imports and 98 ms constructing and validating the application. Two frameworks — aiohttp and robyn — could not be driven in-process at all and have no number here.

What this page reports

Deliberately absent: memory and RSS. They are out of scope for this comparison, so no memory figure appears anywhere on this page. This page reports throughput, latency and startup time, and nothing else.

How the numbers are produced

The harness lives in benches/. benches/bench.py is the entry point, benches/worker.py does one measurement, benches/workloads.py defines the requests and their expected answers, and benches/baseline_app.py is the framework-free floor.

One process per measurement

Every (framework, workload, repeat) runs in a freshly spawned interpreter with its own application and its own event loop. Nothing is shared between repeats or between frameworks.

That is not decoration. Several frameworks in this field patch shared classes at import time, and Sanic cannot stand up a second application in one interpreter at all — its touchup scheme compiles patched handler methods into a table keyed per app, and a second app in the same process leaves applications unable to start. An earlier revision of the harness built one fresh app per repeat inside a single process and watched Sanic fail on the second repeat. Rather than special-case one framework, every framework gets the same isolation.

The cost is real and is stated rather than hidden: one interpreter boot per repeat, which is why a full run takes about six minutes.

Rounds, not blocks

Repeats are interleaved round-robin: round r measures every framework and every workload before round r+1 starts. If the machine drifts — thermal throttling, a neighbour waking up — the drift lands on all participants instead of only on whoever was measured last.

The contract is re-checked in every timed process

The damaging outcome of a benchmark is not a wrong number, it is a fast wrong number. So each timed process re-checks its workload's status code and body against benches/workloads.py on the very application instance it is about to time, and the run aborts on a mismatch. Nothing here is timed that has not just answered correctly.

What each number means

Repeats use --warmup 0.25 (discarded) and --duration 0.5 (timed), one in-flight request at a time, and every statistic is the median of five repeats with its spread.

Warning: These are in-process numbers

httpx.ASGITransport calls the application directly: no socket, no server process, no TLS, no kernel scheduling. That is what makes the comparison repeatable and fair — every participant pays the same client cost, and no one gets a different server configuration — but it is not a deployed-configuration number. A uvicorn/gunicorn/granian deployment over a socket will move these numbers, and the ratio between frameworks can move with it. Read this page as "how much does the framework layer itself cost?", not as "what will my service do in production?".

The latency figures include the httpx client and the ASGI transport, not handler-only time, for the same reason: the client is common to everybody.

Reproducing this page

The pinned dependency set lives in benches/requirements-bench.txt, and one command prepares the environment and runs the harness:

nox -s bench_frameworks

It creates benches/.venv with Python 3.13, installs the pins (re-running the install every time, so the pins are enforced even if the venv has drifted), runs the harness, and prints the path of the artifact it wrote. Arguments after -- go straight to benches/bench.py:

nox -s bench_frameworks -- lauren fastapi          # measure a subset
nox -s bench_frameworks -- --repeat 1 --duration 0.1 --startup-repeats 0

The numbers on this page come from benches/results/frameworks-inprocess.json, committed next to the harness, which records the environment, the installed versions and a SHA-256 of every harness file it ran. If you only want to know whether every framework answers correctly, python benches/run.py re-runs the contract checks in seconds.

The machine and the dependency set

The pins matter to the numbers. Several frameworks switch JSON encoder on whichever fast library happens to be importable, so the same "framework version" can produce a different table on a different machine. The serialization table below records what each participant actually ran.

The five workloads

Throughput (requests per second)

Median of five repeats, higher is better. baseline is the hand-written ASGI callable with no framework in the path — the floor the harness itself can reach, included so you can tell framework cost from client cost. It is not a competitor, and its margin should not be read as one: the 1.13x–1.24x it holds over the best framework in this table is narrower than the spread on most of those cells, and re-running nox -s bench_frameworks has moved that margin by more than its own size — once putting the floor behind the best framework outright. Treat the floor as the reference point for what the client costs, not as a leaderboard entry.

The spread tells you how much of a difference to believe. Median spread across the 45 cells is 19.4%, the tightest is 5.5% (FastAPI's validation) and the widest is 70.0% (Starlette's). Treat any gap smaller than the spread of the cells you are comparing as unresolved: on this machine that means most of the mid-field ordering, which is why this page reports the medians and their dispersion side by side rather than a ranking.

Latency

Median of five repeats of the per-request round trip, in milliseconds, lower is better. The same nine participants, the same workloads.

One range does not fit the axis: Starlette's validation p99 reaches 10.50 ms, nearly four times the next-largest range in the artifact, so the latency error bars are capped at 2.7 ms. That range is cut off at the cap and carries its true maximum above a break marker, rather than flattening every other bar into the bottom of the chart.

p50

p95

p99

A word on the tail: p99 spreads of 40–80% are normal here, because a single slow repeat moves the maximum. The extreme is Starlette's validation, whose p99 median is 1.22 ms across repeats but whose worst repeat hit 10.50 ms (spread 802%). That is the one range the latency figure cannot draw to scale — its error bars are capped at 2.7 ms and the bar carries its true maximum above a break marker. Read this column as an order of magnitude — "does the framework add a millisecond or ten?" — not as a ranking.

Startup

Five cold processes per participant: import, build the application, and get a first request answered. All millisecond values are medians of those five; spread is across them.

process is what a user waits for: the parent measures the whole subprocess, so it includes interpreter boot and, at the end, the framework's teardown. The three phase columns come from inside the child and stop at the first answered request — they sum to child_total per sample, which excludes boot and teardown.

Note: Where the startup time actually goes

Every framework here spends 286–338 ms importing before it does anything else, and that term barely moves between them. The differences live in build_app and first_request: Starlette (34 ms) and Falcon (68 ms) wire themselves up fastest, FastAPI (310 ms), Quart (245 ms) and Litestar (382 ms) slowest, and Lauren's 98 ms is mid-field — its startup validation is a design trade (/docs/comparisons/Lauren vs FastAPI, Litestar & BlackSheep: a startup error is a CI failure, a runtime error is a 3 a.m. page), but this table does not show it as an outlier. The two real handshake costs are Litestar's and Sanic's first_request at 26 ms and 51 ms; everyone else answers their first request in under 6 ms.

Serialization stacks

Numbers are only comparable when the work is comparable. This table records the JSON path, the validation path and the injection mechanism each participant actually used in this run.

Not measured: aiohttp and robyn

Both distributions are installed in the benchmark environment and both are listed in the artifact's not_measured section. Neither can be driven in-process, so neither has a number anywhere on this page.

aiohttp

Installed here as aiohttp 3.14.5. Entry point: none — aiohttp.web.Application is not an ASGI application.

aiohttp.web.Application exposes no ASGI callable: its call is a gunicorn-compatibility stub that takes no arguments and returns self, and the distribution ships no ASGI adapter. The only PyPI shims (aiohttp-asgi, aiohttp-asgi-connector) adapt in the opposite direction — they host an ASGI app on aiohttp's server — so they cannot put an aiohttp app behind ASGITransport. Hand-writing the bridge would measure the bridge, not aiohttp.

robyn

Installed here as robyn 0.88.0. Entry point: none — Robyn is a facade over a compiled Rust HTTP server.

Robyn's Python object configures a compiled Rust server that owns its own socket: the object exposes start()/serve(), defines no call, and the package ships no ASGI module (its submodules are router, ws, responses, ... but nothing ASGI). There is no in-process request path to drive, so no comparable number can be produced without measuring Robyn's own server against a socket, which is a different benchmark.

If you need a number for either, it has to come from a different benchmark: one that stands up the framework's own server and drives it over a socket. That measures the server too, which is a legitimate question — just not the one this page asks.

See also

• Lauren vs FastAPI, Litestar & BlackSheep — the design comparison this page measures one axis of.
• benches/ — the harness, the feasibility probe, the pinned requirements and the raw artifact this page is generated from.
• Why Lauren? — the trade-offs the startup table above is a consequence of.