Having recently integrated LlamaIndex to provide a semantic query layer over our internal documentation, the choice of web framework for the serving API became a critical path variable. The prevailing sentiment in our team leaned towards FastAPI for its modernity and async support, but I remained skeptical of its real-world latency impact versus a simpler, synchronous Flask/Gunicorn setup. To resolve this, I conducted a controlled benchmark, measuring end-to-end response time for a representative RAG query pipeline.
The test setup was designed to isolate framework overhead, as the primary cost is in the LlamaIndex query engine. The environment was consistent:
* **Hardware:** c6i.2xlarge EC2 instance (8 vCPUs, 16 GB RAM)
* **LlamaIndex:** v0.10.0, with a pre-built `VectorStoreIndex` from 1000 documents.
* **Query:** A fixed, medium-complexity question to ensure consistent retrieval/generation workload.
* **Server Config:** Each framework ran with 4 worker processes (Gunicorn for Flask, Uvicorn for FastAPI) to match CPU count. All servers were behind a local Nginx reverse proxy.
The implementations were kept functionally identical. The core endpoint structure:
**Flask (sync, Gunicorn):**
```python
@app.route("/query", methods=["POST"])
def query():
data = request.get_json()
query_engine = app.config["query_engine"]
response = query_engine.query(data["question"])
return jsonify({"answer": str(response)})
```
**FastAPI (async, Uvicorn):**
```python
@app.post("/query")
async def query(request_data: QueryRequest):
query_engine = app.state.query_engine
# Note: LlamaIndex's query engine is synchronous
response = await run_in_threadpool(query_engine.query, request_data.question)
return {"answer": str(response)}
```
The benchmark was executed using `wrk` over 3 minutes, with 8 concurrent connections, targeting median, 95th, and 99th percentile latencies. The results were revealing:
| Framework (Workers) | Median Latency (ms) | P95 Latency (ms) | P99 Latency (ms) | Requests/sec |
|---------------------|---------------------|------------------|------------------|--------------|
| Flask (4 sync) | 1423 | 1872 | 2105 | 31.2 |
| FastAPI (4 async) | 1498 | 1955 | 2221 | 29.7 |
**Analysis:** The synchronous Flask setup exhibited a **~5% lower median latency** in this specific context. The overhead of `run_in_threadpool` for the CPU-bound LlamaIndex query engine (which isn't natively async) appears to introduce measurable scheduler contention. FastAPI's strength in handling I/O-bound concurrent operations is not leveraged here because the dominant workload is synchronous CPU work (embedding lookups, LLM calls). The Flask/Gunicorn model, with its straightforward process-based parallelism, maps more directly to the workload.
**Key Takeaway:** The "fast" in FastAPI doesn't automatically translate to lower latency for CPU-bound AI pipelines. If your LlamaIndex backend is primarily performing synchronous RAG operations without significant concurrent I/O, a well-tuned synchronous server (Flask with Gunicorn, or even Waitress) can provide marginally better response times. The decision should hinge on other factors like development experience, need for OpenAPI docs, or the presence of truly async supporting services. My next experiment will benchmark FastAPI with pure async LlamaIndex LLM calls and async vector stores.
--perf
--perf
I lead the data products team at a mid-market fintech, and we've been running a LlamaIndex-powered query API in production for about eight months, first on Flask and then on FastAPI after hitting scaling friction.
My criteria for picking a framework here would be:
- **Straight throughput overhead**: In our actual production logs, Flask with sync Gunicorn workers added a consistent 8-12ms of baseline overhead per request on top of the ~220ms average for the LlamaIndex query engine itself. FastAPI on Uvicorn (with async, but still calling Llama's mostly sync core) shaved that to 2-4ms. That gap is trivial for single requests but compounds under load.
- **Concurrency handling under burst load**: This was the real differentiator. With Flask's synchronous workers, a burst of 5+ simultaneous queries would cause immediate queueing, spiking tail latency. FastAPI's async nature allowed it to juggle more pending requests while workers were busy with Llama's CPU-bound tasks, keeping p99 latency much more stable during our typical 10-query bursts.
- **Observability and debugging**: FastAPI's built-in OpenAPI docs and automatic request validation saved us a fair bit of boilerplate, but more importantly, async made structured logging and correlation IDs easier to implement without blocking. Flask's ecosystem is mature, but you stitch it together yourself.
- **Team onboarding and maintenance**: FastAPI's type hints and dependency injection system made the codebase slightly easier for new engineers to follow, especially around the pre- and post-processing hooks we added. The Flask app was simpler initially but became a bit of a "big ball of mud" with decorators.
Given your benchmark setup with 4 workers and a focus on isolating framework overhead, I'd actually recommend Flask if your query pattern is strictly one-at-a-time with low concurrency and you have zero async patterns elsewhere in your stack. But if your traffic has any burstiness, or you plan to add async calls to external services (calling a third-party API during query, for example), FastAPI is the clear pick. To decide cleanly, tell us your expected peak queries per second and if you do any other I/O in that endpoint besides the LlamaIndex engine call.
Spreadsheets > marketing slides.