I've been working on a production deployment of a LangGraph application to Azure Container Apps, and I'm stuck on a persistent health check failure. My application builds and runs perfectly locally, and the core graph logic executes as expected when I manually trigger it. However, the Azure Container Apps health probe is consistently reporting unhealthy, which leads to constant restarts and, ultimately, a failed deployment.
My setup uses the standard FastAPI pattern recommended in the LangGraph documentation. I've configured both a `/health` endpoint and the root `/` to return a 200 status. From my audit logging perspective, I can see the health requests coming in, but the container's readiness probe still fails. I suspect the issue might be related to how the LangGraph agent or state graph is initialized during startup, potentially causing a blocking operation that starves the health endpoint thread.
Here is my current `Dockerfile` and the relevant portion of my `main.py`:
```dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]
```
```python
from fastapi import FastAPI
from langgraph.graph import StateGraph, END
# ... graph definition logic ...
app = FastAPI(title="MyLangGraphApp")
@app.get("/")
@app.get("/health")
async def health_check():
return {"status": "healthy"}
# Graph is constructed and compiled as a global variable on module load
workflow = StateGraph(MyState)
# ... add nodes, edges ...
app.state.graph = workflow.compile()
```
And my Azure Container Apps health probe configuration from the `containerapp.yaml`:
```yaml
template:
containers:
- image: myregistry.azurecr.io/mylanggraph:latest
probes:
- type: readiness
httpGet:
path: /health
port: 80
initialDelaySeconds: 10
periodSeconds: 5
```
My leading theories, based on similar issues I've seen with Splunk or DataDog monitoring agents that hang on startup, are:
* The graph compilation or underlying LLM client initialization is synchronous and heavy, potentially exceeding the `initialDelaySeconds` and causing timeouts on subsequent probes.
* There might be a dependency on an external service (like the OpenAI API) during startup that is not accounted for, and its latency or a transient failure is causing the health check to fail, even if the HTTP server itself is up.
* The default uvicorn worker configuration is single-threaded for async, and a long-running initialization task blocks the event loop.
I am looking for a definitive, auditable pattern. Specifically:
* What is the recommended way to structure the FastAPI app and LangGraph compilation to ensure liveness probes pass reliably?
* Should the graph be compiled asynchronously or lazily on first request, rather than at module load? If so, how do you manage the state and ensure thread safety?
* Are there specific Azure Container Apps probe settings (timeouts, failure thresholds) that have proven successful for stateful, potentially slow-to-initialize LangGraph applications?
* Has anyone implemented a more sophisticated health check that verifies not just the HTTP server, but also the readiness of the LangGraph runtime and its critical dependencies (e.g., LLM API, vector database)?
I have full audit logging enabled on the container side, so I can trace the exact sequence of events, but I need to know what to look for and what the correct implementation should be.
Logs don't lie.
Your suspicion about a blocking startup operation is a good starting point. Even though your health endpoint exists, the probe might be hitting it while the main application state graph is still being constructed, which could time out the request before it gets a response.
You mentioned you can see the health requests in your audit logs. That's key. Can you check the timestamp and duration of those logged requests? Compare them to the `initialDelaySeconds` and `periodSeconds` you have set for the readiness probe in your container app configuration. If the graph initialization takes, say, 45 seconds and your initial delay is only 30, the probe will start failing before the app is ready to even serve the health check.
Also, just to rule out the simple stuff, are you absolutely certain your container is exposing port 80? The default ACA ingress expects traffic on that port, but your CMD uses it for Uvicorn. Sometimes there's a mismatch between the container's exposed port and the port the ACA ingress targets.
I think you're on the right track with the blocking initialization suspicion, but your provided `main.py` is incomplete which makes it hard to be definitive. A common pattern I've seen in these deployments is that the LangGraph agent or state machine is instantiated at module level or during FastAPI app creation, before the server starts listening. This can hold up the entire ASGI worker.
You should verify the startup sequence by adding detailed logging right before and after the `app = FastAPI()` line, and again inside your health endpoint. Also, consider moving the heavy graph construction into a background task or a startup event with a clear "ready" flag, and have your `/health` endpoint check that flag. If initialization takes longer than the default probe timeout of Azure Container Apps, which is often 5-10 seconds, you'll need to adjust both the `initialDelaySeconds` and `timeoutSeconds` in your probe configuration, not just the initial delay.
infra nerd, cost hawk
You're right to suspect a blocking startup operation. Seeing the health requests in your logs is a good sign, but if the main thread is busy constructing your LangGraph agent, those requests can't be processed yet.
I'd follow the suggestion to move heavy initialization into a startup event with a flag. Your `/health` endpoint should then check that flag and only return a 200 once the graph is fully built and ready. This decouples the app's readiness from the server simply being online.
Also, double-check your container app's `initialDelaySeconds` against your actual startup time. If initialization takes 60 seconds and your delay is 30, the probe will fail from the very first attempt.
Read the guidelines before posting
The flag pattern is crucial, but you need to synchronize access to it properly if you're using async. A naive boolean check can lead to race conditions where the health probe passes while the graph is still partially initialized. Use an `asyncio.Event` instead of a plain flag.
Also, the bigger issue I see repeatedly is people misdiagnosing the timeout. The probe's `timeoutSeconds` is how long Azure waits for a TCP connection *and* the full HTTP response. If your app is single-threaded and blocked on graph init, the connection itself won't even be accepted, so the probe fails at the TCP layer, not the HTTP layer. That's why you see the request in your logs later, after the init finally finishes--it's queued backlog.
You can confirm this by checking the Azure console for the probe failure type; it often shows a connection timeout error, not a 5xx status code.
Show me the benchmarks.