Hitting a wall with Azure OpenAI and LlamaIndex where every call seems to time out? You're not alone. I've been integrating this into a pipeline for internal documentation, and the default LlamaIndex setup kept failing with `APITimeoutError`. The core issue is often that LlamaIndex's default timeouts don't play nice with Azure's routing and potential cold starts.
After some digging, the fix usually involves tweaking the `AzureOpenAI` client configuration directly. You need to pass a custom `httpx` client with longer timeouts. Here's the working pattern I landed on:
```python
import httpx
from llama_index.llms import AzureOpenAI
from llama_index.embeddings import AzureOpenAIEmbedding
llm_timeout = httpx.Timeout(30.0, connect=10.0)
llm_client = httpx.Client(timeout=llm_timeout)
llm = AzureOpenAI(
model="gpt-35-turbo",
deployment_name="your-deployment",
api_key="your-key",
azure_endpoint="https://your-resource.openai.azure.com/",
api_version="2024-02-15-preview",
http_client=llm_client, # Critical part
)
embed_model = AzureOpenAIEmbedding(
model="text-embedding-ada-002",
deployment_name="your-embedding-deployment",
api_key="your-key",
azure_endpoint="https://your-resource.openai.azure.com/",
api_version="2024-02-15-preview",
http_client=llm_client, # Apply here too
)
```
Key adjustments I made:
* Bumped the overall timeout to 30 seconds.
* Set a explicit connect timeout.
* Used the same `httpx.Client` for both LLM and embedding models for consistency.
If you're still seeing issues, check your Azure region's latency and consider:
* Is your function/app running in the same Azure region as your OpenAI resource? If not, network hops add significant delay.
* Are you behind a VNet or firewall that might be inspecting traffic? That can introduce unexpected delays.
* For production, implement a retry logic wrapper in addition to the longer timeout.
Has anyone else run into this and found a different fix, perhaps at the Azure network layer? I'm curious if setting up a Private Endpoint for the Azure OpenAI resource made a difference for anyone's timeout issues.
terraform and chill
Your fix works until Azure's regional load balancer decides to take a 45-second coffee break. Seen it happen. Also, if you're using async, you'll need to pass `httpx.AsyncClient` with the same timeout config, not the sync client. They'll both throw the same obscure error if you mix them up.
The real sin here is libraries hiding network defaults. Thirty seconds is a lifetime. You should also be setting retries with backoff on that client, because Azure's autoscaling can fail a request or two while it spins up new pods.
Prove it.