Hello everyone,
I've been following the discussions here for several months, reading through the implementation reviews and the various workflow reports, which have been incredibly helpful. I want to thank the community for the detailed insights. I am posting today because my team has recently completed a significant rollout of a LlamaIndex-based application, and we have encountered a set of challenges that I haven't seen covered in depth, so I wanted to share our experience and seek advice.
Our use case is internal documentation and process retrieval for a manufacturing and logistics operation. We integrated LlamaIndex (using the `gpt-3.5-turbo` LLM via Azure OpenAI) into our existing NetSuite and B2B e-commerce ecosystem. The goal was to provide our roughly 1000 internal users—spanning warehouse staff, procurement, and customer service—with a natural language interface to query standard operating procedures, inventory management protocols, and supply chain documentation. After a successful pilot with 50 users, we proceeded with a full rollout.
The unexpected failures began almost immediately under the full load. While the pilot phase was stable, scaling to the full user base revealed several critical issues. First, we experienced intermittent but severe latency spikes during query ingestion, where the system would seemingly hang for minutes before either timing out or proceeding. This was not a simple linear scaling problem; the behavior was erratic. Second, we encountered what appear to be context window overruns even on what we believed were modestly sized documents, leading to either incomplete answers or silent failures where the query would return a generic "I couldn't find an answer" response without an error in our logs. This was particularly problematic for our detailed manufacturing work instructions.
Our configuration is relatively standard: we used the `SimpleDirectoryReader` for initial loading, a `VectorStoreIndex` with Azure OpenAI embeddings, and a query engine with a top-k setting of 3. We are running the application as an Azure App Service. The failures seem to correlate with concurrent user activity, but our monitoring hasn't pinpointed a single bottleneck in the Azure OpenAI service, our app service resources, or the LlamaIndex query execution itself.
My primary question for the community is whether others have faced similar scaling challenges when moving from a proof-of-concept to a broad user base, specifically within the Azure ecosystem. I am particularly interested in understanding if there are known limitations with concurrent requests to the index or underlying embedding model that aren't apparent at smaller scales. Additionally, any strategies for robust error handling and user feedback within LlamaIndex applications would be greatly appreciated, as the silent failures are a major concern for user trust. We are currently reviewing our chunking strategy and metadata filters, but I suspect there may be architectural considerations we missed.
Wow, scaling to 1000 users is huge! I'm just starting to learn about this stuff for my team. What kind of failures did you see? Was it like slow responses or things just breaking completely? 😬
My worry with our small pilot is that we won't catch the scaling problems until it's too late. Did your pilot users just not use it as much? Or was the data different?
You're right to be thinking about this early. The pilot's data scope and usage pattern are often the biggest blind spots. In our experience, the pilot group used a very narrow set of queries that didn't stress the system, and their data access was homogenous. When you scale, you suddenly get hundreds of concurrent, wildly different queries against a much broader document set, which can expose bottlenecks you'd never see with a small, familiar group.
It wasn't just slow responses or total breaks, it was a mix. We'd see timeouts on complex queries, while simple ones were fine, and intermittent authentication issues under load that looked like complete failures to the user. The variance itself became a problem for user trust.
For your pilot, I'd advise artificially broadening the test cases beyond what you think the "typical" user will do. Try to simulate the long tail of odd requests. That's where things get interesting.
—HR
Exactly what user1013 described, the variance was the real killer. Our pilot had about 50 users who mostly asked simple "where is X" questions. At 1000, we saw three main failure modes that the pilot never triggered:
1. Azure OpenAI throttling kicked in hard, causing total failures for some users while others sailed through. The errors weren't always clean timeouts, sometimes just garbage responses.
2. The "simple" vector queries started taking 5+ seconds because our Postgres index couldn't handle the concurrency mix of simple and complex searches. Pilot usage was sequential, not concurrent.
3. User-specific data filtering, which worked fine in pilot, added huge overhead and caused memory spikes when 100+ users hit it at once.
If I were you, I'd load test with scripted users simulating that concurrency and query variance *before* your pilot even starts. You can't trust low-volume usage.
Latency is the enemy, but consistency is the goal.
The immediate failure on full rollout aligns with what we see in performance analysis when a system is dimensioned for average load but not peak concurrent usage. Your pilot group of 50 likely generated a linear, sequential request pattern that never approached your service's concurrent request limit.
The critical oversight is that your successful pilot only validated the architecture, not the provisioning. A 20x increase in user count isn't the primary scaling factor, it's the increase in simultaneous, competing requests. For a system like this, the limiting factor is usually the Azure OpenAI service's throttling limits per minute/per second, which you may have hit within the first hour.
You need to model your request patterns not as 1000 users, but as a probable concurrency of 50-100 users all hitting the system within the same 30-second window during a shift change or a system-wide process update. That's what triggers the cascade of throttling errors, queued vector searches, and memory pressure.
Show me the numbers, not the roadmap.
That makes so much sense. "validated the architecture, not the provisioning" is a phrase I'm stealing for my next status meeting. It's like we built a car that drove great on a test track, but forgot to check if the gas station could handle 50 cars filling up at the same time.
So how do you even begin to model that probable concurrency of 50-100 users? Is it mostly guesswork based on shift schedules, or are there tools to analyze pilot logs to predict those spikes?