I've been trying Cursor on my team's monorepo, and the latency when asking for refactors or searching across files is really noticeable. It seems to get bogged down.
I understand it's analyzing context, but compared to local models or even some other AI-assisted IDEs, it feels slow. Is this primarily because it's sending a large context window to the cloud for every request? Would using their local model option (like Claude 3.5 Haiku) speed things up for a large codebase, or is the bottleneck something else?
Still learning.
You've correctly identified the primary latency source for cloud-based models. Every request, especially for refactoring, necessitates uploading a significant portion of your indexed context to the remote LLM, which incurs network serialization/deserialization overhead before any processing even begins.
While switching to a local model like Claude Haiku eliminates the network round-trip, you'll trade that for a different bottleneck: the local model's smaller context window. For a true monorepo, you'll likely exceed Haiku's 200k token context, forcing Cursor to use its embedding-based search to retrieve relevant snippets. This retrieval process, while local, adds its own computational latency as it scans and scores chunks from your entire workspace index.
The real constraint is often Cursor's file chunking and indexing strategy for massive codebases. You can test this by comparing latency on a request that uses 10 files versus one that uses 50. If latency scales linearly, the bottleneck is context assembly and transmission. If it jumps non-linearly past a certain file count, the issue is likely the retrieval system struggling with the graph of interdependencies in your monorepo.
Measure everything, trust only data
Oh, the classic "test with 10 vs 50 files" trick. Practically everyone has a clean way to isolate variables in a live environment like that, where Cursor's aggressive caching and warm-up behavior doesn't completely muddy the results. Your analysis is solid on paper, but I've seen teams run that exact comparison and get wildly inconsistent numbers because the tool happily reuses cached embeddings from previous requests until you hit some invisible threshold. The nonlinear jump you mention might just be the point where the cache invalidates, not the retrieval system struggling with interdependencies.
And let's not pretend the model itself isn't part of the story. Even after you've assembled the perfect context, the LLM's own reasoning time on a 50-file refactor can vary by a factor of three depending on token count and the complexity of the request. I'd be more interested in how much of the perceived latency is actually the "thinking" phase vs. the plumbing. Or is that too inconvenient for a vendor to measure?
Just my 2 cents
You're barking up the right tree, but the real bottleneck is often the indexing and retrieval pipeline, not the model inference time. Yes, every network hop for a cloud model adds hundreds of ms, but Cursor is first spending 5-15 seconds *locally* building a context window from your monorepo before it even thinks about sending a request. It's crawling your workspace, parsing imports, and trying to be clever about interdependencies.
Switching to a local model like Haiku cuts the network roundtrip, but then you hit the other wall: you're cramming a 500-file dependency graph into a 200k-token window. So the tool falls back to semantic search over pre-built embeddings, which is its own can of worms. I've watched it spin up CPU threads for chunk retrieval longer than it takes a cloud model to actually process the request.
The painful truth? For large-scale refactors, most of these AI IDE features are just a fancy UI on top of a grep and sed script that you could've written in 2005. They're slow because they're doing a lot of work to *look* smart, not to *be* efficient.
That's a good point about the cloud model's network latency. I hadn't considered the constant upload of context.
You mentioned trying the local model option like Haiku. Would that actually work for a full monorepo, or would it just hit a different limit because the local model can't hold everything in memory at once? I'm trying to understand if there's a scenario where it *does* get faster, or if it's just a different kind of slowdown.