I've been using Codeium heavily for the past two months, mostly for inline completions in VS Code. The latency there is fantastic—usually under 100ms for a suggestion to appear. It feels almost instantaneous.
Recently, I started using the standalone chat interface more for longer-form refactoring and explanations. I've noticed a consistent, perceptible delay. We're talking 1.5 to 2.5 seconds for a first token to start streaming in, sometimes more. It's not terrible, but it's enough to break flow when you're iterating quickly.
Has anyone else benchmarked this or run similar comparisons? I'm curious if it's:
* A universal architectural difference (maybe chat models are inherently slower to start?)
* A routing/CDN issue (are chat requests hitting a different, more congested endpoint?)
* Just my local configuration or region
My setup:
* VS Code extension for completions
* Browser-based chat interface (Chrome)
* Same project context for both
The inline completions are so snappy that the chat feels sluggish by contrast. I'd love to see some numbers from others.
ms matters
Yeah, you're definitely not alone. I see the same latency split between inline and chat, especially when the cluster's warming up a new model instance.
I suspect it's partly architectural - chat often uses larger context windows and has to spin up more resources per session. Inline completions are probably optimized for ultra-low-latency, like a pre-warmed pool of lighter models. Could also be session initialization overhead if you're hitting a different service mesh route.
Have you tried the chat from VS Code directly vs. the browser? I sometimes get better times there, might be fewer hops.
#k8s
That's a great point about trying it directly in VS Code vs the browser. I hadn't considered the routing hops. I wonder if there's also a difference in how the initial context is packaged. With inline completions, the 'context' is just the immediate code around your cursor, but a chat session might be bundling the whole conversation history, project structure, or even some system prompts from your profile before it even starts generating. That pre-flight processing could account for the extra second or two.