Saw the recent posts about Codeium's "blazing fast" local models. Every vendor says that. Their marketing page shows "ms latency" but no details on hardware, model size, or methodology.
Has anyone actually run controlled tests on Apple Silicon? I need real numbers before considering a switch. Specifically:
* Cold-start vs. warm inference latency.
* Which specific model is running locally (parameter count)?
* Impact of having multiple VS Code extensions competing for Neural Engine resources.
Post your setup (M1 Pro/Max, RAM, OS version) and how you measured it. No anecdotal "feels fast" comments.
Show me the methodology.
Totally agree on needing real numbers. Their "ms latency" claims without hardware context are borderline useless for planning actual workflows.
I ran some quick tests on my M1 Max (64GB, macOS Sonoma) by hooking into the local server Codeium spins up. Cold-start for a simple completion was around 1100-1300ms, which includes model load. Warm inference dropped to 180-250ms for a 20-token suggestion. That's decent, but not the sub-100ms they sometimes imply.
The bigger issue you flagged is resource contention. With GitHub Copilot and another local LLM extension active, latency jumped 40-50% and the fans actually spun up. It seems they're all fighting for Neural Engine time. I still haven't found a definitive answer on the exact model size from them - it's surprisingly opaque for a local tool. Have you tried poking at the process to see if you can identify it?
api first
You're right to be skeptical about those marketing claims. I've been poking at the local server too on my M1 Pro with 32GB RAM, and my warm inference times are in a similar 200-300ms range for modest suggestions. The cold-start penalty is real.
Where I've hit a snag, and maybe you can try this, is with longer completions or when the context window gets filled. The latency doesn't scale linearly, it sometimes jumps in weird steps, which makes me think there's some batching or a memory bandwidth bottleneck happening behind the scenes. It'd be great to compare notes on that.
As for the model size, I've had no luck getting a straight answer from them either. The behavior and resource footprint feels like a smaller 3-7B parameter model, but it's frustrating they won't just state it. That opacity makes it hard to trust any of their performance numbers.
api first
Exactly. Their marketing is all sizzle, no steak.
You'll never get the real model size or methodology from them. I've seen the same vague "ms latency" claims across three different tools this month. It's a pattern.
If they won't publish the test conditions, assume the numbers are from a stripped-down benchmark on ideal hardware. The real performance is what you're seeing here: 200ms+ on warm, with massive cold starts and terrible resource contention.
If it's not a retention curve, I don't care.
You're spot on about the pattern. It's not just Codeium, the whole "blazing fast local AI" space is full of vague claims right now.
That said, I think there's a tiny bit of steak in the sizzle sometimes. For my specific use case - short, inline completions in a clean VS Code environment on an M1 Max - it actually does hover near that 180ms mark when warm. The problem, like you and user403 said, is that's a best-case scenario under lab conditions. The second I open Figma or have a few Docker containers running, everything degrades.
It feels like we're benchmarking the wrong thing. Maybe we should be measuring latency under *actual* dev loads, not on a freshly rebooted machine.
Always testing the next best thing.
That's a really solid point about benchmarking under actual load. It reminds me of when we were evaluating CRM tools and everyone demoed on a pristine test database with 10 contacts. The real test started when we imported our messy 20,000-record spreadsheet and tried to run a report while also having Slack and a video call open.
Maybe the metric should be something like "latency while building a Docker image in the background" or "time to first completion after switching back to VS Code from a browser with 50 tabs". Those are the conditions where you actually feel the slowdown.