So you're running a local model for "privacy." Let's assume you've already accepted the 30% performance hit and the joy of managing your own inference stack. Before you pat yourself on the back, let's audit what this setup actually gets you.
The default Continue configuration is a one-liner for their cloud. Making it talk to a local Ollama or LM Studio instance is trivial, but the privacy guarantees are only as good as your entire pipeline. Did you remember to disable telemetry in *every* component? I didn't think so.
Here's a bare-bones `config.json` that should work, assuming your local model is actually serving the correct API format. Most tutorials skip the failure modes.
```json
{
"models": [
{
"title": "Local Mistral",
"provider": "openai",
"model": "mistral",
"apiBase": "http://localhost:11434/v1",
"apiKey": "ollama"
}
]
}
```
Key points everyone glosses over:
* `apiBase` must match your local server's exact endpoint. Ollama's `/v1` is crucial.
* That `apiKey` is a placeholder. If your local server isn't bound to `localhost:11434`, you're exposing an unauthenticated API to your network.
* You now own the monitoring, logging, and cost of running this model. What's your plan for tracking its hallucinations in production code?
And let's be clear: this isolates your code from Continue's servers, but does nothing for the security of the model weights you downloaded from somewhere. Or the dependencies in your local inference server. Have you reviewed those?
I'd ask for the postmortem when your local instance OOMs during a large refactor, but I suppose you'll just restart it and lose the context.
- Nina
- Nina
Good point about the unauthenticated API exposure. It's easy to forget that `localhost:11434` becomes a network endpoint if you're running Ollama on a different box or have it bound to `0.0.0.0`.
You mentioned monitoring and logging - that's the real cost. You're now the SRE for this model. If it goes down at 3am while you're debugging a prod incident, you're debugging two incidents. At least set up a basic Prometheus scrape for your Ollama instance and a dashboard to track request rate and latency. A single `OllamaRequestDurationSeconds` spike can save you an hour of head-scratching.
Sleep is for the weak
You're right about the telemetry gap. Even if you nail the config, Continue's own plugin can still phone home depending on your IDE setup. That's where most of these "private" setups leak.
The bigger issue is assuming local endpoints are safe. That unauthenticated API becomes a problem the second you VPN into your work network or forward ports. You're basically trusting your firewall as a security control, which is a bad plan for anything with real IP.
Exactly this. That "privacy guarantee" is a chain, and the weakest link is usually the tooling we assume is safe.
You're spot on about the telemetry blind spot. I'd add that people often forget to check their IDE extensions themselves. Even with a perfect local endpoint, the Continue VSCode or JetBrains plugin might have its own analytics layer that's independent of the model config. It's worth a quick scan of its settings.
The config snippet is a good start, but like you said, it's just the beginning. If you're going through all this trouble for privacy, you need to document your own stack's data flow.
Raise the signal, lower the noise.