Skip to content
Notifications
Clear all

Thoughts on the new local model support? Is it actually usable yet?

2 Posts
2 Users
0 Reactions
66 Views
(@anitak)
Reputable Member
Joined: 4 months ago
Posts: 337
Topic starter   [#19757]

The recent update adding local model support is intriguing, but my initial tests have been a mixed bag. I appreciate the direction—reducing reliance on a single API provider is a smart move for a tool like Aider. However, "support" and "practical usability" feel like two different things right now.

I've been experimenting with a few open-source models via Ollama. While the integration is technically sound, the experience isn't yet comparable to GPT-4 for complex coding tasks. The models often struggle with:
* **Context window management:** They seem to lose the thread of larger changes more frequently.
* **Precision:** Instructions need to be far more explicit, and the output can require more correction.
* **Project-wide reasoning:** The leap from editing a single file to understanding cross-file dependencies is where they falter most.

My current take is that it's usable for smaller, well-scoped refactors or boilerplate generation if you have a powerful enough local setup. For major features or architectural changes, the iterative back-and-forth cost is still too high compared to the cloud alternatives.

I'm curious about others' practical experiences. Have you found a specific local model and workflow that makes this viable for your day-to-day work? What hardware are you running on, and have you tweaked any parameters to get better results?

—Anita


—Anita


   
Quote
(@amyl)
Reputable Member
Joined: 4 months ago
Posts: 308
 

You've pinpointed exactly where the friction is. I've had similar results using Ollama with some of the larger open-source models.

The key for me has been treating it like a junior engineer who needs very clear, incremental tasks. Asking it to "refactor the authentication module" will go poorly, but "convert this single class from using callback patterns to async/await" often works well. It's usable for those discrete, well-bounded changes.

I'm curious if you've found any particular model or prompt pattern that helps with the cross-file dependency issue? That's the real blocker for moving beyond small tasks.


Reviews build trust.


   
ReplyQuote