That's a solid breakdown of the initial experience. The "okay" initial suggestions followed by cross-service confusion mirrors what I saw in a recent evaluation at my place. We had it generate what looked like a valid Docker Compose snippet for a single service, but it pulled environment variable names from a legacy, similar-named service in a `/prototypes` folder.
It feels like it's working from a weighted bag of all tokens across the entire indexed scope, and your prompt's keywords just pull from the top of that bag, regardless of actual file relationships. So it sees `Dockerfile` and `serviceName` as separate tokens to match, not as a single unit belonging to a specific node in a graph.
Your point about it not consuming turbo.json is key - if it's not reading those, then it fundamentally cannot understand the topology, only guess at proximity. Has anyone tried explicitly pasting the relevant section of their nx.json or turbo.json into a prompt as context? I'm wondering if that brute-force method changes the output quality at all, or if it's still just blending.
Yeah, that blending from a `/prototypes` folder is a perfect example of the weighted bag analogy. It's matching tokens, not intent.
I have tried pasting the relevant turbo.json block into a prompt, actually. It's a mixed bag. Sometimes it seems to latch onto the task definitions for a second, but then it'll still suggest a command that uses the wrong package manager because it saw `yarn` in another part of the repo. You're just adding more tokens to the pile, and there's no guarantee which ones win.
It feels like we're paying for a search engine when what we need is a compiler that understands dependencies.
Beta tester at heart
Pasting the turbo.json block is basically feeding a small, local map into a GPS that's still using an outdated world atlas. The model might read it, but it can't reconcile that map against its broader index - it just gets more tokens to potentially mis-weight.
Your package manager example nails it. I've seen it suggest `npm run build` for a service whose turbo config explicitly defines a `pnpm` runner, because the word "build" appeared more frequently with "npm" elsewhere in the repo. The explicit instruction gets drowned out by statistical noise.
It's not a compiler, it's a very expensive token blender.
APIs are not magic.
That initial "okay" feeling is exactly how they get you. It builds just enough trust for the architectural error to slip through review. Our Salesforce mess cost two days of rollback because the generated test class passed compilation but violated a namespace boundary no one caught.
You're right that it's a token match. I'd argue the Dockerfile problem is even worse with CI configs - it'll happily merge a GitHub Actions workflow from your main branch with a long-dead experimental one, because the job names are similar. The resulting YAML looks valid but points to non-existent runners.
Cloud costs are not destiny.
Yep, you've put your finger on the core issue. That "okay then confusing" pattern you describe is exactly how it presents - it feels competent on isolated files, but the monorepo structure breaks the illusion.
Your question about build tool context is key. In my tests, it doesn't ingest turbo.json or nx.json as a graph source. It treats them as just another text file for token matching. So it might see the word "dependsOn" but can't actually *use* the dependency graph to inform suggestions.
The Dockerfile problem gets even trickier with shared base images or multi-stage builds. It'll often suggest a `COPY` command from a sibling service's directory because the path tokens look similar, completely missing the workspace boundaries. Have you tried explicitly naming the root directory in your prompts? Sometimes that helps, but it's not reliable.
Stay factual, stay helpful.