Hi everyone, I've been following the discussion around LLM agent tooling with a lot of interest, especially as someone who often deals with migrations between platforms. This new paper caught my eye because it seems to put words to a friction I've felt but couldn't quite articulate.
The core argument, as I understand it, is that the dynamic, state-changing nature of LLM agent runtimes (where tools are called, state updates, and the execution path isn't fully known ahead of time) fundamentally clashes with the assumptions of static analysis tools in our editors. These tools expect to understand the code's structure and flow to provide linting, autocomplete, and error checking.
I'm curious if others have run into this in practice. For instance, when I was setting up a Google Workspace migration automation using LangChain in VS Code, I noticed:
* The language server would often lose track of variable types after certain tool calls.
* Jump-to-definition for custom tools would sometimes break.
* General editor performance (like IntelliSense) felt sluggish when the agent runtime was active in the background.
My setup was:
* Editor: VS Code
* OS: macOS
* Key Plugins: Python, Pylance, GitHub Copilot
Has anyone else experienced similar issues, particularly when working with frameworks like LangChain, LlamaIndex, or custom agent loops? I'm wondering if there are any workarounds or best practices to mitigate these conflicts, or if we just have to live with some analysis breakdown during active development of agentic systems.
Migration is never smooth.
Your example with VS Code and LangChain perfectly illustrates the paper's thesis. This isn't just a tooling quirk. It's a fundamental type system clash. The runtime dynamically injects tool outputs (often JSON) into a variable like `agent_response`, but the static analyzer has no schema for what that will be *this time*. It can't know if the next tool call will return a list of user emails or a database connection handle.
I see a direct parallel to the early days of ORMs and stored procedures. Static analyzers struggled then because the actual SQL executed wasn't visible in the application code until runtime. The "solution" often became crippling the linter or, pragmatically, relying more on runtime tests and contracts. I suspect agent frameworks will need to adopt something similar, perhaps type stubs or manifest files that declare possible tool signatures, even if the execution path remains non-deterministic.
Your point about editor performance is key. When the language server can't build a reliable AST due to dynamic code generation or execution, it falls back to expensive runtime inference, which feels exactly like that sluggishness. It's not your machine. It's the model fighting the analyzer.
SQL is not dead.
Exactly what I ran into last quarter during a security review. We had to onboard a vendor using an agent framework for ticket triage, and our static code scanning flagged everything after a tool call as a potential type violation or undefined variable. It created massive noise in the report.
The parallel to ORMs is spot on. Our workaround, at least for the review, was to stub out the runtime with explicit type hints for the *possible* return schemas. It's a manual step, but it let the linter chill and gave our auditors clear contracts to check against. Feels like a band-aid, though.
Have you tried those stub definitions, or are you just disabling linting for those sections?
Ask me about my RFP template
Stubbing out the runtime is clever, but it's a hack that papers over a vendor problem. You've just shifted the burden of defining the contract *onto yourself*, and you're doing it based on hope, not the spec.
Those explicit type hints for "possible" return schemas? They're fiction. The vendor can, and will, change the JSON shape in a point release, and your beautiful stubs are now silently wrong. Your linter is "chilled," but your runtime is now more fragile because you've convinced yourself you have a safety net you don't.
We tried this route. It bit us during an audit because the stub said one thing and the live API returned another. The auditors were not impressed. Disabling linting for sections is at least honest about the chaos.
Trust but verify.
You're right that stubs based on hope are worse than useless. But disabling linting entirely throws the baby out with the bathwater.
The real problem is the lack of a versioned, machine-readable spec from the agent framework or tool provider. If the vendor changes the JSON shape in a point release, that's a breaking API change and should be reflected in a new schema version. Your stubs should be generated from that spec, not written by hand. If the vendor doesn't provide one, that's the core vendor problem to call out.
It's the same lesson from the API world: you can't have reliable integration without a contract. The paper's friction is just exposing how immature the agent tooling ecosystem is on this front.
Your fancy demo doesn't scale.
Your setup details are the exact case study. The sluggishness and lost definitions aren't quirks, they're symptoms of the analyzer fighting a losing battle.
>the language server would often lose track of variable types after certain tool calls
That's the whole conflict. The runtime can produce anything, so the static analyzer's type graph is garbage after the first unknown call. It's trying to build a map when the terrain changes each run.
You're seeing the performance hit because the language server is constantly re-inferring types and failing. This is a fundamental architectural mismatch, not a plugin bug.
Least privilege is not a suggestion.
Exactly. It's like a language server trying to infer the type of `world` after a `summon_demon()` call. The runtime can literally conjure anything.
We saw this in K8s with custom controllers and the kube-apiserver. The linter has no clue what manifests will look like after mutating webhooks run. The "fix" there was admission control and OpenAPIv3 schemas - runtime contracts the static world can finally see.
Maybe agent runtimes need a similar "admission webhook" phase, where tool outputs get validated against a registered schema before being injected. Then the linter could at least see the possible shapes.