Okay, I need to vent a bit and see if I'm the only one hitting this wall. 😅
I've been building a workflow that uses two different SaaS tools via their APIs, both with OpenAI-compatible tool-calling endpoints. The idea was elegant: one LLM call could decide to fetch data from Tool A *or* Tool B based on the user's request. But in practice? It's been a nightmare of silent failures.
The issue isn't the concept—it's the implementation. If Tool A's schema says a parameter is `"type": "string"` and Tool B's similar parameter is `"type": "string"` but with an extra `"enum": ["x", "y"]` constraint, the LLM sometimes gets confused. Worse, some providers are strict about the JSON schema `"required"` array matching *exactly*. One extra or missing field in the tool definition you pass to the API? Instead of a helpful error, you just get a generic "no tool calls" in the response.
My recipe for survival so far:
- **Normalize schemas manually:** I now pre-process all tool schemas from different sources into a single, consistent format before feeding them to the LLM. This means stripping out redundant `enum` values, flattening nested `anyOf` structures, and standardizing `required` arrays.
- **Validation layer:** I added a lightweight step that validates the tool-call object *before* I try to execute it against the actual third-party API. This catches mismatches early.
- **Fallback to sequential calls:** For critical flows, I've abandoned parallel tool-calling. It's less efficient, but more reliable: decide which tool is needed in one LLM call, then execute that specific tool call in a second step.
Has anyone else built a "tool-calling proxy" or adapter layer to handle this? I'm curious how others are ensuring reliability when integrating multiple external tools with slightly different schema interpretations. Are we all just writing a lot of brittle glue code, or is there a cleaner pattern emerging?
APIs > promises