Hey everyone! I've been deep in the weeds lately trying to build a feature that uses multiple LLM providers (OpenAI, Anthropic, Google) interchangeably for a structured data extraction task. The dream is to send a prompt and get back perfect, parseable JSON every single time, regardless of which model API I'm calling.
I'm hitting a consistency wall, though. What works flawlessly with GPT-4 sometimes gets a weirdly formatted or narrative response from Claude, and vice-versa. I know each provider has its own "system prompt" conventions and strengths, but I'm trying to find a universal prompt structure that acts as a reliable baseline.
Here’s my current approach, which is *mostly* effective but still not 100%:
* I always use a **system prompt** (or its equivalent) to set the role: "You are a precise data extraction engine. Your output must be valid JSON only, with no additional commentary."
* In the **user prompt**, I specify the JSON schema in detail, often using a placeholder example.
* I explicitly use phrases like "Output a JSON object with the following keys: ..."
* I've tried using markdown code fences (```json) in the prompt, but some models then include the fences in their output, which breaks parsing.
My specific questions for this brilliant community:
* What's your go-to prompt template for JSON output that works across Claude 3, GPT-4, and Gemini?
* How do you handle models that are overly "helpful" and add explanatory text before or after the JSON?
* Is it better to demand a "compact JSON with no whitespace" to reduce chances of extra text, or does pretty-printed JSON with clear structure give the model better guidance?
* Have you found that including an example (like one-shot prompting) significantly increases consistency versus just describing the schema?
I'd love to hear your war stories, successful patterns, and even the weird edge cases you've encountered. Share your snippets if you have them!
keep building
keep building
1. James K., infrastructure lead at a mid-market logistics firm (450 headcount). We run a multi-provider LLM gateway in production for parsing shipment documents, calling about 18k prompts daily across OpenAI, Anthropic, and Azure models.
2.
- **Structured Output Modes Are Not Universal**: OpenAI's `response_format={ "type": "json_object" }`, Anthropic's native XML tool calling, and Google's `response_mime_type="application/json"` are vendor-specific. If you rely solely on one, your portability breaks. Without using these, even with strong prompting, I've seen a 3-5% malformed JSON rate in production.
- **Schema Definition In-Prompt Is Inefficient**: Repeating a full JSON schema in every user prompt increases token cost and latency. For a 15-field schema, this added ~1200 input tokens per call. At our volume, that's an extra $180-250/month just for schema repetition, not compute.
- **Temperature and Top-P Matter More Than Wording**: Setting `temperature=0` and `top_p=0.95` (not 1.0) had a greater impact on JSON consistency than any prompt phrasing. A model with temperature 0.7 will often invent fields or structures, regardless of your instructions.
- **Fallback Parsing Layer Is Non-Negotiable**: You will get malformed JSON. In our system, every response passes through a two-stage cleaner: first a regex extractor for anything between `{` and `}`, then a lenient parser (like `json.loads(text, strict=False)`). This catches about 2% of responses that would otherwise fail.
3. My pick is to implement a prompt wrapper that injects a one-shot example in a `` tag before the user query, combined with a strict temperature setting, and never assume the model's output will be clean. For a clean choice between providers, tell us your average token count per call and whether you can afford the latency of a single retry loop.
James K.