I've been integrating the Luma Dream Machine API into a workflow automation pipeline for the last few months, primarily for generating standardized architecture diagrams and compliance report summaries. The consistency of its output was a key factor in its selection. Following the latest platform update (I believe it was version 1.4.3), a critical prompt pattern I rely on has become fundamentally unreliable.
The prompt was designed to output a specific, structured JSON block that my Terraform automation could parse. The structure was non-negotiable. Here is the exact prompt template that previously worked with 100% consistency:
```json
{
"instruction": "Analyze the following AWS service list and output a risk assessment. List must be categorized by the following criteria: 'public_ingress', 'data_store', 'compute'. Return ONLY a valid JSON object with the keys matching those criteria, each containing an array of the service names that fit.",
"services": ["EC2", "S3", "RDS", "Lambda", "API Gateway", "CloudFront"]
}
```
Post-update, the model now frequently:
* Adds explanatory text before or after the JSON block, breaking automated parsing.
* Sometimes alters the key names (e.g., `public_ingress` becomes `public_access_services`).
* In several attempts, omitted one of the required top-level keys entirely, returning an incomplete object.
This isn't a minor variation in wording; it's a breakdown in structured output compliance. My workflow now requires manual intervention to correct the format, defeating the purpose of the automation.
I'm attempting to diagnose whether this is:
1. A permanent change in the model's behavior regarding instruction adherence.
2. A bug introduced in the recent update.
3. A side effect of other "improvements" that requires a complete prompt redesign.
Has anyone else conducting similar structured output or API-based workflows observed a regression in prompt fidelity? If you have found effective mitigation strategies—such as moving to a different endpoint, employing more stringent system prompts, or a specific syntactic pattern that has restored stability—your findings would be invaluable. Please share any concrete examples of what now works for you.
Adding explanatory text is a regression, full stop. That's exactly what structured output modes are meant to prevent.
Have you tried locking it down with a stricter system prompt? Something like:
```
You are a JSON generator. You do not add commentary, explanations, or markdown formatting. You output only the JSON object.
```
If the API itself changed its output guarantees, you might need to wrap the call in a parser that strips non-JSON before it hits your Terraform. Annoying but sometimes the only fix.
Ship it, but test it first
Your issue mirrors a problem we see constantly in cloud billing APIs when their response schema changes without proper versioning. The addition of explanatory text is a textbook regression that breaks deterministic parsing.
While wrapping the call in a parser is a short term fix, it adds latency and brittleness. I'd recommend a more defensive approach by explicitly requesting a JSON schema in your system prompt and using a response format like this:
```json
{"type": "json_object"}
```
you should version your prompts. Treat them like Terraform modules and include an explicit version directive (e.g., "response_format_v1") that the API can honor or reject, giving you a clear failure mode instead of silent corruption.
If the API can't guarantee schema stability, you might need to treat it as an unreliable external service and implement a validation layer that discards any response not matching your exact key structure, forcing a retry. This is costlier, but maintains your pipeline's integrity.
Every dollar counts.
You've hit on the core issue with using these models as a structured data API. The moment a provider tweaks their system prompt or output sanitization, your integration is fragile. I'd be curious if your prompt is sent as raw user text or within a structured "message" array. The latter sometimes offers more control.
While `response_format: { "type": "json_object" }` is the proper mitigation, your prompt's initial `instruction` key is problematic. It's a JSON object pretending to be a prompt, which might confuse the model's instruction parsing. Consider restructuring your entire request to separate the system role from the user data. For example:
System: "You are a risk assessment classifier. Output ONLY valid JSON."
User: "Categorize these services: EC2, S3, RDS, Lambda, API Gateway, CloudFront. Use keys: public_ingress, data_store, compute."
This separation often yields more stable JSON output as it clarifies intent. Treating the prompt itself as a JSON object is asking for parsing confusion on their end.
- Mike