Hailuo is giving us different results in staging vs. production. Same config. Maddening.

Been there, felt that pain. The "same config" trap is brutal.

Before you go down the vendor rabbit hole, check if your production deployment is embedding a different default system prompt from an env var or init script. I've seen CI inject a "be concise" directive that overrides the config file. Run a quick `env | grep HAILUO` in both environments and diff the output.

Also, what's your failover setup look like? If the primary endpoint times out in prod, you might be silently hitting a fallback model with a shorter context window. Log the response headers for a few calls and look for `x-model-variant` or similar.

Oof, that's frustrating. The "same config" assumption is a classic trap. You've probably already checked, but have you compared the actual network request sent from each environment? A config file can be identical, but the code that *reads* it and builds the API call might pull in an extra default from a different config layer in prod.

Also, what's your deployment look like? If staging uses a recent container image and prod is pinned to an older version, the Hailuo client library itself might be different. A minor version bump could change how it chunks the context or handles retries.

That library version shift is a sneaky one. We burned a week on a bug where a patch update changed the default temperature from 0.0 to 0.2. Everything felt "softer" in prod and we were chasing configs, but it was just the pinned client lib being older.

> the code that *reads* it and builds the API call might pull in an extra default from a different config layer

This is especially true if you're using a shared internal SDK that wraps the Hailuo client. That SDK might have environment-specific overrides baked in. You think you're sending the config, but the SDK is silently merging a defaults file from a production-only config map.

Oh, that's a great call. WAF limits are such a wild card because you're totally blind to them. Our staging endpoint was on a shared cluster, but production used a dedicated one with a stricter set of ingress rules.

We ended up adding a debug header to our requests, like `X-Debug-Payload-Size`, and then asking their support if they could check their side for any request trimming. Turned out the prod WAF had a slightly lower limit for the `Content-Type` we were using. The logs on our end looked perfect, but their layer was just chopping it off.

Have you looked into whether your prod traffic goes through a different API gateway or a different geographical endpoint? That routing alone can change which set of middleware your request hits.

That's a very good starting point, especially the note about secret managers. They're designed to be transparent, which can make the override completely invisible unless you're logging the final constructed request object just before it leaves your app.

I'd add one caveat to checking the `x-model-id` header: some vendors will return the *primary* model you requested there, even if the request was silently rerouted internally. We've had to ask support for audit logs that show the actual processing node to get definitive proof of a failover. The latency spike you mentioned is often the only client-side signal.

Start by isolating the variable everyone overlooks: cost allocation tags. Check if your prod calls are tagged with a different internal project or department code than staging. Vendors often use these tags to route traffic to different service tiers, even if the API endpoint is identical.

Your problem sounds like a classic silent failover to a cheaper, dumber model. The vendor's SLA probably guarantees uptime, not consistency. Log the actual response time for each review in both environments. If prod is consistently faster, that's your clue.

And never trust a config file alone. Log the final, fully-resolved payload sent to the API from each environment, including all headers. The difference is in the runtime assembly, not the source file.

That file order issue is sneaky, and you're right to flag it! It made me think of another angle: even if the *content* is identical, the *metadata* attached to each file in the upload could differ between environments.

We had a case where our staging system injected a last-modified timestamp from the git commit, but prod pulled it from the actual filesystem. The model seemed to treat those timestamps as a priority signal, subtly shifting its focus. Could be worth logging the full multipart form data payloads to compare.