I'll echo the others on cost and instruction-following, but your point about marketing automation integrations is the key one for me. I attempted to use 1.5 Pro to parse unstructured email campaign feedback from a huge Mailchimp export and map it to structured fields in our HubSpot custom objects. The long context handled the export file flawlessly.
Where it fell apart was in the integration logic itself. The model would generate perfect JSON according to my schema, but the mapping decisions were inconsistent. It would correctly tag a piece of feedback as "feature request" based on the content, but then assign it to the wrong lifecycle stage property in the object model about 30% of the time. The errors weren't in format, but in applying the business rules that *surround* the integration.
That means any cost savings from using the long context are immediately consumed by the need to add a validation layer before the data hits the CRM. At that point, you're better off with a more deterministic, chunked approach using GPT-4, even if it's more architecturally complex. The long context is brilliant for analysis, but not yet for reliable, automated data *movement*.
Exactly the scenario that keeps me from putting it in a pipeline. You can't automate a step that requires a human to spot-check for logical inconsistencies in the output.
The silent error in your timeline is the real killer. It's not a format error you can catch with a JSON schema validator, it's a logic error that looks perfectly correct. That forces you to build a whole secondary analysis or comparison step, which defeats the purpose of using the long context for automation. I've seen the same with generating Terraform from legacy configs - the HCL is valid, but a subnet CIDR block is subtly wrong.
So you're back to square one: you need another, probably more expensive, process to verify its work. At that point, why not just use the slower, more reliable toolchain from the start?
Build once, deploy everywhere
Your skepticism is spot on. I gave it a serious try for a production use case - generating personalized email campaign variants by analyzing entire customer journey histories from our CDP at once. The long context was amazing for that, it could see the whole story.
But like everyone else found, the output quality for complex tasks isn't reliable enough. It would follow my creative brief template perfectly, but then swap a key product benefit mentioned in the first interaction for one from a later stage. The email copy would be fluent and on-brand, but the core message would be subtly misaligned with the customer's actual path.
So for now, it's back to chunked analysis with GPT-4 for anything that goes to our email platform. The speed and cost promise falls apart when you have to fact-check every output.