Silent failures in batch jobs are a major red flag. What's your vendor's SLA on error visibility?
Before you dig into logs, answer these:
* What's your ingestion source? S3, API stream, local file?
* Are you using their SDK or direct API calls?
* Did you get any job ID or status endpoint to poll?
* Have you checked their incident history page for known outages?
Without proper error propagation, this points to weak observability in their platform. I'd demand:
* Full request/response logs for the failed job (they should provide this).
* Their error code documentation.
* Details on their batch processing guarantees (at-least-once, exactly-once).
If they can't provide clear debugging paths, reconsider their suitability for any production workload.
Trust but verify
Totally agree on the weak observability point. Been there on a 3 AM shift, staring at a "success" status while our data pipeline was secretly on fire.
One extra demand I'd add: get their *real-time* notification hooks. Waiting to poll a status endpoint is like watching paint dry while the building floods. If they can't push a webhook or event on failure, their platform is stuck in 2015.
And honestly, "reconsider their suitability" is the polite version. I'd say start drafting the migration runbook now.
NightOps
Couldn't agree more about the observability red flag. Everyone wants to sell you a "managed service" until you need to see what's actually happening inside it, then suddenly you're sifting through black box metrics.
> Full request/response logs for the failed job
This is the litmus test. I've had vendors point me to a CloudWatch group with 3 aggregated log lines and call it a day. If they can't show you the raw HTTP call and the error response their own API returned, they've built a system that's hostile to debugging.
The batch processing guarantees question is especially critical. Half these services offer "exactly-once" in their marketing, but it really means "we'll retry your failed chunk five times and then give up silently." You find out a month later when your aggregates are off.
keep it simple