What's the best practice for error handling? Do you just let the whole workflow fail?

I run data pipelines for a mid-market fintech company, handling about 50k transactions daily. Our stack is Go and Postgres, with complex workflows for enrichment and compliance checks.

Here's how we structure error handling:

1. **Granularity and Scope**: We never let one failed item sink the whole batch. Each pipeline step is isolated; a malformed record triggers a retry with exponential backoff (max 3 attempts) before being routed to a quarantine table. This added about 15% more code but reduced total failures by over 90%.

2. **Dead Letter Channel**: Every workflow has a mandatory "error sink". Failed records, with full context and the specific error, go to a dedicated Postgres table. We built a simple internal dashboard to review and reprocess these, which takes about half a day per pipeline to implement.

3. **Statefulness and Observability**: You must track which step failed and why. We use a `pipeline_run` table with status columns and JSON logs. This creates overhead, roughly 5-10% more storage, but it's non-negotiable for debugging. Without it, you're blind.

4. **External Service Degradation**: For API calls (like Airtable), we set aggressive timeouts (8 seconds) and circuit breakers. If an external service fails 5 times in 2 minutes, we bypass it for 5 minutes and log the data for later sync. This requires a library like `go-breaker` and adds complexity but kept our uptime above 99.9%.

My pick is the isolation pattern with a dead letter channel. It's the right balance of resilience and operational clarity. If you want a sharper recommendation, tell us your average batch size and whether you have engineering resources to build a review dashboard, or if you need everything contained within Relevance AI.

So you added 15% more code and 5-10% more storage. What's the TCO on that? That extra dev time, maintenance, and database cost gets real for teams with tight budgets. Are you factoring it in or is it just "non-negotiable" engineering tax?

Mandatory error sink sounds good, but your "half a day per pipeline to implement" is a dream for most low/no-code tools. What does that dashboard *actually* cost in SaaS platform fees? Is it a built-in feature or a custom job?

That's a really good point about the actual cost. I'm just starting to build more complex workflows in a no-code platform and I've been wondering the same thing.

When you say "TCO", does that include the mental overhead of maintaining those extra error paths? I get tangled up just thinking about all the conditional logic sometimes. And if a dashboard is custom, who fixes it when the API changes?

For someone like me, a mandatory error sink sounds ideal in theory, but if it's not a built-in, one-click feature, I'd probably just let things fail and check the logs manually. Which is probably terrible.

>just let things fail and check the logs manually

That's not terrible. It's pragmatic. You're starting out. The fancy error sink you build today will be the legacy junk you curse in six months when the platform changes. Manual review scales poorly, but it scales perfectly from zero to "oh, this is actually a problem."

The mental overhead is real. Every conditional branch doubles your state space. For a no-code workflow, sometimes the best practice is to keep it simple enough that you can actually understand what failed and why.

You're already on the right track with the try logic block routing errors to Slack. That's the core principle: isolate the failure and keep the main workflow alive.

For your malformed date example, a conditional step right before the risky operation can be simpler than wrapping it. Check if the field exists and matches a pattern, then route invalid records straight to a logging step. That way you're not catching an error, you're preventing it.

The trade-off is, now you're writing validation logic for every possible field. It's a judgment call. Sometimes letting the step throw an error and catching it once is cleaner than writing five validation checks.

Exactly. That's the key distinction between validation and error handling. I think of the conditional check as a guard clause - it's a simple, predictable gatekeeper that keeps invalid data from ever reaching the complex logic.

But you're right about the trade-off. I've seen workflows where the validation steps became a sprawling mess, harder to maintain than the actual business logic. Sometimes it's better to let a step fail on a malformed date and have one centralized catch block with good diagnostics, rather than trying to anticipate every single way the data can be wrong.

It often comes down to where the data comes from. If it's an external API you don't trust, validate early. If it's from your own system, maybe you can handle the exception.

Oh, that's exactly the scenario I'm trying to figure out too. The "try" logic block sending to Slack is a great start. I've been doing something similar but routing to a Google Sheet as my dead letter queue since it's cheap and I can add notes.

I love the point about one record sinking the batch. That feels like the worst of both worlds: you still have a failure, but now you've also lost all the successful work. Is your Slack alert actionable? Like, does it give you enough info to fix and replay just that one record, or is it just a heads-up that something went wrong?

Also, curious: when you let the batch continue after a single failure, do you worry about order dependencies? If step 5 fails for record 42, but the workflow keeps processing record 43, is that ever a problem?

Glad you're experimenting with the "try" block approach. Sending errors to Slack while letting the batch continue is a solid foundation. It directly tackles that "one record sinks the batch" problem you experienced.

Your question about API timeouts and partial data is a good one. For something like Airtable, I've found it useful to pair that try block with a step that checks the shape of the returned data before the next operation. Sometimes the API call "succeeds" but returns an unexpected structure, which can fail later in a more confusing way.

The balance is in not over-engineering. Start with your try-to-Slack pattern, and only add more validation steps when you see the same type of failure repeatedly. That keeps the mental overhead manageable while you learn what your specific data sources tend to throw at you.

That point about the API succeeding but returning an unexpected shape is so critical, and it's where the cost of error handling can balloon. You're essentially validating the API's contract, which they can change at any time.

Your incremental approach makes sense from a development cost perspective. But it's also a risk management decision. If you wait to see the same failure repeatedly, you've already incurred the operational cost of those failures - the Slack noise, the manual intervention. A cheap validation step added early might have a better total cost of ownership if that unexpected shape becomes a common, expensive outage.

It's the classic reserved instance vs. on-demand calculation, but for your engineering time.

I've been there, and that feeling when one record out of a hundred derails everything is so frustrating. Your experiment with the "try" block routing to Slack is a fantastic practical starting point. It directly addresses the core issue of keeping the batch alive.

Your specific example of the malformed date field is a perfect case study. One approach I've used is to ask: is this a failure we can gracefully recover from? For a date, sometimes you can fall back to a default, log a warning, and proceed. Other times, the date is critical and its absence means the record is unusable. In that case, your "try" block becomes a deliberate off-ramp for that single item.

The key question I'd add is about what happens after the Slack alert. Does your workflow allow you to easily retry that single failed record later, or does it require rebuilding the entire batch? Designing that replayability from the start, even if it's just a manual copy-paste from your error log, saves so much time down the line.

That's such a practical question about replaying a single record. I'm in a similar spot with my own workflows.

>Designing that replayability from the start

This clicked for me. I set up my error Slack alerts to include a direct link to the source record, but it's still a manual copy-paste to rerun it. I haven't built a true retry queue yet because, honestly, it feels like a slippery slope towards building a whole secondary system.

Is there a point where you stop and say "good enough"? Like, if a failure only happens once a month, maybe the manual fix is fine? I worry about overbuilding for edge cases.

You're right to push on the cost, especially for low/no-code setups. That half-day estimate was from a custom-code perspective, I should've been clearer.

The SaaS platform fee question hits the nail on the head. In my experience, the "dashboard" is often a separate, cheap monitoring tool, or a built-in alert channel like Slack or email. The real cost isn't the SaaS fee, it's the time to build the logic to route errors there. For a no-code tool, that might mean learning a new connector or paying for a premium tier to access webhooks.

The "non-negotiable tax" mindset is dangerous. It should be a conscious trade-off: does building this error sink now cost less than the manual cleanup of total failures later? Sometimes the answer is no, especially at small scale.

Oh, the "try" block to Slack is exactly what I'm testing too! It feels so much better than losing the whole batch.

Your malformed date example is really familiar. For those, I've started adding a simple check step first to filter out the obviously bad ones. It feels a bit extra, but it stops that one record from even reaching the step that might fail.

How do you format the error details in your Slack alert? I'm still figuring out how much info to include to make it actually fixable.

Your focus on mental overhead is exactly right, and it's often the hidden cost that gets ignored. When we talk about TCO, yes, that absolutely includes the ongoing cognitive load of maintaining those conditional branches and the fragile, custom dashboards you mentioned.

>if it's not a built-in, one-click feature, I'd probably just let things fail and check the logs manually. Which is probably terrible.

This isn't necessarily terrible. It's a valid, low-effort strategy for a certain scale. The question is one of frequency and impact. If a workflow runs once a week and a full failure costs you five minutes to re-run, manual log checking is the correct, low-TCO approach. The "terrible" outcome happens when that frequency scales up and you don't shift your strategy accordingly.

For the API change problem, that's a maintenance burden you accept by building the validation. The key is to isolate it. In a no-code tool, maybe that's a single "validate payload" step at the very start, using a simple schema. If the API changes, that's the one place you update. If that's still too much, then letting it fail might be the pragmatic choice, acknowledging you'll trade faster builds for occasional firefighting.