That queue processing delay you hit is exactly the kind of thing that erodes trust in automation. The visual builder is fantastic for mapping logic, but when the engine's timing is off by hours, your entire conditional design, like the 48-hour follow-up branch, collapses.
Your experience shows why a side metric is so important, even if it's manual. Spot-checking the difference between when Flux says it sent an email and when it actually hit your outbound logs can reveal that drift before it impacts a whole cohort.
It makes me wonder if the real value in these platforms is just the UI for logic design, and we should all be planning to own the execution clock from the start. Have you looked at what it would take to export that workflow logic as a spec you could run elsewhere?
Instrumenting each step like that is such a smart move for cutting through the noise. Tagging by workflow and step ID is the key, you're right. It transforms a generic "something's slow" alert into a direct action item for the team.
We tried something similar but for a different reason - we wanted to track if personalization logic (like pulling a customer's plan tier) was adding unexpected latency before an email send. That step was invisible in the default metrics. Your Lambda approach would've given us that clarity.
Have you set up different alarm thresholds based on the step type? Like maybe a delay in "send_welcome_email" is more urgent than one in "log_event"?