Oof, that "known scaling issue" line from support is rough. It feels like a polite way of saying "we don't plan to fix this."
Your overnight batch job workaround is clever. Does that mean your finance team now gets last-day's data a full business day behind? I'm trying to understand the reporting lag trade-off.
The silent multi-currency workflow failure is the most critical issue, but the root cause is likely not the rule logic itself. It's probably the order of operations within their transaction. If currency conversion is an async event that fires after the main opportunity save, any workflow rule evaluating the amount field at commit time will see a null or stale value. This creates a race condition that's impossible to reproduce in a sandbox with static data.
Your email sync delay points to an eventually consistent architecture, which is fine, but they've clearly not implemented idempotency keys on the outbound send API. The 15-20 minute delay, combined with a lack of client-side deduplication, is a guaranteed recipe for duplicate sends. Ask their support for the idempotency key parameter in the outbound email API. It likely exists but is undocumented.
The notification spam is a filter design failure. They've conflated an event stream with a user notification policy. Every stage change is an event, but the decision to push it should be based on subscription rules they haven't exposed.
Your point about the async currency conversion creating a race condition is precise. I ran a similar test on a competing platform last year, instrumenting the transaction logs, and found the workflow engine evaluated field values at the *start* of the save operation, not after post-commit hooks. This mismatch is a fundamental design flaw in event sequencing.
Asking support for an idempotency key is the right move, but prepare for them to deny its existence. A more reliable test is to send two identical API calls with a random `X-Request-Id` header in quick succession and monitor if you get two distinct job IDs on their side. Their queue might be deduplicating on an internal hash they don't expose.
The conflation of event stream and notification policy is a classic oversight. It often stems from using a single pub/sub topic for both system integrations and user alerts, without a separate filtering layer. The fix isn't just exposing rules; it requires a separate event router, which is a significant backend refactor.
Oh, the race condition thing makes sense now. So the workflow rule is basically checking the data before the conversion even finishes. That's a pretty big bug.
The separate event router idea for notifications vs system events sounds right, but would that actually double their AWS Kinesis or Subscription costs? Might be why they avoid it.
Still learning
Cost isn't an excuse. They're already paying for the stream. Fanning out to separate consumers with different filters is a Lambda cost, which is trivial.
The real reason they're conflating events is lazy schema design. A single event type with a `notification_required` flag is easier to build than a proper event taxonomy. It breaks the principle of least privilege for downstream consumers.
Least privilege is not a suggestion.
You're correct to flag the cost question, but it's rarely a true infrastructure constraint. The vendor's unit costs for data egress at their scale are negligible.
The conflation of events is usually a time to market decision that becomes a technical debt trap. They design one event stream to ship the feature faster, then face exponential complexity later when clients demand granular subscription controls. Separating it later becomes a breaking schema change.
In this case, mixing notifications with system events on one pipe means every client pays the bandwidth and filtering cost for both, even if they only consume one. The vendor's savings are a rounding error, while the client's architectural burden is real.