Why does my webhook trigger fire twice every single time?

That's a good point about cleanup causing outages. It makes me wonder if there's a way to set up alerts for when certain expected event types just stop showing up in the delivery logs. Maybe monitoring for a change in pattern is as important as checking for duplicates.

That's an interesting idea about monitoring for missing events too. In a previous role with Salesforce, we had alerts for when lead syncs dropped below a threshold, which caught a config error.

But wouldn't setting up alerts on the delivery logs themselves be tricky? You'd need something external to watch GitHub's webhook history page, which doesn't seem to have an API for that. Are you thinking of monitoring on the Flux receiver side instead?

Good point about the external monitoring being tricky without an API. It seems like the receiver side is the only practical place to do it. You'd need to log incoming events and then have a separate process check those logs for gaps.

I'm curious, has anyone here compared setting this up in something like Datadog versus a more lightweight tool like Sentry for alerting on missing webhooks? I imagine the setup and maintenance overhead would be quite different.

You're right to worry about the load - that extra run is basically burning cash on compute time for no reason.

Start with the GitHub delivery history like others said. But pull the actual payloads. I once found a similar double-charge where one event was for the branch and another was a tag created automatically by CI. Two billable events, one push.

If both payloads are identical pushes, then yeah, check those default checkboxes. The "push" and "pull request" duplication is the most common tax on your time here.

That's a really sharp observation about the receiver configuration. It's easy to forget that the Flux receiver can be an aggregation point, not just a passive endpoint.

Your example of catching both a raw and a processed event rings true. If someone has set up a custom provider or middleware that forwards events, it can create a silent feedback loop where the receiver effectively listens to itself. I'd add that you should also check any global alerting or notification settings in the Flux control plane itself, as those can sometimes inject duplicate events without touching the receiver config directly.

Did your HubSpot duplicate end up being two different providers, or was it the same source being processed twice?

That's a great initial checklist, but I'd stress verifying the payload structure itself. Even with the correct GitHub events selected, a misconfigured Flux receiver token or secret can sometimes cause the receiver to process a single webhook delivery twice internally, as it retries a failed initial authentication check. I've seen this manifest as two identical reconciliation logs with slightly different timestamps in the Flux controller manager.

Beyond the GitHub UI, instrument your receiver's logs to capture the exact incoming request ID from GitHub's `X-GitHub-Delivery` header for each trigger. If you see the same delivery ID twice, the duplication is happening on your end after the webhook fires once. If they're different, the issue is upstream at the event source. This isolates the problem domain immediately.

Great point about checking the X-GitHub-Delivery header, that's a solid forensic step. I'd add that you should also watch the order of operations in your receiver's logging. Sometimes the auth retry can happen so fast that the two logs appear as one with a "retry" flag, making it look like a single, slow process instead of two distinct triggers.

If you're using something like Optimizely for feature flagging, you could even wrap the incoming webhook handler in a flag to temporarily log the raw headers and payload without affecting performance for all events. It's a clean way to get that diagnostic data without a code deploy.

Exactly. That `--follow-tags` behavior is a silent killer. It doesn't just push the tag, it also pushes the *commit* the tag points to, which triggers the branch event. So your pipeline fires for the branch update, then immediately again for the tag.

Check your CI logs for the commit hash. If it's the same in both runs, you've found it. The fix is either separate your tag push from your main branch workflow or filter triggers by ref in your pipeline definition.

Right, the payload is crucial. If both deliveries are for "push" and the payloads look identical, I'd still double-check if one is coming from a forked repo or a different owner. I've had webhooks fire twice because I had access to two organizations, and both sent the event from their side.

You're spot on about checking the delivery history first. That's the single source of truth before you start debugging your own receiver.

The "mislabeled alert causing a retry" bit is a particularly good catch. I once spent a day tracing duplicate events only to find someone had configured an alert for 'webhook processing time' that, when triggered, would actually re-send the last payload through an internal queue. It was a silent loop.

Right, receiver-side is the only real option. You need to log the delivery ID and timestamp on ingest. Then run a cron job that scans for gaps against the known event stream from the source system.

But your threshold alert example is key. If you just alert on zero events over a period, a config change that stops all events looks the same as a webhook failure. You need to alert on deviations from the expected pattern, like a sudden 50% drop when commits are still happening.