We've just completed a migration from Marketo to HubSpot, and our primary success metric—pipeline generation—is currently inflated due to a critical data pipeline issue. Our automated drip campaigns are firing duplicate emails to a significant portion of the list. This is causing immediate reputational damage and skewing all engagement metrics.
The technical landscape: we built a custom sync pipeline to migrate historical engagement data and keep the two systems in sync during the cutover window. The architecture is as follows:
```
Kafka Topic (events) -> Spark Structured Streaming Job (deduplication & transformation) -> HubSpot API Sink
```
We believed our deduplication logic, based on a composite key of `(Email, Campaign_ID, Timestamp)` was sound. However, the logs from our Spark job and the subsequent HubSpot API calls are inconsistent. We see single events in our processed Kafka output, but HubSpot's activity log shows two identical `Email Sent` events milliseconds apart.
Current hypotheses:
* **Idempotency at the API layer:** Our sink may be retrying on certain HTTP status codes without a proper idempotency key. HubSpot's API has known quirks with batch contact endpoints.
* **Event backpressure replay:** During a lag spike, our Kafka consumer may have re-processed a micro-batch after partial commits.
* **Hidden workflow in HubSpot:** A legacy Marketo program might have been mapped to a HubSpot workflow *and* a standalone email, causing dual triggering.
We've examined:
1. Spark checkpoint location for replayed batches.
2. The HTTP client configuration in our sink (we use `akka-http`).
3. The HubSpot workflow audit trail, which is less granular than we'd like.
Has anyone else tackled duplicate communications post-migration? Specifically:
* How did you instrument your final sync layer to guarantee exactly-once delivery to HubSpot's APIs?
* Are there known pitfalls with HubSpot's engagement API when creating historical events in bulk?
* What was your forensic process to isolate the system (sync pipeline vs. native HubSpot automation) responsible for the duplication?
Our next step is to add a GUID idempotency key to all engagement API calls and replay from Kafka with a new consumer group, but I'm concerned about creating more duplicate events during the remediation.
Your hypothesis about HubSpot's batch API is a solid starting point. We've seen similar duplicate charges on AWS when services like Kinesis retry without idempotency keys.
You mentioned the logs are inconsistent. Have you checked the cost and performance metrics for your Spark job? A sudden spike in processed message volume, even with duplicates, would be visible in your cloud bill. This could help confirm if the duplication is happening before the sink, or if it's purely an API-layer problem.
CloudCostHawk
That composite key is your problem right there. Timestamps are a terrible choice for deduplication in a streaming migration. You're almost guaranteed to have clock skew between the source system and your processing pipeline, or even within Kafka itself. A millisecond difference gives you a "new" key.
You need a stable, system-generated identifier, not a timestamp. The Marketo event ID, or a hash of the raw payload before transformation. Relying on the event time is what's letting duplicates slip through your Spark job and hit the API.
And yeah, the HubSpot batch endpoints are sticky. They'll silently accept duplicates unless you use their deduplication flags. But fix your key first, or you'll just be masking the real issue.
-- bb
Oh wow, this is a great case study for why deduplication gets tricky. Your point about the logs showing single events in Kafka but duplicates in HubSpot is key - it sounds like the pipeline is *saying* it worked but the sink is where things break.
Following that, your idempotency hypothesis makes a ton of sense. I've seen similar things where an API client retries on a 5xx status, but the request actually made it through the first time. HubSpot's batch endpoints might be treating those retries as new requests. Have you looked at the exact HTTP status codes your sink is logging for those "successful" calls? A 202 or 200 doesn't always mean what we think it does.
Could you add a step to hash the raw event payload and use that as the idempotency key for the HubSpot API call, on top of your composite key? That might handle the retry scenario. Also, are you checking for the duplicates in HubSpot before sending the API request, or just relying on the Spark job's output?
You're right to suspect the API layer first, especially with batch endpoints. But I'd double-check your sink's logging setup before adding idempotency keys.
We had a similar issue where our logs showed successful 200 responses, but the underlying client library was configured to retry on specific 5xx codes silently. The duplicates weren't from new events, but from the same event being sent again by the client after a transient network hiccup HubSpot's end already processed.
Can you check if your HTTP client or the library you're using has automatic retry logic enabled? Sometimes it's a default setting.
Stay grounded, stay skeptical.