Hey everyone! 👋
I've been wrestling with duplicate data in HubSpot for what feels like forever, especially after importing lists or running LinkedIn automation into our sequences. It throws off lead scoring, messes with outreach cadence, and just generally makes our revenue ops team grumpy.
So, I built a Python script to tackle this, and I've decided to open-source it. It’s specifically tuned for businesses running a high-volume, inbound-heavy model (think 5k+ new contacts/month). The logic focuses on merging records based on email domain and recent activity, which has been a game-changer for us.
Here’s what it does:
* Scans for duplicates based on configurable fields (we default to email, company domain, and LinkedIn profile).
* Prioritizes which record to keep based on engagement data (last form submission, email open, etc.).
* Logs every merge for compliance and can run on a schedule.
* It's built to work with HubSpot's API, but the pattern could be adapted for other CRMs.
I'd love for you to try it out, contribute, or suggest improvements. It's saved us countless hours of manual cleaning.
You can find the repo here: [Link to GitHub Repo]
Has anyone else built something similar? How are you handling the duplicate problem in your stack?
— Dan
spreadsheet ninja
Nice approach on using engagement data to prioritize merges. That's way better than just picking the newest record, which is what a lot of scripts do.
Have you run into any rate limiting issues with the HubSpot API during large scans? I've found you sometimes need to build in pretty aggressive exponential backoff, especially for those 5k+/month volumes.
I'm also curious about the logging for compliance. Are you capturing just the merge action, or the full state of the merged records before the change? That latter bit can be important for some audit trails.
Cloud cost nerd. No, I don't use Reserved Instances.
Great point on both counts! The rate limiting is real, especially when you're pinging the API for all that activity data. We found that adding a small, randomized delay between batches (even on top of backoff) helped us glide under the radar more consistently.
On logging, you're spot on. We're capturing the full state of both records pre-merge into a separate audit log. It's a bit more data, but it's saved our skin a few times when someone asked "wait, what was in that old record?"
null
This looks super useful! That focus on engagement data to decide which record to keep is the key piece most similar scripts miss. Just picking the newest one can accidentally bury your most active lead.
I'm curious, have you thought about packaging it as a Pipedream workflow or a Make scenario template? Would make it way more accessible for the no-code folks in RevOps who need this exact thing but can't run a Python script directly.
dk