Okay, so I’m usually the person talking about stitching together HubSpot and Salesforce with a side of Zapier, but this was a whole different beast. Our forcing function was brutal: we had to decommission a legacy billing system (like, truly ancient, mainframe-adjacent) before the vendor support vanished. The problem? Five years of customer and invoice data lived there, and our modern CRM + ERP only had the last 18 months synced via a shaky, now-broken API bridge.
We decided on a full-stack rebuild of the "data pipeline" layer for financial records. The sequence was: 1) extract everything from the legacy system, 2) land it raw in a new cloud data warehouse, 3) transform and map it to our new schema, 4) feed the CRM/ERP. Steps 2-4 were straightforward with modern tools (think Fivetran, dbt, and custom scripts). But step 1… that was the nightmare.
The legacy system had no usable API. No bulk export. Just a green-screen terminal interface. Our engineering team built what we called a "Claw" probe—a custom script that essentially automated terminal emulation. It would:
* Log in via SSH and navigate the menu structure using expect-like logic
* Run a series of pre-defined reports (by date ranges, by customer segments)
* Scrape the fixed-width text output from those reports
* Parse and batch the data into JSON files for staging
It was ugly, but it worked… until it didn't. Halfway through the extraction, the probe started returning partial data sets. No errors, just… missing chunks. We had no visibility. Panic set in.
Then we remembered: the legacy system had an audit log feature (mostly used for compliance) that tracked every report run, by user, with a row count. We reconfigured the Claw to run smaller, time-bound batches and to cross-reference each batch against the audit log's row count *immediately* after each pull. The audit log entries became our source of truth.
Where things slipped:
* We didn't account for the system's internal report queue. Sometimes reports would be delayed, and our probe would move on too fast. The audit log saved us here—we built a retry loop that checked the log for completion.
* Mapping legacy status codes (like "A" for active) took twice as long as estimated because we discovered regional variations in the data that weren't documented.
In the end, having those granular audit logs let us validate every single batch. Without them, we would have had a massive, corrupt data load and no way to trace the gaps. It was a stark reminder that sometimes the most valuable tool isn't the shiny new pipeline, but the dusty old log file you almost ignored.
Has anyone else had to build a "Claw" for a legacy system? How did you validate completeness?
— Emma
If it's not measurable, it's not marketing.