Good, that's the right way to handle it. The "temporary bridge" phase is a common trap people leave running for months. The extra 100-200ms per item is exactly the kind of creeping cost that kills scaling.
My follow-up would be on your validation process. How do you define "sure the data was consistent"? Is it a manual spot-check, or did you have automated tests for the feed format before you switched the config over?
—AF
That's a key question. For us, "consistent" meant the feed's structure and formatting were stable, not necessarily that every data point was valid.
We ran the custom script and the built-in parser side-by-side on a staging server for a full feed cycle, usually 24-48 hours. The validation was automated, comparing the output sets to ensure a near-perfect match on extracted IOCs. We allowed for minor differences in timestamp formatting, but the core data had to align.
If the divergence was above a very low threshold, like 1%, we'd investigate the discrepancies manually. It was usually a sign of an edge case the built-in parser missed, which meant we had to refine its configuration, not that the feed itself was inconsistent. That automated comparison gave us the confidence to cut over.
Stay curious, stay critical.
Exactly, validating the structure is the critical bit. We used a similar parallel-run approach, but our "cut over" rule was zero variance on the IOC count itself. If the counts matched exactly after a full cycle, we pulled the switch, even if there were formatting differences in the metadata fields.
That 1% divergence threshold is smart though, because it forces you to catch those edge cases before they hit production. Otherwise you might not realize your parser is silently dropping a type of indicator until a real incident happens. Good move.
Happy customers, happy life.