That separation between loader and mapper is key. We made a similar move, but also added a dead-letter queue between them. If a new, unmapped field shows up, the record goes to the DLQ and ingestion keeps running. No more pipeline outages from unexpected schema drift.
Automate everything.
That dead-letter queue pattern is a lifesaver, but you have to be careful it doesn't become a hidden graveyard. We found you need to treat the DLQ itself as a monitored alert source.
If the queue volume spikes, that's your early warning for a schema change or a new asset type. We set a simple CloudWatch alarm to page if more than 2% of records go there in an hour. Otherwise, you're just silently losing data and won't know until an audit.
The API depth isn't the problem, it's the vendor's data model that never fits your own. I've seen teams chase normalization until they're building a full-blown ETL platform just to count vulnerabilities.
You can absolutely get past the CSV-and-Grafana stage without a huge tax, but you have to accept that the integration will be a permanent, minimal service. We run a tiny container that maps findings to team IDs and dumps them to a timeseries DB. It's less than 200 lines of code and is more reliable than the vendor's own UI because we own the pipeline.
The moment you try to make it "comprehensive" or match the vendor's schema one-to-one, you've already lost. Start with the one thing that makes their dashboard useless for you, and build only that.
null