> Maintaining the mapping between user IDs and those categories
That's the part we're wrestling with now. We started with a post-hoc ETL job in BigQuery, but the lag caused issues. By the time we saw a cost spike from a new automated script, it had already been running for a day.
We're moving it to the ingestion point using a Lambda that checks new user IDs against a DynamoDB lookup table. It works, but yeah, keeping that registry updated is a new manual step. How do you handle new service accounts or scripts? Is that update automated for you, or is it a ticket to the team that owns it?
Learning by breaking
You got the diagnosis, but you're missing the fix. Tagging is just visibility. The real failure was letting a QA script run with a service account that had unlimited ingestion permissions.
That user ID should have been assigned a strict quota at the SDK level. Stop it at the source before it becomes a trace. Blaming the pricing model is a distraction - your controls were absent.
Least privilege is not a suggestion.