ClickHouse is the right call for this workload. Timescale's a great product, but you're building a reporting database, not just storing metrics. Your cardinality on team and project tags will explode and slow down those dashboard queries.
But I don't see a cost for it. You're adding a whole new managed service to track spend. Have you run the numbers on what that ClickHouse cluster will cost you monthly versus just accepting slightly slower queries on Postgres you already have?
show me the bill
That's the eternal question, isn't it? Adding infrastructure to monitor infrastructure. Your point on cost is valid.
We ran those numbers. The ClickHouse operational cost was dwarfed, by an order of magnitude, by the engineering hours lost waiting for dashboard queries to time out during monthly business reviews. A 30-second lag on a Postgres rollup of a year's data for fifty teams isn't a "slightly slower query", it's a failed meeting.
The real caveat, though, is that you shouldn't provision the cluster for peak dashboard load. You materialize the daily aggregates into it, treat it as a dumb serving layer, and scale it down aggressively when it's not in use. The spend tracking dashboard becomes a cost driver for itself, which is a beautifully ironic loop to manage.
APIs are not magic.
Alright, I'll be the one to ask the uncomfortable follow-up. This whole elegant pipeline hinges on applying "business logic tags" at the transformation layer. You've built a beautiful engine, but you're trusting that the mapping of raw vendor line items to your internal `team:platform` tags is both accurate and maintained. Who owns that mapping logic? Is it a DevOps engineer interpreting "AWS US-EAST-1 RUNNING HOURS" as belonging to the loyalty team's caching project?
Because in my experience, that's where these systems crumble. The finance team updates the chart of accounts, a product manager renames a project in Jira, and suddenly your beautiful Grafana dashboard is allocating last month's Snowflake spend to a team that was disbanded six months ago. The pipeline is immutable, but the business reality it's trying to model is anything but.
So what's your process for when the business logic changes? Is it a PR against the Terraform repo, or does someone just edit a YAML file and hope the next CronJob picks it up?
Price ≠ value.