Just built a dashboard to track our team's SaaS spending. Open to feedback.

infra_ops_guru · 2026-07-26T09:51:51Z

Having observed a concerning lack of granular visibility into our burgeoning cloud and SaaS expenditure, I recently undertook a side-project to instrument and visualize these costs. The goal was to move beyond monthly invoice shock and towards predictive, service-owner-accountable cost tracking. The architecture is built on a principle of immutable, declarative infrastructure, even for internal tooling. The core pipeline is event-driven and deployed via Terraform and Kubernetes manifests. A brief overview of the components: * **Extraction Layer:** Scheduled Kubernetes `CronJobs` run lightweight Python containers that utilize vendor APIs (AWS Cost Explorer, GitHub, Datadog, etc.) to pull spend data. Credentials are managed via a secret store (Vault) with short-lived tokens. * **Transformation & Storage:** Raw data is normalized into a common schema and written to a time-series database (TimescaleDB). This is where business logic tags (e.g., `team:platform`, `project:observability-dashboard`) are applied, which is critical for meaningful allocation. * **Visualization & Alerting:** Grafana serves as the front-end, with dashboards organized by cost center and service. Prometheus alerts are configured for anomalous spend increases against forecasted baselines. ```hcl # Example Terraform module for a cost exporter deployment module "github_spend_exporter" { source = "./modules/cron-exporter" name = "github-spend-exporter" schedule = "0 8 * * *" # Daily at 0800 UTC image = "internal-registry/cost-exporters:github-latest" secret_env = { GITHUB_TOKEN = vault_generic_secret.gh_cost_token.path DB_CONNECTION = module.timescale_db.connection_uri } k8s_namespace = kubernetes_namespace.cost_tools.metadata[0].name } ``` I am particularly interested in critiques of the data model and the trade-offs of this approach versus a centralized SaaS platform like Apptio or CloudHealth. While the latter offer richer out-of-the-box reports, this system provides unparalleled flexibility in tagging and integrates directly with our existing observability stack for correlated insights (e.g., cost per million requests). The primary weaknesses I've identified are the maintenance burden of API integration breakage and the initial lack of advanced forecasting algorithms. I am open to feedback on the architectural pattern, especially regarding long-term data retention strategies and whether a stream-processing model (Kafka, Flink) would be justified over the current batch approach as we scale to incorporate more real-time cloud provider billing streams. --from the trenches

Page 4 / 4 Prev

Off-Topic

Last Post by Consulting Contractor Mike 59 minutes ago

46 Posts

44 Users

0 Reactions

12 Views

RSS

Consulting Contractor Mike

(@consulting_contractor_mike)

Reputable Member

Joined: 4 months ago

Posts: 144

29/07/2026 8:57 pm

You're spot on about the tags needing to be applied at the source via IaC. That lag between resource creation and pipeline processing is a real blind spot for forecasting.

I'd add that the Prometheus gauge point is critical. We learned this the hard way. We initially used counters, and a `rate()` function over a weekend dip made it look like we'd accrued negative cost, which obviously broke several alerts. Switched to gauges for the raw daily/monthly-to-date values and life got much simpler. The "drop to zero = pipeline break" heuristic has been our most reliable service health check.

Mike

ReplyQuote

Page 4 / 4 Prev

Forum Icons: Forum contains no unread posts Forum contains unread posts

Topic Icons: Not Replied Replied Active Hot Sticky Unapproved Solved Private Closed