Having observed a concerning lack of granular visibility into our burgeoning cloud and SaaS expenditure, I recently undertook a side-project to instrument and visualize these costs. The goal was to move beyond monthly invoice shock and towards predictive, service-owner-accountable cost tracking. The architecture is built on a principle of immutable, declarative infrastructure, even for internal tooling.
The core pipeline is event-driven and deployed via Terraform and Kubernetes manifests. A brief overview of the components:
* **Extraction Layer:** Scheduled Kubernetes `CronJobs` run lightweight Python containers that utilize vendor APIs (AWS Cost Explorer, GitHub, Datadog, etc.) to pull spend data. Credentials are managed via a secret store (Vault) with short-lived tokens.
* **Transformation & Storage:** Raw data is normalized into a common schema and written to a time-series database (TimescaleDB). This is where business logic tags (e.g., `team:platform`, `project:observability-dashboard`) are applied, which is critical for meaningful allocation.
* **Visualization & Alerting:** Grafana serves as the front-end, with dashboards organized by cost center and service. Prometheus alerts are configured for anomalous spend increases against forecasted baselines.
```hcl
# Example Terraform module for a cost exporter deployment
module "github_spend_exporter" {
source = "./modules/cron-exporter"
name = "github-spend-exporter"
schedule = "0 8 * * *" # Daily at 0800 UTC
image = "internal-registry/cost-exporters:github-latest"
secret_env = {
GITHUB_TOKEN = vault_generic_secret.gh_cost_token.path
DB_CONNECTION = module.timescale_db.connection_uri
}
k8s_namespace = kubernetes_namespace.cost_tools.metadata[0].name
}
```
I am particularly interested in critiques of the data model and the trade-offs of this approach versus a centralized SaaS platform like Apptio or CloudHealth. While the latter offer richer out-of-the-box reports, this system provides unparalleled flexibility in tagging and integrates directly with our existing observability stack for correlated insights (e.g., cost per million requests). The primary weaknesses I've identified are the maintenance burden of API integration breakage and the initial lack of advanced forecasting algorithms.
I am open to feedback on the architectural pattern, especially regarding long-term data retention strategies and whether a stream-processing model (Kafka, Flink) would be justified over the current batch approach as we scale to incorporate more real-time cloud provider billing streams.
--from the trenches
infrastructure is code