How do you handle dev/staging environments without paying production rates?

A perennial challenge in our cloud infrastructure management is the financial inefficiency introduced by non-production environments. While development, staging, and QA systems are indispensable for software delivery, their observability telemetry—logs, metrics, and traces—often incurs costs at the same per-gigabyte or per-sample rate as production. This is fundamentally a misalignment of value versus expenditure, as the business impact and thus the required fidelity of data in these environments is orders of magnitude lower.

I propose a multi-layered strategy to decouple dev/staging observability costs from production pricing models. The core principle is to aggressively reduce volume, fidelity, and retention for all telemetry signals outside of production.

**Log Management Strategy:**
* **Ingestion Control:** Implement mandatory filtering at the agent level (e.g., Fluent Bit, Datadog Agent, OpenTelemetry Collector) to discard irrelevant logs. This includes DEBUG/VERBOSE levels, health-check pings, and noisy dependencies.

```yaml
# Example Fluent Bit filter to drop DEBUG logs and high-volume chatty module
[FILTER]
Name grep
Match app.*
Exclude log_level DEBUG
Exclude component stats_collector
```
* **Sampling:** Apply head-based sampling at the source for application logs. For example, sample 1 in 10 log events in staging.
* **Destination Routing:** Route all non-production logs to a separate, cost-optimized observability backend. This could be a dedicated, lower-tier tenant in your commercial vendor or, more radically, a self-managed stack (e.g., Grafana Loki, ELK) on cheap compute (spot instances, low-cost object storage).

**Metrics & Tracing Strategy:**
* **Reduced Resolution:** Configure metrics collection in dev/staging to a 1-minute or 5-minute granularity, as opposed to 15-second in production. This directly reduces the number of time-series datapoints stored.
* **Cardinality Culling:** Enforce strict limits on labels/tags in dev environments. Prevent high-cardinality attributes like `user_id`, `request_id`, or `session_id` from being emitted on metrics.
* **Trace Sampling:** Implement a high-rate tail-based sampling policy. For example, sample only 5% of traces in staging, and 0.5% in development. Ensure error traces are always sampled, but successful requests are heavily down-sampled.

**Architectural & Billing Considerations:**
* **Separate Accounts/Subscriptions:** Provision development and staging workloads in entirely separate cloud accounts or subscriptions. This creates hard cost boundaries and allows for the application of different pricing tiers or commitment plans (e.g., no Reserved Instances for observability tools in dev).
* **Retention Policy:** Enforce a strict, automated retention policy. Dev logs and traces need not be kept beyond 7 days. Staging data might be retained for 15-30 days. This is often the most potent cost lever.
* **Vendor Feature Tiers:** If using a commercial vendor, actively downgrade the feature set for your dev/staging data. Do you need AI/ML anomaly detection, advanced analytics, or unlimited alerting on your staging dashboard? Almost certainly not.

The critical success factor is automation. These policies must be codified in Infrastructure as Code and applied uniformly. The goal is to make the low-fidelity, cost-optimized observability for non-production the default, requiring an explicit, justified override to elevate it to production-grade fidelity.

What specific numerical reduction have you achieved by implementing such measures? I am particularly interested in the percentage differential between your production and non-production per-environment observability spend, and the breakdown of savings between log volume reduction, metrics cardinality control, and trace sampling.

Show me the bill.