Skip to content
Notifications
Clear all

How do I convince engineering that 'cost as a metric' is their job too?

3 Posts
3 Users
0 Reactions
3 Views
(@db_diver)
Estimable Member
Joined: 4 months ago
Posts: 93
Topic starter   [#17662]

This is a foundational operational challenge I've observed across numerous organizations, particularly as data layer complexity and managed service adoption increase. The core issue is often a misalignment of incentives and a lack of visibility. Engineering teams are typically measured on availability, latency, and feature velocity. Cost, especially infrastructure cost, is viewed as an opaque "ops" or "finance" problem, a tax on their work rather than a direct output of their architectural decisions. To shift this mindset, you must make cost a tangible, actionable engineering metric, as integral to the review process as p99 latency.

The most effective strategy is to instrument and attribute cost with the same rigor you apply to application performance. Abstract cloud bills are useless for engineers. You need per-service, per-feature, and often per-query cost telemetry. For database services—my primary domain—this is critical. Consider the vast cost difference between these two approaches to the same logical operation:

**Scenario: Fetching user activity history.**

*Option A (Naive):*
```sql
SELECT * FROM user_events WHERE user_id = ? AND event_time > now() - interval '30 days';
```
*This could scan terabytes in a poorly partitioned table, incurring massive I/O costs in Aurora or BigQuery.*

*Option B (Optimized):*
```sql
SELECT event_type, event_time FROM user_events_by_date
WHERE user_id = ? AND date_partition = '2024-10'
ORDER BY event_time DESC LIMIT 1000;
```
*This uses a partitioned table and a covering index, limiting scanned data to a few gigabytes.*

The performance difference is obvious to an engineer. The cost difference, however, is invisible unless you have tooling to map scanned bytes or consumed RCU/WCUs back to the specific service and API call. Without this, the financial impact of choosing Option A over Option B is completely abstracted away.

To operationalize this, you must:

* **Implement granular cost attribution.** Use labels, tags, or resource hierarchies (AWS Resource Groups, GCP Projects/Folders) to assign all resources—especially managed databases, caching layers, and message queues—to specific teams, services, and environments. A $5,000/month Cloud SQL or Spanner bill is a talking point. A $1,200/month bill for the `user-preferences-service`'s Cassandra cluster is an engineering action item.
* **Bridge the gap between technical metrics and cost drivers.** Export metrics like:
* DynamoDB/ Cosmos DB: Provisioned capacity utilization vs. on-demand request charges.
* BigQuery/ Snowflake: Bytes scanned per query job.
* PostgreSQL/ MySQL (RDS, Cloud SQL): Total IOPs consumption, replica lag forcing read primaries.
* Redis (ElastiCache): Evictions leading to cache misses and increased database load.
* Dataflow/ Spark: Shuffle I/O and worker hours.
Correlate these with the cost feed. Show the team that a new feature's inefficient query pattern directly increased the weekly BigQuery slot commitment by 15%.
* **Integrate cost into the development lifecycle.** This is non-negotiable.
* **Design Reviews:** Include a "cost impact" section. What is the expected load profile? What is the estimated monthly run-rate for the new DynamoDB table or Memorystore instance?
* **Pull Requests:** For infrastructure-as-code changes (Terraform, CloudFormation), require estimated cost diff output from tools like `infracost`.
* **Post-Mortems & Performance Reviews:** Include significant cost overruns or inefficiencies as a learning item, alongside outages and latency regressions.

Ultimately, convincing engineering requires translating the finance team's spreadsheet into the engineering team's dashboard. It's about demonstrating that architectural elegance isn't just about speed and resilience—it's also about economic efficiency. A service that meets its SLOs but costs 10x more than a slightly slower alternative is a poorly designed service. When an engineer can see that refactoring a hotspot key in their Cassandra data model or adding a composite index to their Aurora PostgreSQL instance will drop the weekly bill by a measurable amount, cost becomes a satisfying optimization puzzle, not an administrative burden.


SQL is not dead.


   
Quote
(@coffeegoblin)
Estimable Member
Joined: 1 week ago
Posts: 82
 

Ah yes, the classic "just give them the data and they'll magically care" approach. I've seen this play out. You slap per-query cost tags on everything, and what happens? Engineers optimize for the wrong thing. They start caching everything in memory to avoid a $0.0002 query, building a $500/month Redis cluster to save $12 in database reads. Or they batch operations into huge monolithic queries that take 30 seconds but cost fewer pennies, and now your p99 latency is in the toilet.

The real problem isn't visibility. It's that cost is a lagging, collective outcome while feature velocity is a leading, individual reward. Until you tie a bonus or a promo to staying under a budget line, those nice cost dashboards are just pretty wall art. Plus, good luck getting a team to agree on what "per-feature" attribution even means when your monolith has 15 microservices all hitting the same Aurora cluster.

Let me know how that `SELECT *` telemetry works when the finance team asks why the "user activity history" cost went up 20% and the engineer says "well, we added a join".


Buyer beware.


   
ReplyQuote
(@davidm78)
Estimable Member
Joined: 1 week ago
Posts: 64
 

You're spot on about needing per-query granularity. That example is perfect - I've seen those exact queries and the surprise on devs' faces when you show them the billable compute difference.

One caveat from my Looker work: even with perfect per-query costing, you need to give engineers a clear, simple heuristic. Like a "cost per request" target for API endpoints, or a budget cap for a scheduled job. Otherwise, the data is just noise. The goal is making cost a first-class constraint during design, not an autopsy metric.

How do you actually enforce those thresholds, though? That's where we usually hit cultural roadblocks.


Data doesn't lie, but dashboards sometimes do.


   
ReplyQuote