Alright, let's get straight into the muck. I've spent the last three days trying to stop my engineering manager from having a heart attack because Kubecost is screaming that our monthly AWS bill is projected to be double what AWS Cost Explorer shows. Naturally, the first assumption is that someone left a crypto-mining rig running in a dev cluster. After auditing everything, that's not the case. The discrepancy is systematic.
The core issue, as it often is, lies in the fundamental difference between how these tools *attribute* and *calculate* cost. AWS Cost Explorer shows you what AWS charges you on your invoice. Kubecost tries to show you what your Kubernetes resources *actually cost*, which involves a layer of allocation and estimation that AWS doesn't do. Here are the usual suspects I've hunted down, in order of likelihood:
* **Unallocated Cloud Costs:** This is the big one. Kubecost ingests your AWS CUR (Cost and Usage Report) and then tries to map those line items to Kubernetes concepts (Nodes, PVs, LoadBalancers). Anything it can't map gets dumped into an "Unallocated" bucket. This includes:
* Classic Load Balancers (not the Kubernetes Service type)
* RDS instances, ElastiCache, S3 buckets, Data Transfer not tied to a k8s Service
* Reserved Instance/Savings Plan discounts that haven't been amortized correctly
* **Check your Kubecost "Allocation" view and look for a massive "Unallocated" slice.** That's your money, but Kubecost can't tell which namespace to blame.
* **Node Pricing Resolution:** If Kubecost can't pull pricing for a specific node type from your cloud provider API (maybe an API glitch, or you're using a custom AMI), it falls back to its own internal pricing database. This database is sometimes... optimistic. You need to verify the node cost Kubecost is using matches your on-demand or reserved instance pricing.
* **Asset Reconciliation Window:** Kubecost doesn't see costs in real-time like AWS Billing does. It processes the CUR, which has a latency of several hours. If you're looking at "today's" cost in Kubecost vs. AWS, you're comparing estimates (Kubecost) against partial actuals (AWS). Always compare fully closed periods, like last week or last month.
Here's a snippet from our Kubecost `values.yaml` for the Helm install that we had to adjust to get better AWS cost reconciliation. The key was ensuring the `awsSp` and `awsAthenaProjection` settings were correct so Savings Plans were applied properly.
```yaml
kubecostProductConfigs:
cloudIntegrationSecret: "cloud-integration"
awsSpotDataFeed: "s3://our-aws-spot-data/prefix"
awsSp: "true"
awsAthenaProjection: "true"
prometheus:
server:
global:
external_labels:
cluster_id: "our-cluster-name-prod"
```
The final step was building a simple reconciliation script to compare a closed period. It's not pretty, but it runs in a CI job weekly to flag discrepancies >5%.
```bash
# Pseudo-logic:
# 1. Query Kubecost API for last month's total cluster cost.
# 2. Use AWS CLI to get the actual cost for the same period from Cost Explorer.
# 3. Calculate the 'Unallocated' from Kubecost's allocation API.
# 4. If (Kubecost Total - Unallocated) is within ~10% of AWS Actual, it's probably working.
# The discrepancy is just stuff outside k8s.
```
If your numbers are still wildly off after checking all this, the problem is usually in the CUR setup or the IAM permissions for Kubecost to read it. Don't trust the defaults. Assume they're wrong until you prove otherwise.
fix the pipe
Speed up your build