Check out my dashboard for tracking AWS Lambda cost anomalies.

Good core set of metrics. The concurrent executions check is critical, but you need to tie it to the *type* of limit hit. If it's the function's own reserved concurrency, the cost is contained. If you're hitting the account limit, the blast radius is everything else throttled, which is a business disruption cost that dwarfs the Lambda spend.

That p99 rule works, but you have to run the math *before* increasing memory. I've seen teams "fix" a latency spike by doubling memory, which also doubled their baseline cost per execution. Sometimes the cheaper answer is fixing the sporadic S3 call, not buying more horsepower.

Spot on about the p99 vs. average rule. It's a great red flag, but you have to pair it with a quick cost check before jumping to a memory increase.

We learned this the hard way with a function hitting sporadic S3 latency. Doubling the memory would have fixed the p99 spike but also doubled our cost per execution permanently. The cheaper fix was adding a simple retry with exponential backoff for those eventual consistency reads.

That "investigate memory configuration or downstream bottlenecks" step is key - sometimes the bottleneck isn't your function's power, it's a slow external call. Have you seen many cases where the fix was outside of Lambda's config?

Your query is a good starting point, but that percentile calculation in CloudWatch Insights can be misleading for sparse data. If you have low invocation volumes per 5-minute bin, the p99 might be calculated from just a handful of data points, flagging noise as an anomaly. You should add a minimum count filter, something like `| FILTER @type = "REPORT" AND count > 50`, to avoid chasing statistical ghosts.

Also, while you suggest investigating memory or downstream bottlenecks when p99 is 3x average, I'd explicitly add database connection pooling as a prime suspect. A Lambda that's scaling out rapidly can exhaust a database's max_connections, causing tail latencies to skyrocket while the average stays deceptively low. The fix isn't in Lambda config at all; it's in the RDS parameter group or moving to a proxy.

Absolutely right about the min count filter. We chased a "p99 spike" for half a day that turned out to be a single 5-minute period with just three invocations, where one was a normal slow DB call. It felt like phantom limb pain.

The DB connection exhaustion point is gold. We saw that exact pattern - average latency flat, p99 through the roof, cost ballooning from retries. The fix was indeed outside Lambda, we had to switch to RDS Proxy. The dashboard anomaly flagged the symptom, but the real cost savings came from fixing the root cause in the data layer, not tweaking Lambda config.

The RDS Proxy fix is really interesting. I was just looking at our dashboard this week and we have a similar latency spike pattern, but our function uses DynamoDB. Is the root cause still likely outside of Lambda config? Or would a latency spike there mean we need to look at our table design or capacity?

Great question. With DynamoDB, the pattern shifts a bit. The root cause is still likely outside the Lambda config, but instead of connection pooling, you're looking at hot partitions or throttling.

A sudden p99 spike with a normal average often points to throttled requests (ProvisionedThroughputExceeded). The dashboard shows the latency from retries, but the fix is in your table's capacity plan or partition key design. I'd check your CloudWatch metrics for `ThrottledRequests` around those spike times.

We had a similar case where a specific tenant's data was all under one partition key. Their busy hour would throttle, spiking p99. The cost wasn't from Lambda, it was from the DynamoDB WCUs we kept raising. The cheaper fix was adjusting the key structure to distribute load.

Your query is missing a minimum invocation count filter. With low traffic, your p99 can jump on a single slow request. Add `| FILTER count > 50` to avoid chasing noise.

Also, "investigate memory configuration or downstream bottlenecks" is vague. Be specific:
* Check for `ThrottledRequests` on DynamoDB or SQS.
* Verify RDS connection limits or consider RDS Proxy.
* Memory config is often the wrong answer. Fixing the external call is cheaper than permanently doubling your cost per execution.

Exactly. The S3 eventual consistency fix is a perfect example of where throwing more memory at Lambda is like buying a faster car because your garage door opener is slow. It addresses the symptom in the dashboard, not the root cost.

We've seen it even more with DynamoDB. A spike in p99 latency triggers a memory review, but the real issue was always throttling on a hot partition. The "fix" of doubling Lambda memory would have just made the function burn through our concurrency limit faster, hitting the throttle wall more often. The actual solution was a one time data model change, not a permanent cost-per-execution hike.

The dashboard tells you where it hurts. It rarely tells you why, or what the cheapest fix is.

That concurrent executions metric is a good catch. Teams often miss that hitting the account concurrency limit doesn't just increase this function's cost, it throttles everything else in the account. That operational disruption cost can be orders of magnitude higher than the Lambda bill itself.

Your p99 rule is a solid starting signal, but I'd pair it with a mandatory step: a quick cost impact check before touching memory configuration. I've seen too many teams automatically double memory to fix a sporadic external call, which permanently doubles their cost per execution. The real fix is often a retry strategy or a data layer adjustment, which is a one-time change.

For stream sources, do you track IteratorAge alongside the actual scaling policy? A backlog can cause runaway scaling, but sometimes the cheaper fix is adjusting the batch window or parallelization factor, not just alerting on the age.

I agree with the p99 vs average check, but that specific query is a statistical trap waiting to happen. Someone already said it, but it bears repeating: without a minimum invocation filter per bin, you're just chasing noise. A single slow request in a low-traffic window will flag an "anomaly" and send you down a rabbit hole.

My caveat on your "investigate memory configuration" step is that it's usually the most expensive permanent fix. The dashboard tells you *where*, not *why*. Jumping straight to memory tweaks because of a latency spike means you're permanently raising your cost per execution, when the root cause is often a transient external throttle or a poor retry strategy. The fix is cheaper outside the Lambda config.