Hey everyone! I’ve been lurking here for a bit, trying to figure out our AWS setup for my small team. We’re running a few Lambda functions for data processing, and I kept getting overwhelmed trying to balance performance and cost. The memory setting felt like a total guessing game.
I finally got fed up and spent a weekend building a script that looks at the last 100 invocations of a function (using CloudWatch Logs Insights) and suggests an optimal memory size. It’s not super fancy, but it calculates the actual memory used vs. allocated and checks if the duration would change. The idea is to run it manually every couple of weeks as our usage changes.
I'm sharing because I figure other small teams might find it useful, and honestly, I’d love some feedback from people who’ve been down this road. Does this approach make sense, or am I missing something obvious? I’m a bit worried about over-optimizing too early, but our bill last month gave me a scare 😅
Here’s the gist of the logic:
- It fetches the `maxMemoryUsed` and `BilledDuration` from logs.
- It uses the inverse relationship between memory and duration (more memory = faster, to a point) to estimate if a change would be cost-effective.
- It spits out a recommendation like: "Your function uses 890MB max. At 1024MB, duration might drop by ~15%. Cost per run would be roughly the same, but it'd be faster."
Has anyone else tried something like this? I saw there are paid tools that do this, but for our scale, even a small managed service fee feels like a lot. Curious if you think this is a good stopgap or if we should just bite the bullet and use something off-the-shelf.
Small team, big decisions
Your inverse relationship assumption is the flaw. Memory scaling only impacts CPU proportionally up to 1,792 MB. Past that, you get more memory but no more CPU. Your script might over-provision without benefit.
You're also missing cold starts. Higher memory reduces cold start duration, which is critical for intermittent functions. Your 100-invocation sample might miss that if runs are clustered.
Consider the actual cost formula. It's about GB-second. If you double memory and duration only drops by 30%, you're losing money. The break-even math is simple:
`(old_mem * old_dur) > (new_mem * new_dur)`
Manually running it is a waste of time. You should set up a weekly cron with the script and auto-apply changes if the projected savings exceed, say, 10%. Otherwise you'll stop doing it in three months.
cost per transaction is the only metric