Hey folks, been a while! I've been neck-deep in a migration for our main transactional email pipeline and wanted to share some real-world experience.
We were using a popular managed RabbitMQ service for years. It was great for complex routing and guaranteed ordering within a queue. But as our volume grew, the cost started to sting, and the operational overhead for monitoring and scaling felt heavier than I wanted.
So, we took the plunge and re-architected around SQS (Standard Queue) and Lambda. The core trade-off is moving from a powerful, feature-rich broker to a simpler, "dumber" queue system. Here’s what we gained and what we gave up:
**The Wins:**
* **Cost:** Our bill dropped by about 60% for the same message volume. SQS pricing is just hard to beat.
* **Operational Simplicity:** No more worrying about node health, disk space, or connection storms. It’s just there.
* **Seamless Scaling:** Lambda polling SQS scales almost magically with the queue depth. Spikes in send requests are handled without a hiccup.
* **Tighter Integration:** With everything else already on AWS, the IAM and CloudWatch integration is cleaner.
**The Trade-offs & Adaptations:**
* **Loss of Sophisticated Routing:** This is the big one. We had to move our "routing logic" (e.g., "send to this template if user is in segment X") from the broker into the producer and consumer Lambdas. It's more code for us to manage.
* **No Guaranteed Ordering:** SQS Standard doesn't guarantee FIFO. For our emails, this is acceptable (two welcome emails in the wrong order is fine), but it's a critical design consideration.
* **Message Size:** SQS has a 256KB payload limit. We now store large payloads (like complex personalization JSON) in S3 and just pass a reference in the message.
The switch forced us to simplify our message structures and think more about idempotency in our consumers. For a high-volume, cost-sensitive workflow like transactional email where "at-least-once" delivery is good enough, it's been a fantastic move. I wouldn't use this pattern for a financial transaction system, but for marketing comms? It's a powerhouse.
Would love to hear if others have made similar moves or found clever ways to handle the routing challenge!
Billy
Always A/B test.