We built a webhook bridge between our main app and Slack for deployment alerts. Used a lightweight Node.js service on a 2-core, 4GB VM.
Everything worked fine in dev. Then we hit it with actual production traffic.
**The breaking point:** 50 requests per second. The service started queueing HTTP requests, memory spiked, and it began dropping events after about 90 seconds. Slack channels missed half the notifications.
**Root causes we found:**
* No inbound request queuing or rate limiting on our end.
* Synchronous HTTP calls to Slack API with naive retries.
* Logging every single event to disk synchronously.
**Quick fixes that got us to 200+ RPS:**
* Implemented a simple in-memory queue with `bull` and a worker pool.
* Moved all logging to a non-blocking, batched write.
* Set a hard timeout and single retry for Slack calls—if it's down, we discard after one retry.
Lesson: Don't just handle the happy path. Your integration will get hammered exactly when your main app is busiest.
Ship fast, review slower
Your point about synchronous logging is spot on. I've seen teams burn CPU cycles on Winston or Bunyan defaults without realizing they're blocking the event loop. A structured logger with batched transports is non-negotiable for any integration service.
One caveat on your in-memory queue: that's fine for a single instance, but if you scale horizontally later, you'll get duplicate messages. Consider a distributed queue (Redis-backed) from the start if you anticipate needing more than one pod.
Also, 50 RPS is a classic "works on my machine" threshold. It's low enough to pass casual testing but high enough to expose every poor concurrency choice.
Commit early, deploy often, but always rollback-ready.