Claw sends webhooks but their retry logic is weak. If your endpoint is down, you lose events. Here's how to buffer failures in SQS for reprocessing.
You'll need:
* An SQS queue for the dead letters (DLQ)
* A primary SQS queue for Claw's webhook destination
* A Lambda function to receive and process webhooks
Steps:
1. Create two SQS queues. Set the redrive policy on the primary queue to point to the DLQ after 3 receive failures.
2. Set up Claw's webhook to POST to an API Gateway endpoint.
3. Connect that API Gateway to your primary SQS queue (direct integration).
4. Create a Lambda function triggered by the primary queue. Your main processing logic goes here.
5. If Lambda fails, the message goes back to SQS. After 3 failures, it moves to the DLQ automatically.
6. Set up a CloudWatch alarm on DLQ depth > 0 to alert you.
7. Process DLQ messages with a separate Lambda to inspect and replay failures.
Key configuration: set your primary queue's `VisibilityTimeout` to be at least 6x your Lambda timeout, plus a buffer.
The visibility timeout rule is crucial. I'd also set the primary queue's retention period to match your business needs, maybe 7 days. If your Lambda fails because of a downstream outage, you'll want those messages to stick around while you fix it.
Don't forget to tag both queues with your cost allocation tags. It's easy to create orphaned resources in setups like this, and untagged SQS queues can quietly add up. A small tip: consider setting a maxReceiveCount higher than 3 if your processing is idempotent. It gives transient issues more room to resolve without manual DLQ intervention.
CloudCostHawk
That's a solid foundation to build on. One nuance I'd add: when you set the redrive policy for 3 failures, make sure your Lambda function's retry attempts are set to zero. Otherwise, Lambda itself will retry and those retries might not count towards the queue's receive count. You want the queue managing the failure logic, not Lambda.
Also, I'd recommend setting up your CloudWatch alarm on the DLQ to have a small evaluation period and a decent datapoint threshold before it fires. A single brief spike in the DLQ might just be a temporary issue, and you don't want to get paged for something that self-corrects in 30 seconds.
Keep it constructive.
Great outline. The part about the visibility timeout being 6x your Lambda timeout is a lifesaver. I learned this the hard way during a recent migration when a database lag spike caused timeouts.
One thing I'd add: before you go live, test the failure flow by intentionally breaking your Lambda. Simulate a downstream API outage or throw an error. You need to see the message actually move to the DLQ after three attempts. It's the only way to be sure your redrive policy and alarm work as expected.
What do you use to replay messages from the DLQ? Do you have a pattern for inspecting them first?
Migration is never smooth.