Just discovered Lambda Destinations. Was trying to build a simple 3-step pipeline and wanted to avoid Step Functions cost/complexity for now.
You can configure a function to send its result (or failure) directly to another service. For my use case, I chained two Lambda functions. The first processes a raw file from S3, the second loads the result into BigQuery.
My setup:
* First Lambda (`process_raw`) has an S3 trigger.
* On success, it sends its output to an SQS queue.
* Second Lambda (`load_to_bq`) is triggered by that queue.
The key was the async invocation and the destination config. Here's the CDK Python code for the destination:
```python
# ... inside the first function's definition
process_raw_fn = lambda_.Function(...)
# Configure the destination for successful invocations
success_destination = destinations.SqsDestination(queue)
process_raw_fn.configure_async_invoke(
on_success=success_destination
)
```
It works! The payload flows automatically. Main gotchas I found:
* The destination payload wraps your function's result in a JSON structure. Your next function needs to parse `event.detail.responsePayload`.
* Only works for asynchronous invocations (like S3 triggers), not synchronous calls (like API Gateway).
* You can also send failures to a different destination (like a dead-letter queue).
Is this a common pattern? When would you *not* use this over Step Functions? I'm thinking error handling and complex retry logic might get messy.
That's a neat use case! I've been using SQS queues as destinations too, mostly for error handling to send failures to a "dead-letter" process. I never thought to use the success path for a direct chain like this.
> Main gotchas I found: The destination payload wraps your function's result in a JSON structure.
This caught me out the first time as well. The nested payload means your second function's logic is tied to that specific format. It's simple, but it's a bit of a lock-in if you ever want to trigger that second function from somewhere else later.
Have you run into any issues with the async invocation's retry behavior? I know it'll retry twice on failure before sending to a failure destination, which could cause duplicate queue messages if your first function is idempotent.
sales with substance
That's a clean implementation for a simple two-step chain. Your point about the cost avoidance versus Step Functions is valid for low-volume workflows, but I'd be curious about the numbers at scale.
Have you calculated the cost differential? With this pattern, you're paying for two Lambda executions and SQS message charges. Step Functions charges per state transition. The break-even point is surprisingly high; I've seen cases where Step Functions becomes cheaper only after several million transitions per month due to its free tier. For a simple linear chain like yours, the native destination pattern likely wins on cost.
One caveat beyond the payload wrapping you mentioned is observability. You now have two separate invocations in CloudWatch Logs, and the trace between them is the SQS message. Compare that to a single Step Functions execution ID that tracks the entire flow. That can increase debugging time, which is an indirect cost.
CostCutter
Oh, the cost break-even point is surprisingly high? I'd love to see those calculations in the open because every time I run the numbers for a linear three-step chain, Step Functions eats my lunch well before a million transitions. You're paying per state transition, and with even one retry built in you double that cost. The free tier for Step Functions is only 4,000 transitions free per month, which is practically nothing. SQS costs pennies per million requests. So unless you're talking about truly massive scale where you can absorb the per-state overhead, the native destination pattern wins across a pretty wide band.
As for observability, the lock-in complaint is a bit precious. You can stuff a correlation ID into your SQS message body and track it through CloudWatch Logs just fine. If you're truly paranoid, use X-Ray for the async invocation path - it traces across SQS if you instrument properly. Yes, it's not the single execution ID magic of Step Functions, but let's not pretend debugging a two-function chain is a months-long ordeal. For anything beyond five steps, sure, I'd rather use Step Functions. But for a simple pipeline, you're trading a few minutes of digging through logs against the overhead of managing yet another state machine definition. Pick your poison.
Price ≠ value.