We have a multi-tier app running on ECS Fargate, with RDS, SQS, and CloudFront. Sumo Logic collects logs from all these sources. The problem: tracing a single user request's journey is painful. A transaction ID exists in our app logs, but it's lost when that request triggers an SQS message or a CloudFront error.
Current setup highlights:
* App containers log to stdout, collected via the Sumo Docker collector.
* AWS sources (CloudFront, SQS metrics, RDS) use Sumo's AWS CloudWatch sources.
* The app injects a `correlation_id` field into its JSON logs.
The gap: AWS service logs don't automatically pick up that ID.
What's the most efficient method to bridge this? I'm evaluating:
1. **Enrich AWS logs at ingestion**: Use a Sumo Logic `Transformer` to parse and inject a `correlation_id` from, for example, SQS message attributes or a query parameter in CloudFront URLs. Is the performance hit negligible at high volume?
2. **Query-time correlation**: Use a `join` or `subquery` in a Sumo query to link app logs to AWS logs on a common key (like `request_id`). This seems expensive and slow for ad-hoc debugging.
3. **Custom Lambda forwarder**: Bypass CloudWatch, write a Lambda to receive logs, enrich them with context from our app, then send to Sumo. Adds complexity.
Has anyone implemented a clean, low-latency solution for this? Specifically:
* What's the query performance difference between method 1 and 2?
* Any pitfalls with the Transformer approach for high-cardinality fields?
* Is there a way to pass the `correlation_id` through SQS message attributes and have Sumo index them without a custom collector?
Example of our current app log structure:
```json
{
"timestamp": "2023-11-07T10:00:00Z",
"level": "INFO",
"correlation_id": "req_a1b2c3d4",
"message": "Processing request"
}
```
Need to see that same `correlation_id` in the corresponding SQS `SendMessage` log entry.