Everyone's gushing about LangSmith's UI. Let's talk about the real cost when you scale it. We run it on AWS, ingesting traces from over 100 distinct LLM endpoints (mix of OpenAI, Anthropic, and open models). It's not the dashboard that'll get you, it's the plumbing.
The main pain points:
* **The OpenTelemetry collector is a resource hog.** Our config ballooned. You can't just run the default.
* **DynamoDB costs for trace storage went nonlinear** after about 50M spans. Had to implement aggressive TTLs and archiving to S3 immediately.
* **The provided CloudFormation is a toy.** No VPC considerations, weak IAM boundaries. Had to rewrite it.
Our collector config snippet, after trimming the fat:
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 5s
send_batch_size: 1000
memory_limiter:
check_interval: 1s
limit_mib: 500
exporters:
awsxray:
region: us-east-1
role_arn: arn:aws:iam::...
logging:
loglevel: warn
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [awsxray]
```
Had to drop the `logging` exporter to prod because verbosity was killing the pod. X-Ray integration is the only way to keep storage costs semi-sane, but then you're locked into AWS for querying.
Biggest pitfall: everyone assumes it's "set and forget." It's not. It's a full-time infrastructure component that needs its own monitoring and scaling. The value is there, but the hype undersells the ops burden.
-- old school
-- old school
Yeah, the collector config is where the real fun begins, isn't it? We hit a wall with the memory_limiter too, but for us it was the batch processor. Had to tune it almost hourly at one point because the span volume from our live chat evaluations was so spiky. Ended up with a 2-second timeout and a much smaller batch size than yours, otherwise the queue just ate everything. The "toy" CloudFormation is spot on - I spent a whole weekend just rebuilding it with proper private subnets and service-linked roles. That default setup would give any security person a heart attack
Oh man, you mentioning the batch processor gave me flashbacks. We had the exact same spiky traffic pattern from our support chatbot tests, and tuning that batch size felt like trying to balance a wobbly table. We actually ended up on a 5-second timeout, but we had to really crank up the `send_batch_size` limit. The queue eating everything is such a perfect way to describe it, silent data loss is the worst.
And that whole weekend on the CloudFormation... been there! I think the real joke is that the "toy" setup probably works fine for a demo with one EC2 instance, so that's what they ship. But for any real deployment, you're absolutely right, you're rebuilding it from the ground up. Did you set up a VPC endpoint for DynamoDB too, or are you letting it go public? I went back and forth on that for a while.
DynamoDB costs are the silent killer. Even with TTL, we hit provisioning issues. Switched to on-demand and saved 40% but still had to archive hourly.
Your collector config is missing the attributes processor. Strip everything at the source with something like:
```yaml
processors:
attributes/delete:
actions:
- key: some.internal.tag
action: delete
```
Plop it before the batch processor. Cuts your span size way down, which helps with that batch queue problem.
YAML all the things.
Your point about the collector config ballooning is exactly right. We found the default memory allocation for the heap was completely inadequate for high span volume, leading to constant GC pauses that would back up the pipeline. We had to move beyond the basic `limit_mib` and start using the `spike` and `ballast` settings in the memory_limiter to handle bursts from concurrent model evaluations.
Regarding the DynamoDB costs, moving to on-demand was our first step as well. The real architectural shift that curbed costs was implementing a two-tiered storage strategy. We use DynamoDB only for the hot trace data from the last 48 hours, which LangSmith's UI queries against. Everything else gets transformed and written as Parquet files to S3 via a Firehose delivery stream, which Athena can then query. This cut our DynamoDB table size by over 90%.
The CloudFormation point can't be overstated. Beyond VPC and IAM, the default provides no consideration for collector auto-scaling. We ended up replacing it entirely, deploying the collector as a sidecar in our EKS clusters with Horizontal Pod Autoscaler rules based on OTLP receiver queue size. The provided stack is indeed a starting point that assumes a monolithic, perimeter-based deployment model, which falls apart at any real scale.
Single source of truth is a myth.