Hi everyone! I'm still pretty new to all this observability stuff, but my team is looking at platforms and I'm the one tasked with running the POCs. There are so many options (Datadog, New Relic, Grafana Cloud, etc.) and I'm a bit overwhelmed trying to compare them fairly.
My main fear is that we'll build something during a trial that we can't easily move away from later. I've heard horror stories about vendor lock-in, especially with custom instrumentation and dashboards.
Could you share what a "proper" POC looks like for an observability platform? I'm thinking about:
* What are the key things to actually test during the trial period? I know we need to check query speed and alerting, but what else?
* How do you structure the test data and use cases to be representative but not overly complex?
* What are the biggest "lock-in" traps to avoid when setting things up? Is it mostly about agent configuration, dashboard definitions, or something else?
* How do you handle exporting your data and configuration if you decide to switch later?
For context, our main stack is Python microservices on Kubernetes, with Snowflake for data warehousing. We're currently just using some basic logs and metrics, but want to get into traces and maybe more advanced SLOs.
Any advice from those who've been through this would be so appreciated! I've already had a few pipeline failures this month and really don't want to add a platform migration disaster to the list 😅
null
Your fear about lock-in is spot on, especially with observability. The trap isn't just the dashboards; it's the proprietary agents, the custom query syntax, and the alert definitions.
For your Python/K8s stack, structure your POC around open standards. Use the OpenTelemetry collector to send data to every platform you're trialing. This keeps your instrumentation neutral. Test the same three critical scenarios: a latency spike, an error rate increase, and a pod eviction. Time how long it takes from incident to actionable alert in each platform. Ignore the fancy pre-built dashboards; build your own for those specific scenarios to feel out the query language.
The biggest lock-in is always the agent. If you let Datadog's agent handle tracing and metrics, migrating means replumbing everything. Stick to OTel. For exporting, before the trial ends, try to export a dashboard as code and your alert definitions as JSON. If the vendor makes that difficult, consider that a red flag. Snowflake integration is another can of worms; ask each vendor exactly how they push/pull data from it and what the egress costs look like.
Measure twice, cut once.
OpenTelemetry as a silver bullet is getting a bit tiresome. The collector's great in theory, but you're just trading one lock-in for another. Now you're locked into OTel's semantic conventions and its own set of complexities. What happens when you need a piece of data their spec doesn't cover? You're back to vendor-specific extensions anyway.
And timing "incident to alert" is a nice academic exercise, but it misses the real cost. The lock-in pain starts *after* you buy, when you need to customize something the vendor didn't anticipate. Try modifying that exported JSON alert definition six months in when their schema has changed. Good luck.
The red flag isn't just difficulty exporting, it's what the exported artifact actually is. A "dashboard as code" that's 500 lines of their proprietary DSL isn't portable, it's a souvenir.
Buyer beware.