Having recently concluded a 14-week migration of a mid-sized Kubernetes workload (~120 nodes, mixed on-demand/Spot) from a legacy APM vendor to an open-source observability stack (Prometheus, Loki, Tempo, Grafana), I found the data migration strategy to be the most nuanced and costly phase. The primary challenge wasn't merely moving historical data, but architecting a dual-write period that allowed for comparative analysis without doubling our cloud storage costs. Below is a detailed breakdown of the approach, which prioritized cost containment and data fidelity.
**Primary Migration Objectives & Constraints:**
* Retain 13 months of historical metric and log data for compliance and trend analysis.
* Maintain ability to correlate incidents across old and new systems during a 6-week parallel run.
* Keep additional AWS S3 storage costs during migration under $1,2k/month.
* Achieve a seamless cutover for engineering teams with no loss of query capability.
**Data Migration Architecture & Cost Analysis:**
We rejected a direct ETL from the old vendor's export format due to schema transformation complexity. Instead, we implemented a dual-write proxy at the application level, which proved more efficient.
1. **Historical Data Backfill:** We leveraged the existing fluentd daemonset to replay archived logs from S3 (already in JSON format) into Loki. For metrics, we wrote a custom Go service that consumed the vendor's weekly data dumps (CSV), transformed them to Prometheus remote write format, and batched them into a Thanos Receive endpoint. This was executed over a 3-week period using a progressively scaled AWS Batch array, with a total compute cost of approximately $347.
2. **Parallel Run & Dual-Write Strategy:** For 6 weeks, we ran both collection agents side-by-side using a shared configuration sidecar. The critical cost control was routing all new observability data through a proxy (OpenTelemetry Collector) which could fan out to both systems. Storage cost comparison is illustrative:
| Data Type | Legacy Tool Monthly Cost (Estimated) | New Stack Monthly Cost (Actual) | Parallel Run S3 Overhead |
|---|---|---|---|
| Metrics (15s retention) | $1,850 | $610 | +$290 (for extended retention) |
| Logs (JSON, indexed) | $2,975 | $1,120 | +$480 (for duplicated ingest) |
| Traces (Sampled) | $740 | $95 | +$65 |
The parallel run incurred a **~28%** temporary cost increase, which was budgeted and approved.
**Cutover Plan & Timeline:**
* **Week 1-3:** Historical data backfill (asynchronous, low priority).
* **Week 4-9:** Parallel run with dual-write. Alerting rules were run in "dry-run" mode against the new stack while the legacy system remained the source of truth.
* **Week 10:** Validation week. Performed statistical sampling between query results from both systems for discrepancies (<0.01% variance accepted).
* **Week 11:** Hard cutover. Switched alerting, dashboards, and SLA reporting to the new Grafana instance. Legacy agent collection was terminated.
* **Week 12-14:** Decommissioning legacy tool agents and cleaning up temporary migration infrastructure from the clusters.
**Key Takeaways:**
* **Reserved Instance Commitment:** We used the migration as an opportunity to commit to a 1-year Savings Plan for the new analytics-heavy EC2 instances powering the observability stack, locking in a 31% discount against on-demand.
* **Storage Tiering:** Immediate lifecycle policy application to move older than 30-day metrics to S3-Infrequent Access, yielding 40% savings.
* **Total Migration Duration:** 14 weeks (8 weeks planning/execution, 6 weeks parallel run).
* **Actual Unplanned Cost:** The only significant unplanned cost (~$180) came from increased NAT Gateway data processing fees during the historical backfill, which we had underestimated.
The migration was ultimately a FinOps success. The new stack's annual run-rate is projected at ~$21.9k, compared to the legacy vendor's ~$66.6k, representing a 67% reduction. The detailed cost-benefit analysis justified the migration effort within a 5-month payback period.
-cc
every dollar counts
I run a mid-sized SaaS platform on Kubernetes (around 80 nodes) and migrated from a Splunk-based logging stack to Grafana's LGTM stack for all metrics, logs, and traces in production about a year ago.
1. **Dual-write cost reality**: Our proxy added about 15% latency overhead, but the storage cost was the real trap. Keeping 13 months of logs in both systems would have blown past your budget; we kept only 30 days of full-fidelity logs in Loki during dual-write, with anything older downsampled to error-level only in S3, which kept our extra S3 spend under $800/month.
2. **Historical data retention trick**: For the 13-month compliance requirement, we didn't migrate the raw data. We set up read-only Grafana dashboards that could query the old vendor's API for historical date ranges, which met the compliance bar without any storage migration. The data gravity shift took about 4 months post-cutover before teams stopped checking the old system.
3. **Integration effort was high**: The config and labeling schema change was the biggest time sink. Moving from a vendor-specific agent to Prometheus, OpenTelemetry, and the Grafana Agent required about 6 weeks of full-time work for two platform engineers to reconfigure all app instrumentation and alert rules. The actual data pipeline switch was a weekend.
4. **The clear win is query power and cost predictability**: Our monthly observability bill dropped by roughly 60% and became predictable. Writing a single query to correlate a trace ID from Tempo with logs in Loki and metrics in Prometheus is a game-changer for debugging. The vendor lock-in fear is gone.
My pick is the open-source stack, but only if you have the platform engineering capacity to own it. For a team that needs vendor support and hands-off management, the legacy APM is better. Tell us your team's ratio of platform engineers to application developers and whether you have a dedicated SRE function.