The prevailing narrative around data migration between observability or infrastructure platforms often centers on scheduled downtime windows and disruptive cutovers. This is an unnecessary compromise. Having recently executed a migration from a legacy APM tool to an OpenTelemetry-based pipeline for a multi-terabyte dataset, I can confirm that a zero-downtime migration is not only possible but should be the default expectation for stateful services. The key is to treat the legacy and target systems as active-active endpoints during the transition period, using API scripts to dual-write and validate data integrity before switching read traffic.
The architectural pattern we implemented consisted of three distinct phases, orchestrated via a combination of Python scripts and Kubernetes CronJobs for scheduling.
**Phase 1: Historical Backfill with Throttling**
We first seeded the new system with historical data. The critical nuance here is implementing client-side rate limiting and jitter to avoid impacting the source system's production performance. We used a producer-consumer model with a persistent checkpoint store.
```python
import requests
import time
from datetime import datetime, timedelta
def backfill_with_throttling(source_url, target_url, start_date, api_key, max_rps=50):
current_date = start_date
while current_date <= datetime.utcnow():
params = {'from': current_date, 'to': current_date + timedelta(hours=1)}
response = requests.get(f"{source_url}/api/v1/metrics", params=params, headers={'Authorization': api_key})
if response.status_code == 200:
# Transform payload to OTLP format
otlp_payload = transform_to_otlp(response.json())
# Dual-write: store to checkpoint and send to new system
requests.post(f"{target_url}/v1/metrics", json=otlp_payload)
# Respect source system rate limit
time.sleep(1 / max_rps + random.uniform(0, 0.1)) # Jitter
current_date += timedelta(hours=1)
save_checkpoint(current_date) # Persistent checkpoint for resume capability
```
**Phase 2: Dual-Writing Live Traffic**
Once the historical data was within a tolerable lag window (e.g., 24 hours), we reconfigured our application's export configuration to send identical data to both the legacy and new systems concurrently. This was achieved by deploying a sidecar agent or, in our case, modifying the OpenTelemetry Collector configuration to have two identical exporters.
```yaml
# otel-collector-config.yaml
exporters:
logging:
loglevel: debug
prometheusremotewrite:
endpoint: " https://legacy-system.com/api/v1/write"
tls:
insecure: false
otlphttp:
endpoint: " https://new-system.com/v1/otlp"
tls:
insecure: false
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheusremotewrite, otlphttp, logging] # Dual write
```
**Phase 3: Validation and Cutover**
The dual-write phase ran for a full business cycle (7 days) to ensure data parity under all load conditions. We ran daily reconciliation jobs comparing aggregate metrics (count, sum, min, max) for cardinality groups between the two systems, flagging any discrepancy beyond a 0.1% threshold. The final cutover was then a simple configuration change: removing the legacy exporter from the pipeline and decommissioning the backfill scripts.
**Key Metrics & Timeline:**
* **Data Volume:** ~8 TB of metric time-series data.
* **Backfill Duration:** 14 days (throttled to 60 RPS to avoid source degradation).
* **Dual-Write Phase:** 7 days.
* **Active Scripting Work:** ~25 person-days for development, validation, and monitoring.
* **Observed Discrepancy:** < 0.05% during dual-write, attributable to network latency and millisecond-level timestamp differences.
The primary advantage of this approach is risk mitigation. At any point before the final cutover, you can revert to the legacy system as the single source of truth by simply disabling the new exporter. The operational cost is the additional egress/ingress fees during dual-write, which is almost always negligible compared to the business cost of an outage or data loss incident.
No free lunch in cloud.