Everyone's chasing the AI analytics dragon, but our legacy Grafana dashboards still needed to serve real users. The "seamless cutover" promised by the vendor was fantasy. Here's how we kept the old system alive while the new one flailed.
We ran both systems in parallel, fed by the same source data. The trick was a dumb, robust buffer script that didn't care about trendy vector databases. It just did its job.
Key moves:
* **Dual-write from day one:** Any new data pipeline had to output to both the legacy format (flat files) and the new AI system's API. No exceptions.
* **Legacy system as source of truth:** The new AI dashboard's numbers had to match the old ones before we'd even consider a switch. They never fully did.
* **Simple canary:** A cron job compared summary metrics between systems hourly and blasted a warning to a dedicated channel if they drifted.
The glue was a bash script that handled the dual-write. Nothing fancy.
```bash
#!/bin/bash
# process_and_route.sh
# Takes incoming data, writes to legacy location, then attempts to post to new AI endpoint.
# If the new endpoint fails, legacy dashboards are unaffected.
LEGACY_PATH="/mnt/legacy_ingest/$(date +%Y%m%d).log"
NEW_ENDPOINT="https://new-ai-api.company.com/ingest"
# Write to legacy system (always works)
cat >> "$LEGACY_PATH"
# Try to send to new system, but don't break the pipe
curl --max-time 5 --silent --show-error
-X POST -H "Content-Type: application/json"
-d @- "$NEW_ENDPOINT" > /dev/null 2>&1 ||
logger -t migration "Warning: Failed to post to new AI endpoint"
```
This let the "migration team" play with their new toys without us getting paged at 3 AM. The old dashboards stayed up. When the new system finally stabilized months later, we just flipped the data source on the dashboards and turned the script off. No drama.
-- old school
-- old school