We ran the Claw migration with a parallel shadow run for six weeks before cutting over. This isn't just "test in staging"—it's about running the new agent against *real* production events without affecting downstream systems, and comparing the outputs byte-for-byte with the legacy Marketo flows.
The core pattern: every time our webhook service received a user event for Marketo, it was also duplicated to a new Kafka topic (`user_events.claw_shadow`). A separate Airflow DAG consumed this topic, ran the Claw logic, and wrote its proposed output to a dedicated Snowflake shadow schema. The existing Marketo output continued to its production tables as usual.
Key pieces of the setup:
* **Event Duplication:** Done at the router level. Simple, but critical.
```python
# In the webhook handler
produce_to_kafka('user_events.prod', event) # Existing Marketo feed
produce_to_kafka('user_events.claw_shadow', event) # Shadow feed
```
* **Isolated Execution:** The shadow DAG had its own connection IDs, schema (`analytics_shadow`), and compute warehouse. Zero shared resources with prod.
* **Validation Table:** A daily reconciliation job compared records between the Marketo output table and the Claw shadow table, flagging mismatches on:
* Total event counts
* Specific transformed field values (e.g., email campaign ID mapping)
* Timeliness (processing latency)
We found three categories of issues:
1. **Data mapping errors:** Our new agent had a different default for `null` country codes.
2. **Timing bugs:** Marketo's API had a quirky delay we hadn't replicated, causing sequence problems.
3. **Volume mismatches:** Some late-arriving data was being handled differently.
Without the shadow run, these would have been production outages. The side benefit was that the comparison metrics built confidence with the business team—they could see the new system was matching and then exceeding parity.
Biggest lesson: invest in the comparison framework first. If you can't automatically validate the shadow output against the old system, you're just running another untested pipeline.
garbage in, garbage out