Has anyone tried using Hightouch syncs for the historical backfill part?

I've done exactly this, and it works, but your pragmatism is warranted. The main gotcha isn't the sync itself, it's cost and orchestration.

> How did you handle timestamp fidelity?
You need to be meticulous about your SELECT query. Cast your timestamp field explicitly in your warehouse view and ensure it's in the ISO format RudderStack expects. Don't rely on Hightouch's default mapping. Do a small sample sync first and verify the event order in the destination.

> rate-limiting or performance issues
You will hit API throttling if you just fire off a million rows. You need to segment the syncs, maybe by date range or user cohort. Use Hightouch's sync scheduling to add delays between batches. The performance bottleneck is usually the destination API, not Hightouch.

On data type mapping, we had issues with nested JSON objects in Snowflake. Hightouch serialized them as strings, which RudderStack read as a string, not an object. You'll likely need to use `PARSE_JSON` or similar in your SQL to pre-format those fields.

The real question you should ask is whether this is cheaper than a one-time script, considering Hightouch's compute costs for a massive, slow-running sync. It's cleaner, but not always the most cost-effective.

100% on the JSON serialization issue. We hit the same thing with BigQuery structs. Hightouch just flattens them into strings unless you pre-process.

One more cost factor to add: watch out for row updates if you have to re-run a batch. If your sync isn't append-only, a re-sync can double your bill for that segment real quick. Learned that the hard way.

That's a really good callout on the re-sync cost, hadn't considered that. If a batch fails and you have to restart, you're essentially paying twice for the same data if it's not a pure append.

Makes me think you'd almost want to create a separate, immutable staging table in the warehouse just for the backfill, so your Hightouch source is always a fixed set of rows. No risk of accidentally updating.

What did you end up doing to pre-process those BigQuery structs? Did you have to write a transformation in the warehouse view, or handle it somewhere else?

Tried it once. It can work, but you're adding a new failure layer for a time-sensitive, one-time job. Your main risk is the complexity you're trying to avoid.

> How did you handle timestamp fidelity?
You don't. Hightouch's job queue does. You need to verify the order post-sync with a sample, and your validation script is now mandatory. Any drift means re-running the batch.

For a million records, a purpose-built script or temporary Airflow DAG is less abstract. You control retries and observability directly. Using Hightouch for this is solving a load problem by introducing a orchestration and cost opacity problem. Are your engineers more skilled in debugging Hightouch syncs or Python scripts?

The real question is why you'd risk a third-party sync's reliability and cost uncertainty for a critical migration step.

> "cost opacity problem"

Exactly. With a script, the compute costs hit your AWS bill where you can track them. Hightouch charges per row, and if a batch fails, you're paying twice for the same data with zero visibility until the invoice arrives. So much for avoiding complexity.

You're just swapping one maintenance headache for a cost surprise. And good luck debugging their job queue at 2 AM when your migration window is closing.

I've managed two such migrations using Hightouch for backfill, and the pragmatic concerns here are correct. You've identified the critical constraints: timeline integrity and schema mapping.

On timestamp fidelity, the core issue is that Hightouch processes jobs in a queue, not strictly in event-time order. Your mitigation is to segment your syncs by a monotonic key like a date partition or a sequential user ID range, and never allow these batches to overlap. Validate the order in the destination for each batch before proceeding. This turns the problem into a deterministic, verifiable process rather than hoping for sequential execution.

For your Snowflake schema mapping, the surprise is often with variant columns containing nested JSON. Hightouch will convert these to strings unless you explicitly use `TO_JSON` or `PARSE_JSON` in your source query to force proper serialization. You must build a view that pre-transforms these columns into the exact string representation RudderStack expects. This adds a layer of warehouse logic, but it's far more reliable than post-sync transformation.

The performance bottleneck is indeed the destination API. You'll need to calculate RudderStack's throughput limits and set your Hightouch sync row batch size and delay intervals accordingly. It becomes an exercise in tuning a third-party system for a bulk load it wasn't primarily designed for.

I actually used this exact approach for our Segment to RudderStack migration last quarter, and it did work, but with significant prep. On your specific points:

The timestamp fidelity requires a rigid approach. You must create a staging view in Snowflake where your `timestamp` field is explicitly cast and formatted (like `to_varchar(your_timestamp, 'YYYY-MM-DDTHH:MI:SS.FF3Z')`). Then, test with a small date-range sync and verify the order in RudderStack's raw data explorer before any full batch runs.

For the million records, you'll absolutely need to chunk it. We split by user_id ranges and used Hightouch's sync scheduling to add 5-minute delays between batches to respect RudderStack's API limits. The surprise for us wasn't performance, but the silent stringification of Snowflake VARIANT columns. All our nested event properties arrived as JSON strings in RudderStack, which broke our downstream models until we added a JSON_PARSE step in our warehouse view.

Honestly, the biggest lesson was that the "clean solution" feeling disappears once you're managing 30 separate Hightouch sync jobs for one backfill. It becomes its own orchestration puzzle.

The stringification of Snowflake VARIANT data is a critical schema mapping surprise. It doesn't just flatten nested JSON into strings; it strips the type metadata, making it difficult for RudderStack to correctly interpret arrays or nested objects on ingest. You'll need to pre-process that column in your staging view using `PARSE_JSON(to_varchar(variant_column))` to force a clean JSON string Hightouch can pass through.

Regarding rate-limiting for a million records, chunking by user_id or date is necessary but insufficient. You must also calculate the rows-per-second against RudderStack's documented API limits and configure Hightouch's sync speed accordingly, as its default might still overwhelm the destination. This isn't a set-and-forget task; it requires active monitoring of the first few batches to tune the delay.

The cost argument against scripts is valid, but the counterpoint is operational debt. A one-time script, even if messy, is deleted after validation. A Hightouch sync model becomes a permanent fixture in your stack that someone will inevitably try to reuse or modify later for a different purpose, creating a hidden maintenance burden.

Good points on the schema mapping. That Snowflake VARIANT gotcha is real. We had to use `TRY_PARSE_JSON` in our view to avoid errors on malformed fields, but it did the trick.

On the performance side, chunking by date wasn't enough for us either. RudderStack's API started returning 429s until we added a concurrency limit directly in the Hightouch sync settings, not just the batch scheduling. Definitely test with a non-trivial batch size first.

It's a valid path, but as others said, you're trading script maintenance for active monitoring and cost risk. For a clean migration window, you might find that extra vigilance more stressful than writing a one-off Python loader.