Showcase: the custom Cloud Function we wrote to batch-transform old events.

Our migration from Segment to RudderStack required us to transform nearly three years of historical JSON event data, stored in BigQuery, to align with RudderStack's expected schema and typing. The primary challenge was the volume—approximately 2.1 billion events—and the need for strict idempotency and manageable batch processing. A simple BigQuery SQL transformation was insufficient due to complex nested field remapping and the need for precise error logging per event.

We developed a Cloud Function in Go to serve as a highly controllable, batch-oriented transformation layer. It was triggered by Pub/Sub messages containing pointers to specific date-partitioned tables, allowing us to parallelize the migration by date and maintain a clear checkpoint system. The core logic involved:

* Reading a configurable batch of rows from the source BigQuery partition.
* Applying a series of transformation rules defined in a separate mapping YAML file (e.g., `user.id` -> `userId`, coercing string timestamps to ISO 8601, flattening certain nested contexts).
* Writing successfully transformed events to a RudderStack-backed GCS bucket in the `batch` format.
* Writing any rows with unresolvable schema errors, with full error context, to a separate BigQuery error table for later analysis.

The function's configuration allowed us to tune throughput and cost by adjusting batch size and concurrent invocation limits.

```go
// Core transformation loop pseudocode
for _, oldEvent := range bigQueryRows {
transformed, err := applySchemaMap(oldEvent, config.Mappings)
if err != nil {
writeToErrorTable(oldEvent, err, processingDate)
continue
}
batchBuffer = append(batchBuffer, transformed)
if len(batchBuffer) >= batchSize {
uploadToGCS(batchBuffer, targetPath)
batchBuffer = nil
}
}
```

Key performance metrics from our final run:
* Average processing rate: ~28,000 events/second
* Total function runtime (spread across parallel invocations): ~21 hours
* Error rate: 0.012% of total events, primarily from legacy null values in required fields
* Cost: Under $45 for Cloud Function and BigQuery processing combined

The most significant benefit was the detailed error logging, which enabled us to retrospectively fix and backfill the problematic events with a targeted second pass. This approach proved more reliable and auditable than a monolithic ETL job. The mapping YAML remains as documentation of the schema translation, which has been useful for onboarding new engineers to the event structure.

Regards, luke