After a six-month evaluation of Relevance AI for orchestrating our multi-source analytics pipeline, my team has decided to revert to a purpose-built Python script orchestrated with Prefect. While Relevance offers a compelling low-code promise, particularly for rapid prototyping, we found that the abstraction cost became significant as our data volume and transformation logic complexity grew. The primary friction points weren't in the initial setup—which was indeed fast—but in the day-to-day operational control and the subtle "bending" of the platform to fit our specific, evolving use cases.
Our core pipeline ingests semi-structured JSON from several SaaS APIs, applies a series of normalization and enrichment steps (including some custom business logic for entity resolution), and then loads the results into BigQuery with specific partitioning and clustering schemes. A secondary flow generates derived aggregates in PostgreSQL for our operational dashboard. Here is where the cracks began to show:
* **Transformation Rigidity:** While visual transformers are excellent for simple mappings, our multi-step enrichment, which required conditional logic based on combined fields from two separate source streams, became a labyrinth of nested "recipe" blocks. The mental overhead to trace data flow was higher than reading our well-commented Python functions.
* **The Connector Gap:** Although their connector list is extensive, our primary data source uses a slightly non-standard OAuth2 flow. The custom connector framework felt like building a connector *inside* their paradigm, which was more time-consuming than simply using the `requests` library in a script. We were constantly working around the connector's assumptions about rate limiting and error handling.
* **Operational Opacity:** Debugging a failed sync involved clicking through multiple layers of UI to inspect intermediate state. Logs were surfaced but not easily integrable into our central logging (Datadog). In contrast, our Python script logs structured JSON directly, giving us immediate, queryable insight into failures.
* **Cost vs. Control:** The pricing model, based on active syncs and compute minutes, became less predictable than our raw GCP costs. For the same workload, we found we were paying a premium for the abstraction layer that we were increasingly fighting against.
The final catalyst was a required change to our BigQuery load strategy. We needed to implement a merge (UPSERT) pattern instead of append-only to handle late-arriving data. Configuring this within Relevance required a specific, non-obvious combination of settings and custom SQL blocks. The equivalent in our Python pipeline was a clear, 30-line function using the BigQuery client library.
```python
# Simplified example of our script's core merge logic
def upsert_to_bigquery(table_id, records, key_columns):
# Generate temporary table
# Load records into temp table
# Execute SQL MERGE statement
# Cleanup
# Log detailed outcomes
```
The migration back wasn't trivial—it required about two developer-weeks to rebuild the orchestration, idempotency, and monitoring we had initially. However, the result is a system where:
* Every business rule is explicit in code (tracked in Git).
* We can unit-test transformation functions.
* Resource utilization (CPU, memory) is directly observable and tunable.
* The entire pipeline can be run locally for debugging.
For teams with stable, simple data models and a strong preference for avoiding code, Relevance remains a valid option. But for those where data logic is a core, competitive piece of infrastructure and where requirements are in flux, the long-term maintainability and precision of a well-architected custom script, paired with a solid orchestrator, often outweighs the initial velocity gain. The control is, unequivocally, worth the dev time.
Extract, transform, trust
I'm an SRE at a 300-person logistics tech company, where we run a multi-stage data ingestion and ML feature pipeline using a mix of Prefect Core for orchestration and custom Python transformation code, all deployed on GKE.
Our team evaluated Relevance last year during a platform consolidation phase. Here's a side-by-side breakdown based on that and our current setup.
* **Unit Cost and Scaling Model**: Relevance's per-workflow-run pricing becomes a significant line item once you move beyond prototyping. At ~500k runs/month, the projected cost was 3-4x our combined infrastructure spend for equivalent GKE pods running our Python modules. For a stable, high-volume pipeline, the marginal cost of compute is lower than the managed platform premium.
* **Operational Transparency**: Debugging a stalled Relevance workflow meant tracing through their UI's execution graph, which abstracted away retry logic and state handling. With our Prefect flows, the state is explicit in the code, and we get full pod logs in Grafana/Loki. This was crucial when diagnosing a memory leak in a pandas enrichment step - we could directly see the heap profile in the same view as the task failure.
* **Customization and Integration Effort**: While Relevance connectors are fast for standard SaaS APIs, our pipeline required a custom connector for a legacy internal gRPC service. The work to wrap that as a Relevance block felt more cumbersome than simply adding the Python client to our existing Docker image and handling retries in the function.
* **State and Schema Evolution**: Our entity resolution logic changes semi-annually. In Relevance, updating a transformation and backfilling required cloning the project and managing parallel runs, which was administratively heavy. With our code-based approach, we version the logic in Git, and Prefect handles parameterized flow runs against historical data partitions idempotently.
Given your description of multi-step enrichment with custom logic and specific BigQuery partitioning needs, I'd recommend sticking with the Python/Prefect route. The control over execution environment and cost predictability outweighs the initial development speed. To be absolutely sure, you should tell us your team's ratio of data engineers to SREs/devs, and whether your source schema changes more than quarterly.
Error budgets are for spending.