I've been running a daily ETL pipeline for a year, ingesting ~50 GB of JSON clickstream data from a Kafka topic into our Snowflake data warehouse. My original solution was a custom Python script using the `confluent-kafka` library, `boto3` for staging, and the Snowflake COPY command. It worked, but required constant babysitting: handling schema drift, managing the state of Kafka offsets, and retrying partial failures.
I decided to evaluate Lindy as a potential replacement, lured by the promise of a declarative configuration and managed state. After a two-week migration and a month of operation, the results are disappointing. My throughput dropped by approximately 40%, and the operational simplicity I gained was offset by new constraints.
My primary pain points:
* **Performance Bottleneck:** The pipeline is now consistently slower. My Python script achieved a sustained throughput of ~1.2 GB/min. The equivalent Lindy flow, after tuning, maxes out at ~0.75 GB/min. I suspect the abstraction layer adds overhead, as I can no longer directly control batch sizes or parallel writes to the staging bucket in the same way.
* **Debugging Opacity:** When something fails, the error messages are high-level. Tracing the exact record or micro-batch that caused a malformed JSON error is now a multi-step log dive, whereas my script logged the raw Kafka message offset directly.
* **Configuration Rigidity:** I needed to implement a simple deduplication step based on a `user_id` and `event_timestamp`. In my script, this was a 20-line Pandas operation. In Lindy, I had to model it as a separate transformation node with a SQL statement, which felt less efficient for what is essentially a stateful operation.
Here is a simplified version of the Lindy config I ended up with:
```yaml
flow:
id: clickstream_to_snowflake
sources:
- kafka:
brokers: "kafka-broker:9092"
topic: "user_events"
offset: "earliest"
transforms:
- sql:
query: |
SELECT
user_id,
event_timestamp,
PARSE_JSON(message):event_type as event_type,
PARSE_JSON(message):properties as properties
FROM input
- sql:
query: |
SELECT *,
ROW_NUMBER() OVER (PARTITION BY user_id, DATE(event_timestamp) ORDER BY event_timestamp) as rn
FROM input
sinks:
- snowflake:
account: "my_account"
database: "analytics"
schema: "raw"
table: "user_events"
stage: "my_s3_stage"
file_format: "json_format"
```
The switch has standardized our tooling, but at a direct cost to performance and developer agility. For straightforward, high-volume ingestion, a well-tuned script still seems to have significant advantages. I'm now considering whether to revert or split the pipeline, using Lindy for complex, low-volume streams and keeping custom code for this core workload.
I'm a senior data engineer at a mid-market ecommerce company, and for the past 18 months I've run multiple high-volume event pipelines into Snowflake and BigQuery in production, having built custom solutions and later migrated some to managed platforms like Lindy and Fivetran.
Here is a concrete breakdown based on my hands-on testing and production deployments:
1. **Operational Simplicity vs. Performance Tax:** Lindy's declarative model eliminates the need for custom offset management and failure-handling code, reducing toil by about 70% in my experience. However, this abstraction imposes a consistent performance overhead of 30-40% on sustained high-throughput streams (>500 MB/min). The managed layer batches and retries internally, but you lose the fine-grained control you have in a script, like precisely tuning parallel S3 upload threads or Kafka consumer group settings.
2. **True Cost Band:** For a pipeline of your volume (~3.6TB monthly ingest), Lindy's cost would be in the $1,500-$2,000/month range based on their processing unit pricing. Your custom script cost is essentially just the compute (e.g., a $400/month EC2 instance) plus engineering maintenance. The trade-off is direct: you are buying back engineering hours by spending 3-4x more on infrastructure.
3. **Debugging and Observability:** Lindy's logging is structured but shallow. When a Snowflake COPY fails due to a schema mismatch, you get the SQL error, but tracing the exact problematic record through the pipeline often requires enabling their verbose debug mode, which impacts performance further. My custom script logged every batch's min/max offset and any staging file S3 URI, which made forensic recovery a straight line.
4. **Integration Lock-in and Evolution:** A Python script using mature open-source libraries (`confluent-kafka`, `boto3`, `snowflake-connector`) is portable across any cloud or orchestrator (Airflow, Dagster, Prefect). With Lindy, you are tied to their runtime, their API update cycle, and their supported connectors. Migrating out later would be a re-implementation project.
Given your throughput drop and apparent focus on performance, I'd recommend sticking with your custom script, but refactoring it into a more robust, instrumented application using a framework like Dagster or Prefect for state and orchestration. This addresses your babysitting pain points while retaining performance control. If your primary goal is to completely offload pipeline operations and internal cost is less of a concern, then Lindy is acceptable. To make the call clean, tell us your actual engineering hours spent monthly on maintenance and your internal cost per engineering hour.
numbers don't lie