We needed to connect our data warehouse to HubSpot and Salesforce simultaneously for customer journey analysis. The vendor hype was all about "real-time sync" and "unified data." I decided to test the actual data throughput and latency.
I built a pipeline to stream 2 million updated contact records from each CRM to a ClickHouse table, measuring lag and observing pipeline stability. Here are the key metrics from the 7-day run:
**Throughput (records/sec, sustained peak):**
* Custom Salesforce Bulk API sync: ~1,850
* HubSpot Events API (via webhook): ~950
* Third-party ETL platform (vendor named omitted): ~420
**Observed Latency (P95, event generation to queryable in DW):**
* Salesforce: 8.7 seconds
* HubSpot: 22.4 seconds
* Third-party ETL: 4.2 **minutes**
**The bottleneck was never the data warehouse.** It was always the API quotas, connector logic, and intermediate queuing. The vendor solution added significant overhead. My direct sync code:
```sql
-- ClickHouse table for event stream
CREATE TABLE crm_events
(
record_id String,
crm_source LowCardinality(String),
event_type String,
event_data JSON,
ingested DateTime64(3, 'UTC') DEFAULT now64()
) ENGINE = MergeTree
ORDER BY (crm_source, record_id, ingested);
```
The takeaway: For high-volume CRM data, you must bypass the middleware if latency and throughput are critical. The "real-time" claims only hold if you control the ingestion logic and respect the actual API limits.
Numbers don't lie.
This is the kind of data-driven breakdown we need more of. Thanks for sharing.
Your point about the bottleneck shifting from the warehouse to the API layer and queuing is critical. I've seen teams burn weeks optimizing database writes only to find their real constraint is the vendor's API call latency or the concurrency limits in the connector's transformation logic. Your custom sync likely avoided those queues by handling backoff and batch sizing directly.
The 4.2-minute P95 for the third-party platform is telling. That overhead often comes from their universal "adaptor" pattern, where every action passes through multiple internal staging layers for compatibility. It adds resilience for simple use cases but becomes a tax on throughput for high-volume streams like yours.