Hot take: You don't need a full middleware platform for most of these integrations.

The prevailing narrative in our data architecture circles is that any system-to-system integration requires a dedicated middleware layer—a Kafka, a RabbitMQ, or a cloud-native iPaaS. While these platforms are powerful for complex, high-volume event streaming, I've observed that a significant portion of proposed use cases are better served by a simpler, more maintainable approach: treating your data warehouse as the integration hub and using orchestration tools you already own.

Consider the common pattern: syncing data from a SaaS platform (like a helpdesk tool) to a CRM, or from a marketing platform to a customer data platform. Instead of standing up a net-new service, you can often implement a robust integration using a scheduled dbt model or a Python script within your existing Airflow/Dagster/Prefect orchestration. The pattern is straightforward:

1. **Extract:** Use the source system's API (via a simple Python call or a Singer tap) to land raw JSON data into a `raw` schema in your warehouse.
2. **Load & Transform:** Model this data in dbt to produce a clean, typed, business-ready view.
3. **Push:** Write an operational query or a small script that selects from the transformed model and pushes the necessary records to the destination API.

This approach centralizes your logic, provides full auditability, and leverages your team's existing SQL and data pipeline expertise. For example, a daily sync of new support tickets from Zendesk to Salesforce might culminate in a simple operational script run by your orchestrator:

```sql
-- In dbt: models/integrations/zendesk_to_sf_opportunities.sql
with new_tickets as (
select
ticket_id,
subject,
requester_email,
created_at,
'zendesk' as source_system
from {{ ref('stg_zendesk_tickets') }}
where created_at >= current_date - interval '1 day'
and status = 'new'
)
select * from new_tickets
```

```python
# In your orchestration task (e.g., Airflow PythonOperator)
def push_tickets_to_sf(**context):
new_tickets_df = context['ti'].xcom_pull(task_ids='query_new_tickets') # or query warehouse directly
for _, ticket in new_tickets_df.iterrows():
sf_client.Opportunity.create({
'Name': ticket['subject'],
'CloseDate': '2024-12-31',
# ... map other fields
})
```

The benefits are tangible:
* **Reduced Complexity:** No new infrastructure to monitor, secure, or pay for.
* **Enhanced Data Quality:** The integration logic is subject to the same testing, documentation, and version control as your core data models.
* **Single Source of Truth:** The warehouse *is* the integration state, making debugging a matter of querying tables, not parsing disparate log files.

This pattern breaks down for real-time requirements or extremely high-frequency events, but for daily/hourly batch syncs—which cover the majority of business needs—it's remarkably effective. We should challenge the instinct to reach for a new tool before fully utilizing the powerful, queryable integration plane we've already built.

- dan