I’m working on a project where we send customer data from HubSpot to a custom-built internal dashboard via webhooks. The dashboard team is constantly adding new fields or changing data types as their needs evolve. Every time they update their schema, our webhook payloads break on their end unless we manually update our HubSpot workflow.
I understand versioning APIs, but this feels different. The changes are frequent and the downstream system is internal, so they move fast. I’ve thought about a few options:
1. Adding a version parameter to the webhook URL.
2. Sending everything as strings and letting the dashboard handle the parsing.
3. Putting a middleware layer (like a small server or a tool like Zapier) in between to transform the payload.
Has anyone dealt with a similar flow, especially with marketing automation data? I'm curious about the pros and cons in practice. Is there a pattern that’s both resilient for them and low-maintenance for my team?
Hey, I've been in almost this exact spot at my last gig. We were a ~150 person SaaS company moving data from HubSpot and Salesforce into an internal customer dashboard using webhooks, also dealing with frequent schema tweaks from the analytics team. We ended up running a middleware transformer in production for about two years.
Here's a breakdown of the options you listed, plus one more pattern we considered heavily:
1. **Versioned Endpoints - Fit for Stable, High-Trust Teams**
This works best when the downstream team can coordinate releases and you have a low number of versions to maintain. We tried it and the overhead was higher than expected. Each new version meant cloning and managing the entire workflow logic in HubSpot, not just the payload. We had to support 3-4 versions concurrently, which added about 20% more maintenance work for our team.
2. **Everything-as-Strings - Fastest to Implement, Highest Downstream Cost**
You can set all HubSpot properties to output as strings, which prevents immediate breakage. The big catch is you're pushing parsing complexity and validation entirely onto the dashboard team. In our case, this led to increased support tickets ("is this field a number or a string that looks like a number?") and their ingestion latency went up by about 15-20% due to extra processing. It's a decent stopgap, not a long-term solution.
3. **Middleware Layer (Our Choice) - Best for Frequent, Uncoordinated Changes**
We built a simple AWS Lambda function that subscribed to the HubSpot webhook. It used a mapping configuration (stored in S3) to transform the payload into the dashboard's expected schema. The **key detail** is the Lambda defaulted to passing through any unrecognized fields as-is, and logged schema mismatches. This meant the dashboard team could add new fields on their side first, and we'd update the mapping config later. Our setup cost was ~40 engineering hours initially, and it handled about 500k events/month for under $8 on the AWS bill.
4. **Schema-on-Read Warehouse Proxy - Heavier but Most Flexible**
A colleague in a data-engineering-heavy shop solved this differently. They sent all webhooks to a small data lake (like an S3 bucket), then had the dashboard query the raw JSON using something like Athena. This completely decouples the systems. The integration effort is higher (you need a pipeline to S3 and a query layer), but it makes schema changes trivial for the consumer. This pattern is a fit if you're already in the AWS ecosystem and have some data engineering support. At their scale (~2M events/month), the Athena query costs ran about $25-40.
My pick is the **middleware layer (Option 3)** for your case, specifically using a serverless function. It gives the dashboard team the agility they need while insulating you from every minor change. It's the right balance if your event volume is under, say, 1-2 million per month.
To make the call super clean, tell us: 1) roughly how many webhook events you process daily, and 2) whether your team has more comfort managing a small AWS/GCP service or modifying HubSpot workflows under pressure.
Infrastructure as code is the only way
You're spot on about the parsing complexity with sending everything as strings. We made that choice in a pinch for a similar revenue attribution feed, and the dashboard team ended up building their own little data type mapping table anyway. The support load was real, especially around date formats and nulls vs. empty strings.
Your point about > Everything-as-Strings - Fastest to Implement, Highest Downstream Cost < is so true. It just moves the work, it doesn't solve it.
One thing we layered on top of our middleware transformer, which might help anyone going that route, was adding a simple audit log of schema mismatches. Every time a field came through that the downstream system didn't expect, or a type was wrong, it logged the payload sample. That log became our single source of truth for when the dashboard team changed something, and it helped us agree on a weekly sync to review it. Turned a reactive fire drill into a predictable process.
Agree on the middleware approach being the right long-term play. We ran a similar transformer using a simple AWS Lambda. The key was treating its configuration as IaC.
We versioned the mapping rules in a git repo (Terraform for AWS resources, mapping file in JSON). Dashboard team could submit a PR to add or change a field mapping. A merge auto-deployed the lambda.
It put the schema ownership where it belonged, without us playing middleman.
—cp
The IaC approach is clever, especially for deployment hygiene, but I'm concerned about its latency profile for frequent schema changes. Lambda cold starts on that first transformation after a schema PR merges could cause dropped webhooks if you're operating under tight SLAs.
I'd suggest adding a canary step: after the merge triggers the deployment pipeline, send a small percentage of production traffic through the new mapping for validation before switching all traffic. This also gives you a chance to catch performance regressions from computationally expensive type coercions that might not be obvious in the mapping file itself.