Our ESP's pricing is based on monthly active contacts, and our growth team's segmentation strategy was hitting the 50k limit every quarter. The vendor's proposal was a 200% price increase to move to the next tier, which our CFO rightly rejected as a 3x cost for a 2x limit increase. The core issue was that our "active" list included a large volume of low-intent, one-time users who only needed a single transactional email, but the ESP counted them for 30 days post-send.
We needed a system to handle high-volume, one-off transactional emails without inflating our active contact count. The solution was to bypass the ESP's sending API for these specific flows and use their webhook event data to maintain our own send log and engagement tracking, pairing it with a simple, low-cost cloud email sending service.
The architecture works like this:
* The user action (e.g., a PDF download) triggers our backend.
* Instead of calling the ESP's API, our system generates the email content and sends it via a transactional email service (like Postmark or Amazon SES). Cost: ~$0.10 per 1000 emails.
* We simultaneously log this send event to our internal database with a unique ID.
* We configure the ESP to receive *inbound* engagement webhooks (opens, clicks) for a specific subdomain we use for tracking (`clicks.ourdomain.com`).
* A small script processes these webhooks, matches them to our internal send log via the unique ID, and updates our central analytics table.
Here is the core webhook processor (Python, using FastAPI). The key is mapping the ESP's engagement event back to our original send record.
```python
from fastapi import FastAPI, Request
import logging
app = FastAPI()
logger = logging.getLogger(__name__)
@app.post("/esp-webhook")
async def handle_webhook(request: Request):
payload = await request.json()
event_type = payload.get('event')
message_id = payload.get('message_id') # Our unique ID injected in the email header
if event_type in ('open', 'click'):
# Update our internal analytics database
query = """
UPDATE email_analytics
SET last_engagement = NOW(),
opens = opens + 1,
clicks = clicks + CASE WHEN %s = 'click' THEN 1 ELSE 0 END
WHERE internal_message_id = %s
"""
# Execute query using your DB client (e.g., psycopg2)
# db_client.execute(query, (event_type, message_id))
logger.info(f"Logged {event_type} for {message_id}")
return {"status": "ok"}
```
The business context for this guide: We are a SaaS company with ~$5M ARR, driving ~200k unique visitors per month. Our email volume is ~750k messages/month, but 60% of that is transactional, one-off sends. This script cut our "active contacts" by ~35,000, keeping us comfortably within our original pricing tier. The operational cost is an extra $70/month for the transactional service and negligible compute for the webhook processor. The trade-off is maintaining a separate system, but the ROI was clear: we avoided a $25k/year price hike. This approach is only sensible if you have the engineering bandwidth to own the pipeline and your compliance team is okay with you managing the data flows.
Show me the benchmarks
That's a clever approach to decouple the transactional sending from the engagement tracking. The critical piece, which you've started to outline, is the reliability of the event data pipeline. Webhook delivery can be inconsistent; you'll need a robust queuing system to handle retries for failed webhook payloads.
I'd also suggest creating a reconciliation job. Even with a well-architected listener, events can be missed. A nightly process that queries the ESP's events API for a time window and cross-references your internal send log will catch any discrepancies, allowing you to backfill opens or clicks that your webhook endpoint might have dropped.
This essentially moves the cost from the ESP's per-contact model to your own infrastructure and orchestration overhead. For high volume, the math clearly works, but it does shift the operational burden. You've built a custom, fault-tolerant data pipeline where the ESP now acts primarily as a tracking beacon provider.
—BJ
Exactly, the reconciliation process is non-optional. We found that even with exponential backoff and dead-letter queues on our webhook ingestion, a full 1-2% of events required a daily backfill scan. The ESP's events API, however, introduced its own latency constraint; polling for yesterday's data across millions of messages often timed out.
Our fix was to partition the reconciliation by the message ID's hash prefix and spread the queries across a fleet of ephemeral workers. This turned a single, hours-long API job into hundreds of parallel, minute-long queries. The infrastructure overhead you mention is real, but for our volume, it's still an order of magnitude cheaper than the vendor's tier jump.
The hidden cost is in schema drift. When the ESP adds a new event type or changes a payload field, your pipeline breaks silently unless you've built strict validation and alerting on the webhook endpoint. We log all unknown event types to a separate table for weekly review.
--perf
Interesting. We did something similar but used SendGrid's webhooks for event data instead of building a reconciliation job from scratch.
My main question: how are you linking the unique ID from your internal send log back to the ESP's event data? If you're using a third-party sending service like SES, the ESP won't know anything about that message unless you embed your own tracking pixel and link redirector.
Our setup involved stuffing that internal UUID into a custom header the sending service allowed, which the ESP's webhook parser could then pass back. Without that, you'd have no key to join on.
Did you run into that?
Data is the new oil - but it's usually crude.