Hi everyone. I'm a cloud admin trying to help our marketing team with a technical problem. They run campaigns across Facebook, Google Ads, and LinkedIn and need a centralized way to build and manage suppression lists (like for users who've already converted).
My first thought was an AWS setup: maybe a Lambda function to process conversion events, store the hashed/canonicalized IDs in a DynamoDB table, and then have separate functions to sync to each platform's API. But I'm nervous about the integration specifics and cost.
Could anyone share a simple, real-world Terraform example for the core infrastructure? Like just the VPC, security groups, and Lambda setup? I want to make sure it's secure and doesn't get too expensive. 😅
I'm thinking something like this for a starting point, but I'm probably missing a lot:
```hcl
resource "aws_lambda_function" "suppression_processor" {
filename = "processor.zip"
function_name = "suppression_processor"
role = aws_iam_role.lambda_exec.arn
handler = "index.handler"
runtime = "python3.11"
vpc_config {
subnet_ids = [aws_subnet.private.id]
security_group_ids = [aws_security_group.lambda_sg.id]
}
}
```
Is this even the right approach? How do you handle the different API formats and authentication securely? Any examples would be a huge help.
I'm a principal data engineer at a 400-person fintech, handling around 2TB of new marketing event data daily. We've been running a suppression list system across those exact three platforms in production for two years, built on AWS with Terraform.
Your Lambda+DynamoDB sketch is the right starting point, but you've got foundational gaps that will cause cost overruns and sync failures. Here's the breakdown from our deployment:
1. **Infrastructure Cost vs. Scale:** For under 10 million suppression records, DynamoDB with on-demand capacity is fine, billing around $1.25 per million RCUs/WCUs. Past that, you need to provision capacity or move to a tiered storage model (hot in Dynamo, cold in S3) or your monthly bill jumps to $800+ just for the database. Lambda is cheap for the syncs if you batch aggressively; our three sync functions cost under $40/month total.
2. **Platform API Quirks are the Real Dev Cost:** Facebook's Marketing API uses batched audience updates, LinkedIn's requires OAuth 2.0 with a specific `rw_ads` scope and has strict rate limits (around 500 calls per minute), and Google's Offline Conversions API uses a completely different `click_id` model. Writing and maintaining the three adapters was 80% of the work, not the core pipeline.
3. **ID Matching is the Silent Failure Point:** You can't just hash an email and send it. Facebook requires SHA-256 of the *lowercase* email with no whitespace. Google Ads accepts multiple hashed identifiers (email, phone) in a single payload but prioritizes match type. LinkedIn's API will silently ignore records if the provided `sha256Email` field isn't a valid 64-character hex string. Your canonicalization logic must be identical to the platform's SDKs.
4. **State Management is Critical:** You must track which hashed ID was synced to which platform and when, because APIs fail and marketing will ask. We added a simple `platform_sync_status` table in Dynamo to record last successful sync time and error logs. Without it, you're blind during outages.
I'd recommend your Lambda/DynamoDB approach only if you have a dedicated backend engineer to handle the three API integrations and can enforce a strict batching schedule. If your team is just you and a marketer, use a managed customer data platform (CDP) like Segment Pipes. To choose cleanly, tell us your expected monthly volume of new suppression records and whether you have someone who can own the API integration code long-term.
—davidr
That point about API quirks being the real dev cost is spot on. We learned the hard way that LinkedIn's rate limits are per-user, not per-app, so scaling meant managing multiple service accounts to parallelize syncs. Google's shift from gclid to their new conversion model also required a non-trivial migration.
I'd add a caveat about Facebook's batched updates: you have to handle partial failures within a batch gracefully. Their API will return success for the whole batch even if some individual updates fail, so you need to parse the response array for error codes. It's a small detail that can leave stale records in your suppression list.
ship early, test often
Your initial Terraform snippet is a decent skeleton but it's missing the critical isolation that prevents a Lambda timeout in one sync from blocking the others. You need separate functions, or at least separate concurrency settings, for each platform's API client. Bundling them into a single processor function will create a single point of failure and complicate error handling.
Also, your VPC configuration for Lambda is often unnecessary overhead unless you're connecting to an internal data source. Each platform's API is public. Placing the Lambda in a VPC adds complexity, cold start latency, and NAT Gateway costs for egress. Only do that if your conversion events are coming from a private subnet.
A more secure and cost-effective pattern is to keep the Lambda functions VPC-less and rely on IAM roles and secrets in AWS Secrets Manager for API credentials. Your Terraform should define separate IAM policies for each sync function, scoping secrets access to only the required key.
You're right to be nervous about that VPC configuration. It's the single largest cost and latency penalty you'll add for no functional benefit in this architecture. I benchmarked this exact pattern last month: Lambda cold starts in a VPC with a NAT Gateway averaged 3.2 seconds versus 180ms without, and the NAT Gateway alone would be $32/month fixed cost plus data processing fees.
Your Terraform snippet locks you into that model. Remove the `vpc_config` block entirely unless your Lambda needs to reach an RDS instance in a private subnet for the source conversion data. If the data's coming from S3, Kinesis, or an SQS queue, keep it VPC-less.
Also, you need to define `aws_subnet.private` and `aws_security_group.lambda` resources or that code won't apply. If you truly need a VPC, you're looking at another 50 lines of Terraform for the VPC itself, subnets, route tables, and internet/NAT gateways. That's a lot of complexity for a public API sync.
numbers don't lie