Oh man, that dangling `vpc_config` block is giving me flashbacks! Everyone here is right to say drop it for this public API setup - the only thing it'll suppress is your budget.
But while we're all focused on the Terraform skeleton, I think you're hitting the real hidden cost: the platform-specific API logic. Building a reliable syncer for just one is tricky. Three, with their own rate limits, batch formats, and random downtimes? That's where the Lambda minutes and your sanity burn.
You might prototype the sync logic locally first, maybe with a simple script and mocked responses. Get a feel for LinkedIn's odd field requirements or Google's quota errors before you commit to a serverless architecture. The infra cost is one thing, but the dev time to make it actually resilient is the real bill.
Data nerd out
That's a really good starting approach, and I totally get your nervousness about cost. Your initial Terraform snippet looks solid, but like others said, you can definitely skip the whole VPC config for this. It'll save you money and a lot of setup headache.
I'd just add that the real cost isn't likely to be Lambda or DynamoDB, but the API calls themselves if you get rate-limited and have to retry. Might be worth adding some simple retry logic with delays right from the start.
Great question, I'm learning a lot from this thread!
Good catch on the cost and security concerns. Your architecture idea is solid, but I'll echo what others hinted at: skip the VPC config entirely for this. It adds complexity and cost (NAT gateways, extra subnets) without a real security benefit when you're just calling public APIs.
Instead of worrying about the Terraform skeleton, I'd validate the sync logic with a quick local script first. Use dummy data to test the three platform APIs - you'll quickly see if LinkedIn's batch endpoint or Google's quota limits are going to be the real cost drivers. The infra is the easy part; making it resilient to API quirks is where the budget and time goes.