Hey everyone! 👋 I've been diving into the world of Infrastructure as Code over the last few months, starting with Terraform (HCL) and some CloudFormation. It's been great for defining basic resources!
But everywhere I look in the "IaC Migration Stories" forum, I see people talking about moving *to* tools like Pulumi or CDK that use "real" programming languages (Python, TypeScript, Go, etc.). As someone who uses Python and SQL daily for analytics, I'm super curious but also a bit confused.
From my newbie perspective, HCL/CloudFormation templates get the job done. So I'm hoping for a down-to-earth, ELI5-style breakdown.
What are the *actual, concrete* day-to-day benefits? I'm thinking about things like:
* Does it just mean you can use `for-loops` and `if-else` more easily, or is it bigger than that?
* How does it help with managing larger, more complex sets of resources?
* Does it make creating reusable modules/components a lot simpler?
Basically, when you migrated, what was the "aha!" moment where using a general-purpose language felt clearly better than a dedicated DSL? I'd love any specific examples from your projects, especially if they relate to data pipelines or analytics infrastructure!
Great question! Since you already use Python, you'll feel right at home. The big "aha" for me was less about `if/else` and more about composition and abstraction.
You mentioned data pipelines. Imagine you need ten similar S3 buckets for different data stages. In HCL, you might copy/paste or use a clunky `for_each` with a map built elsewhere. In Python (with Pulumi, for example), it's just:
```python
for stage in ['raw', 'staged', 'curated']:
bucket = s3.Bucket(f'project-data-{stage}', ...)
```
But the real win is reusing logic. You can make a function that returns a configured bucket, a network module that's a proper class, or import any Python library to, say, generate names or pull config from a database. Your IaC can directly use your team's existing helper functions.
It makes complex, multi-service setups feel like building with LEGO instead of carving each block from a rock. You still have to manage state, but the authoring experience becomes so much more fluid.
it's always an API issue