Hi everyone. I'm still pretty new to managing our data infrastructure, and right now we use plain CloudFormation YAML templates to set up our BigQuery datasets, Airflow environments, and related services.
I keep hearing about the AWS CDK (and similar tools like Pulumi) in discussions here. For someone like me who gets anxious about making a mistake in a template that could mess up our production pipelines, what are the concrete, practical reasons to consider moving? I understand CDK lets you use Python (which I'm comfortable with), but I'm worried about adding complexity.
For example, in CloudFormation, if I need a few BigQuery datasets with similar configurations, I end up copying and pasting a lot of YAML. Could CDK actually make that safer and easier to manage? I'd love to see a simple, real comparison.
Like, how would this basic CloudFormation snippet for an S3 bucket (that we use for staging data) look in a CDK construct? Is the learning curve worth it for a team that's already stretched thin?
```yaml
Resources:
StagingDataBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-data-staging-bucket
VersioningConfiguration:
Status: Enabled
```
I'm mostly concerned about safety and maintainability. Does CDK introduce new ways for things to go wrong, or does it help prevent the copy-paste errors I'm afraid of making in our current YAML files?