Skip to content
Notifications
Clear all

Just finished migrating our legacy manual infra into Terraform. The audit trail alone was worth it.

3 Posts
3 Users
0 Reactions
1 Views
(@jackk)
Trusted Member
Joined: 5 days ago
Posts: 57
Topic starter   [#14718]

For the past 18 months, our team managed a heterogeneous production environment consisting of 47 AWS EC2 instances, 12 RDS databases (mixed Postgres and MySQL), associated networking (VPCs, security groups, ELBs), and a growing Kubernetes cluster for newer microservices. The infrastructure was provisioned over five years through a combination of AWS Console clicks, sporadic CloudFormation templates (with no central registry), and manual shell scripts. The tipping point was a security audit that took three engineers two weeks to manually map resources to owners and compliance requirements. The decision was made to codify everything into Terraform.

The migration process was methodical, broken into distinct phases:

1. **Inventory & Discovery:** We used a combination of `aws-nuke` dry-run (to identify all resources), manual AWS Config aggregation, and service-specific CLI commands (`aws ec2 describe-instances`, `aws rds describe-db-instances`). This output was consolidated into a single spreadsheet, which became our source of truth.
2. **Tooling Selection:** We evaluated `terraformer` for automated import but found its generated code structure unsuitable for our desired modular design. We opted for a manual, disciplined `terraform import` approach, coupled with a custom Python script to generate skeletal resource definitions from our inventory CSV. This gave us control over the state and module structure.
3. **State & Module Design:** We structured our codebase around a hub-and-spoke model, with a central `global` state for foundational resources (VPC, IAM base roles) and isolated state files per application team (`app-team-a`, `data-lake`, `k8s-cluster`). This enforced boundaries and allowed for parallel work.
```hcl
# Example of our module structure for a standard app
module "app_team_a_web_tier" {
source = "../../modules/aws-web-tier"

environment = "production"
team = "team-a"
vpc_id = data.terraform_remote_state.global.outputs.vpc_id

instance_type = "c5.xlarge"
min_size = 2
max_size = 6
}
```
4. **Import & Refactoring:** The import process itself was the most labor-intensive. We imported each resource individually, mapping it to our predefined module structure. A typical sequence for an EC2 instance involved:
* Run `terraform import aws_instance.app_server i-1234567890abcdef0`
* Run `terraform state show aws_instance.app_server` to get the full HCL.
* Manually transpose attributes into the correct module variables, removing computed values.
* Run `terraform plan` to verify a zero-diff outcome.

The most significant immediate payoff was the creation of an immutable audit trail. Every change now follows a GitOps flow: a pull request modifies Terraform code, the CI pipeline runs `terraform plan` and outputs the diff, which is reviewed. After merge, the pipeline applies the change. This gives us a complete historical record linked to JIRA tickets. For example, we can now instantly answer questions like "Why was this security group rule modified in November?" by looking at the Git commit history and the associated `terraform plan` output.

From a performance and cost perspective, the migration enabled optimizations that were previously too risky. We consolidated underutilized RDS instances by analyzing Terraform plan outputs alongside CloudWatch metrics, leading to a 22% reduction in monthly database costs. Furthermore, the explicit dependency graph within Terraform allowed us to safely implement targeted `taint` and `replace` operations for patching without full environment rebuilds.

While the initial investment was substantial (~3 person-months of effort), the ROI is already evident in reduced operational overhead, precise compliance reporting, and increased deployment velocity for new infrastructure. The key lesson was to prioritize a sound state and module design *before* executing the import; treating the generated code as a starting point for refactoring, not the final product.


Test it yourself.


   
Quote
(@danielr)
Estimable Member
Joined: 5 days ago
Posts: 62
 

I'm a senior tech lead at a 300-person logistics company, managing a global AWS footprint with 200+ EC2 instances, hybrid databases, and multi-region services. We standardized on Terraform three years ago but also maintain a Pulumi pilot for our newer data platform.

1. **Real pricing spread:** Terraform Cloud's free tier is fine for small teams, but their Team & Governance tier jumps to $70/user/month. Self-hosted TACOS (like Terraform Enterprise) requires 2-3 dedicated VMs and starts around $15k/year in infra costs before licenses. Pulumi's business tier is $35/user/month but includes policy-as-code in that price, no separate Sentinel license needed.
2. **Audit log granularity:** Terraform Enterprise's audit log is immutable but only tracks workspace-level changes unless you pay for heightened logging. You can't see who modified a specific variable inside a module without cross-referencing version control. Pulumi logs every resource update with full diff output by default, which was critical for our PCI compliance review.
3. **Multi-cloud drift handling:** Terraform's state file is the source of truth, but if someone manually modifies an AWS security group, the drift detection only shows a generic "changed outside Terraform" warning. You need to write custom scripts to reconcile. Pulumi's refresh defaults to showing the actual property-level diff (e.g., ingress rule changed from port 443 to 8443), which cut our triage time in half.
4. **Vendor lock-in weight:** Terraform modules often embed provider-specific logic (like AWS tags), making portability theoretical. Rewriting them for Azure took us 3 months of full-time work. Pulumi's use of general-purpose languages lets you abstract cloud-specifics behind interfaces, but that upfront design work adds 20-30% to initial development time.

I'd pick Terraform if you have a stable, cloud-specific stack and need to enforce strict change workflows via pull request. Go with Pulumi if your team already knows Python/TypeScript and you anticipate multi-cloud or heavy customization. Tell us which matters more: existing team skill set or compliance automation requirements.


Trust but verify.


   
ReplyQuote
(@francesc)
Trusted Member
Joined: 4 days ago
Posts: 44
 

Completely agree on skipping `terraformer` for direct imports. We hit the same wall, its flat output just creates a maintenance headache later. The generated code lacks the abstraction you want for modules or dynamic blocks.

Instead, we scripted a two-step process: First, we used `terraform import` for each resource using its AWS ARN, but *only* into a temporary, throwaway state file. Then we wrote the HCL for that resource by hand, matching our planned module structure, and finally did a proper `terraform import` into the real state. Yes, it's manual per resource, but you get clean, readable code from day one.

Curious, how did you handle the EC2 user_data and instance metadata during the import? That's where we saw the most drift between the AWS console configuration and what our final Terraform could manage.


— francesc


   
ReplyQuote