Just built a migration log parser to track time spent on different resource types.

In the process of migrating a significant Terraform codebase to Pulumi, a persistent question emerged from our project retrospectives: where, precisely, is the engineering time being allocated? We had anecdotal evidence that certain resource types—particularly those involving IAM configurations or complex networking constructs—were consuming a disproportionate share of our migration effort, but we lacked quantitative data to validate these observations and guide future planning.

To address this, I constructed a specialized migration log parser. The core concept is straightforward: ingest the detailed execution logs from `pulumi up` (or equivalent `terraform apply` logs from the source system), categorize each line item of activity by the resource type being operated upon, and aggregate the cumulative time spent in create, update, delete, or read operations per category. The parser keys on log lines that denote the start and completion of a resource operation, extracting the resource's URN (in Pulumi) or address (in Terraform) and the timestamps.

The implementation, a Python script leveraging regex and a simple state machine, looks something like this:

```python
import re
from datetime import datetime
from collections import defaultdict

# Simplified pattern for Pulumi log lines (RFC3339 timestamps)
LOG_PATTERN = re.compile(r'^(?Pd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d+Z)s+(?P.*)$')
START_PATTERN = re.compile(r'^starting:s+(?Pcreate|update|delete|read)s+(?Purn:pulumi:.*)$')
END_PATTERN = re.compile(r'^finished:s+(?Pcreate|update|delete|read)s+(?Purn:pulumi:.*)$')

def parse_log(file_path):
current_ops = {}
resource_times = defaultdict(lambda: defaultdict(float))

with open(file_path, 'r') as f:
for line in f:
log_match = LOG_PATTERN.match(line)
if not log_match:
continue
ts = datetime.fromisoformat(log_match.group('timestamp').replace('Z', '+00:00'))
msg = log_match.group('message')

start_match = START_PATTERN.match(msg)
if start_match:
key = (start_match.group('urn'), start_match.group('op'))
current_ops[key] = ts
continue

end_match = END_PATTERN.match(msg)
if end_match:
key = (end_match.group('urn'), end_match.group('op'))
start_ts = current_ops.pop(key, None)
if start_ts:
elapsed = (ts - start_ts).total_seconds()
# Extract resource type from URN (e.g., 'gcp:sql/databaseInstance:DatabaseInstance')
res_type = key[0].split('::')[-1]
resource_times[res_type][end_match.group('op')] += elapsed
return resource_times
```

Initial results from parsing a week's worth of migration logs have been illuminating. The aggregated data clearly showed that resources of type `gcp:projects/iAMBinding:IAMBinding` and `gcp:compute/network:Network` accounted for nearly 40% of the total 'apply' time, despite representing only about 15% of the total resource count. This was primarily due to provider-level eventual consistency and the sequential nature of certain operations that our initial code structure had enforced.

This analysis directly informed our migration strategy in two key ways:
* We prioritized the refactoring of IAM resource modules to use bulk operations where possible, significantly reducing the number of discrete API calls.
* We introduced targeted `dependsOn` overrides and explicit `provider` configurations to parallelize independent network resource operations.

The tool has proven its value beyond simple retrospectives; it now provides a feedback loop for estimating time costs for remaining modules and for validating the efficiency gains of our refactoring efforts. I am curious if others in the community have adopted similar instrumentation during their IaC migrations. What metrics did you find most actionable? Have you encountered particular resource types in GCP, AWS, or Azure that consistently emerge as time sinks during state imports or large-scale updates?