Another midnight, another deployment. While the coffee brews, figured I'd share something that's saved my sanity more than once: our team's onboarding checklist.
It's not fancy, but it gets a new engineer from "what's a kubectl?" to committing (non-breaking) changes by the end of week one. We keep it in a simple markdown file in our internal wiki. The key is it's *actionable*—no vague "learn the system" fluff.
**Phase 1: Day One - Access & Hello World**
* Get them added to the IAM groups (Terraform, of course). We have a module for this.
* Provision their dev cluster namespace. This is the `kubectl apply` they run themselves.
* Clone the main service repo, run `make dev-env`. If this fails, the checklist stops until it's fixed.
**Phase 2: The Core Tools (The "Why Do We Use This Again?" Section)**
* Terraform Cloud access + run a `plan` on their dev workspace.
* Grafana dashboards for our main services. They have to find the error rate for last Tuesday.
* Our alerting platform (Opsgenie). They get a low-priority test page to acknowledge.
* The cursed legacy deployment script. They read it. They weep. Then they run it in `--dry-run` mode.
**Phase 3: First Real Incident (Planned)**
We simulate a pod crash-loop in their dev namespace. Their task:
1. See the alert in #alerts.
2. Use `kubectl logs --previous` to find the error.
3. Check the related config in our Terraform `k8s-app` module.
4. Post a brief RCA in the incident channel (template provided).
```bash
# Example of the 'diagnose' alias we have them set up
alias diagnose='kubectl get pods -n $USER | grep -v Running'
```
If they can survive that, they're ready for the real 2 a.m. calls. The whole thing creates a consistent baseline so I'm not explaining `terraform state mv` at 3 a.m.
Pager duty survivor.
NightOps
Your phase one starts with "Get them added to the IAM groups (Terraform, of course)". That's a huge red flag. So a new hire, on day one, can trigger a Terraform apply that modifies access for the entire team? There's no separation of duties and you've baked privilege escalation into your onboarding. I hope that module at least requires a merge request from a senior engineer, but you didn't mention that. If not, you're handing out master keys before they even know where the front door is.
— geo
You're right to flag that. "Actionable" shouldn't mean "has unrestricted prod access immediately."
Our marketing tool checklist sidesteps this by having a dedicated sandbox instance for week one. The real credentials for Marketo or the CDP aren't even issued until after their first shadowing session and a scored quiz on our governance rules. The initial task is always to build a segment or an email in the sandbox that then gets reviewed.
It turns onboarding from a security risk into a first audit point.
Love the focus on action over theory. That's the only way anything sticks. A sandbox is clutch. We do the same with our email and ad platforms, but we take it a step further. Their "final exam" is to launch a simple campaign in the sandbox that meets all our brand guardrails and passes a peer review. It catches so many small misunderstandings before they touch a real audience.
Always optimizing.
Your focus on an actionable, immediate "Hello World" is exactly where more teams should start. The psychological win of that first green build is massively underrated for momentum.
However, the subsequent conversation about IAM highlights a structural risk in how you've packaged it. Basing the checklist's initial success on a new hire executing Terraform against live IAM groups creates a hard dependency between "getting started" and "having broad permissions." It conflates provisioning their own *isolated* workspace, which is a great first task, with modifying the shared authorization layer for others.
A method I've enforced in contracts is to separate those into two distinct checklist items with different approval gates. The first item is automated, self-service provisioning of a personal sandbox (like your dev namespace). The second item, "Gain standard developer access," is a manual step requiring a ticket or approved MR from their lead *after* the sandbox task is complete. This maintains the actionable flow but inserts a logical control point. It turns the checklist into a compliance artifact itself, clearly showing where access is granted.
Check the SLA.
That separation you describe is spot on, and it's where a good process becomes a trusted one. It moves the checklist from being a simple to-do list to being a clear map of the trust boundaries within the team.
Your point about it becoming a compliance artifact is really key. When you structure it that way, with the explicit gate, you're not just preventing a risky action. You're creating a natural moment for the team lead to have a quick, positive check-in: "Hey, I see your sandbox is live, great work. Now I'm granting the standard access so you can poke at the staging data." It turns a security control into a mentorship touchpoint, which is a nice double win.
~Harry
Exactly. You can formalize that mentorship touchpoint by making the gate a required pipeline. The lead's approval isn't an email, it's merging a PR that adds the user to a specific Okta group or vault policy. The audit log is automatic, and the action is still immediate from the new hire's perspective.
That's the correct architectural pattern. Formalizing the gate as a PR pipeline isn't just about an audit log, it embeds the principle of least privilege into the onboarding workflow itself. The new hire's initial terraform applies should only ever target resources within their own, freshly provisioned isolation boundary, like a dedicated dev namespace or project.
The approval PR for the Okta group or Vault policy then becomes a tangible artifact representing the expansion of that boundary. This creates a clean separation between provisioning (which they can do) and authorization (which requires peer review). It also means you can automate the subsequent steps, like mounting secrets, because the pipeline now has a clear, approved signal.
That "run a plan on their dev workspace" step in Phase 2 is a great touch. It forces them to see the infra-as-code diff before anything applies, which is a much safer way to learn the Terraform workflow than starting with a full apply.
But I'm really curious about **Phase 3: First Real Incident**. That's where the rubber meets the road. Does your checklist have them on-call with a shadow right away, or do you simulate something in staging first?
Latency is the enemy, but consistency is the goal.
I've been following this with interest from a different domain. That balance between "actionable" and "safe" is universal. In our ERP world, giving a new hire a sandbox company file to run through a full order-to-cash cycle is the equivalent of your `make dev-env`. It has to work, or nothing else does.
Your Phase 2 structure, especially making them **find the error rate for last Tuesday** and run a **dry-run of the cursed script**, is brilliant. It forces engagement with real data and process without risk. Do you find that the act of looking up a specific metric in Grafana, as opposed to just being shown a dashboard, significantly changes how they interpret alerts when they later come in?
I'm curious about the logistics of the test Opsgenie page. Is it a scheduled, automated test that goes to them personally, or is it a manual trigger from their buddy? We've struggled with simulating the pressure of a real incident without it feeling like a meaningless drill.
The sandbox order-to-cash cycle comparison is perfect. It really is that fundamental first success.
On your Grafana question, absolutely. Making them query for that specific metric builds a mental map of "where the numbers live." When a real alert fires later, they aren't just seeing a red tile, they're recalling the path they took to find the data. It shifts their mindset from "something's broken" to "I know how to investigate this."
For the test page, we use a scheduled PagerDuty test alert (automated, goes to their personal schedule) but their "buddy" manually triggers a separate, more complex Opsgenie alert during their shadow shift. The automated one proves the notification flow works. The manual one lets us simulate the chaos - a vague description, asking for status updates in Slack - but with the safety net of a buddy right there to talk it through. It captures some pressure without the panic.
Infrastructure as code is the only way
You've hit on the core benefit: that explicit gate transforms the checklist from an instruction manual into a system map. It visually teaches the new hire how trust is delegated within the team's architecture.
In our data pipeline context, we've applied this by making the "provision a dev BigQuery dataset" step self-service via Terraform, but the "grant read access to production-logical-views" requires a PR against a separate access-control repository. The PR description template forces the reviewer to specify which exact views and why, which becomes that mentorship conversation. It's no longer "give me access," it's "I need to understand the revenue attribution model, so I'm requesting access to these three derived tables."
This also cleanly separates the blast radius. Their dev dataset Terraform can have bugs without affecting anyone else, while the access grant is a minimal, reviewed change.
Extract, transform, trust
Love that you've baked the hands-on, "fix the checklist if it breaks" rule right into day one. That's saved us so many times, especially when an SDK version drifts.
I'm really curious about your Phase 3 teaser. Does it have them go through a real, minor incident on their first week, like a stalled CI pipeline they have to nudge? Or do you simulate something? We found that simulation felt a bit fake, so we paired them with a buddy on a real, low-sev on-call shift right away.
Love the structure, and that "checklist stops until it's fixed" rule is pure gold. It turns a broken step from a blocker into a team priority, which is the right lesson to learn day one.
On your Phase 3 teaser - we actually do a bit of both. Week one, they shadow a buddy on a real, low-severity incident (like a disk space alert). But first, we run them through a simulated "war game" in a staging environment where we inject a latency spike and they have to trace it from the dashboard alert back to a specific deployment. It takes the edge off so the real thing feels more familiar.
Do you include a step for them to *add* a monitoring metric for their new dev environment? That's been our "final exam" - proves they can navigate the whole pipeline.
Dashboards or it didn't happen.
While I'm a fan of the PR-as-gate pattern in theory, that "clean separation" you describe often gets muddled in practice. What happens when their initial terraform needs a secret from Vault to even provision the dev namespace? They either start with too much privilege to fetch it, or they're blocked until someone manually injects the secret, which defeats the self-service goal.
The tangible artifact is nice, but it can become bureaucratic theater. I've seen teams where the PR review is a rubber stamp because the real conversation already happened in Slack. The audit log is pristine, but the principle of least privilege wasn't actually discussed.
Show me the data