After several iterations of managing SSH tunnels and brittle VPN configurations for my homelab, I've standardized on Tailscale as the secure overlay network for all services. The primary win is the deterministic connectivity it provides between geographically distributed nodes, which is crucial for my CI/CD pipelines.
My current network map consists of:
* **Primary CI Server:** On-premise Jenkins controller (Ubuntu 22.04 LTS)
* **Build Agents:** Three Raspberry Pi 4s (ARM64), one old Intel NUC (amd64), and a cloud-based VM for cross-architecture Docker builds.
* **Artifact Repository:** A Nexus container running on a Synology NAS.
* **Monitoring Stack:** Grafana, Prometheus, and Loki on a dedicated monitoring host.
All these nodes are Tailscale subnet routers, advertised from their respective hosts. The key configuration is the `--advertise-routes` flag, which allows seamless routing between physical subnets without exposing ports.
```yaml
# Example systemd unit override for the subnet router
# (/etc/systemd/system/tailscaled.service.d/override.conf)
[Service]
ExecStart=
ExecStart=/usr/bin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock --port 41641 --advertise-routes=192.168.1.0/24
```
The Jenkinsfile for any pipeline can now reference internal hosts directly via their Tailscale IPs (100.x.y.z) or MagicDNS (`hostname.tailnet-name.ts.net`). This eliminates any network-dependent failure modes in the pipeline itself.
```groovy
pipeline {
agent any
stages {
stage('Build & Push') {
steps {
sh 'docker build -t nexus.tailnet123.ts.net:8083/myapp:${BUILD_ID} .'
sh 'docker push nexus.tailnet123.ts.net:8083/myapp:${BUILD_ID}'
}
}
}
}
```
The most significant operational benefit is the simplicity of access control lists (ACLs). I can define tag-based policies to, for example, restrict the cloud builder to only communicate with the Nexus repository and the Jenkins controller, following a principle of least privilege.
For those managing hybrid environments, Tailscale provides the network abstraction layer that makes infrastructure-as-code truly portable. The main pitfall to avoid is ensuring your subnet routers have stable connections and that you monitor their status; a downed subnet router can silently break routing.
--crusader
Commit early, deploy often, but always rollback-ready.
That `--advertise-routes` flag is the game-changer, isn't it? I set up something similar for my own test clusters. I'd recommend adding a quick iptables rule on your subnet router hosts to masquerade the Tailscale traffic. It saves a lot of headache with return path routing. Something like:
`iptables -t nat -A POSTROUTING -o tailscale0 -j MASQUERADE`
Do you find the ARM64 builds on the Pis keep up with your pipeline demands, or do you lean more on the NUC/cloud VM for heavier jobs? I've had to get creative with resource constraints there.
Great tip on the MASQUERADE rule, that's essential for clean routing. I learned that one the hard way after a few hours of debugging 😅
For the Pi build agents, they handle all our ARM-specific Docker builds and light testing just fine. Anything heavier, like full integration test suites or compiling larger Go binaries, automatically gets routed to the NUC or cloud VM via labels in Jenkins. The key for us was setting explicit resource constraints in the agent config so the queue doesn't get bogged down.
How do you handle resource tagging in your setup? I'm always looking for better ways to steer jobs.
You've added the iptables complexity for routing, which is something Tailscale should handle out of the box but doesn't. That's a hidden operational cost right there.
Resource tagging is fine, but you're still locked into Jenkins' scheduling model. For light steering, have you looked at just using basic host tags and filtering on the agent's actual free memory at job start? It's simpler than maintaining complex labels and you won't get jobs stuck on a Pi that's temporarily overloaded from another process.
What's your plan when Tailscale changes its pricing model again?
I hear you on the iptables complexity, it's definitely a manual step that feels like it should be automated. For what it's worth, I treat it as a one-time setup cost per subnet router, and it's been stable for me for over a year now.
I like your point about checking actual free memory instead of static labels, that's a clever way to avoid resource starvation. In my sandbox, I've been playing with a lightweight script on the agents that reports live memory stats as a Jenkins node property, which then gets picked up by a simple conditional step in the pipeline. It's a bit more work upfront but really cuts down on those "stuck job" scenarios.
As for Tailscale's pricing, that's a valid concern with any SaaS tool. My hedge is keeping the configs documented and portable. The wireguard configs underneath are standard, so if the pricing ever becomes untenable, I'm prepared to migrate to a headscale setup or even roll my own with plain wireguard, though I'd miss the magic DNS and user management. Have you built in any similar escape hatches?
If it's not measurable, it's not marketing.