There's really no redundancy. If your headscale node goes down it can take your tailnet with it. In particular, if you drop in some ACLs that have issues, your entire tailnet can drop until you get it fixed. The recent (0.21) "configtest" can help there, but it still feels a bit brittle.
Headscale can use a lot of system resources. ~100 nodes can saturate a t3a.small instance in cpu time and disc access. Reducing the update frequency can help, but there are hard limits here. I'm imagining much of this is database updates to sqlite, but I haven't tried switching to an external postgres server yet to see how much of the load is database related.
Just for a counterpoint: I've been running headscale for 11 months, with just over 100 tailscale nodes, and it's been pretty good. There was one version upgrade that completely exploded memory use (it originally was running on a 1 or 2GB VM, with the upgrade I had to switch to 16GB to avoid thrashing), but that was fairly quickly resolved.
I would say it's been a pleasant experience, headscale and the headscale devs have been fantastic.
However, I would also agree with the statement that I wouldn't use it in production. In particular: I was hoping to use it as an overlay network for basically all traffic, between production machines and to user workstations. For the overlay network, my biggest fear there is that when headscale goes down, the entire network pretty much immediately stops responding. The usual case for this is when I make an ACL update and make an error, the entire overlay is down until I get the ACL fixed.
For replacing our OpenVPN, headscale+tailscale is going to be a clear win.
For the overlay network, I probably should go with Nebula. Headscale has these things over Nebula: Easier user onboarding (users can just login, no key exchange required), tailscale was able to route around some network problems we saw in Comcast (though it sounds like Nebula has experimental ability to do that now), and headscale has vastly better ACLs. Tailscale's are even better. Another downside of tailscale is that you can only connect to one tailnet at a time, so you can't have a "work" and "home" tailnet and be connected to both -- you have to switch.
Nebula has the benefit that there is no coordination server, so no worries about that going down. Even in the case of the Defined Networking SaaS, an outage of the control plane would just interfere with the ability to manage the network, until keys start expiring your network will continue to work.
ZeroTier also is very good, I'd classify it as closer to Tailscale, but it does have the ability to connect to multiple networks. ZeroTier in many ways is very slick, but I ended up removing it from my list of options because of a bad interactions with their sales team. It's ACLs are pretty obtuse though.
FYI: I've been self-hosting headscale for 9 months or so, and it's pretty brilliant. I didn't find it very hard to set up. A dedicated DERP server was pretty hard to set up, but most of that was I was trying to host it behind our office load balancer, and that's no bueno. But once I put it on a dedicated IP,my secondary DERP was pretty easy.
But, if you are going to self-host, seriously consider Nebula instead of tailscale. Unless you need non-technical users accessing it, tailscale has a better story there.
(edit) The biggest downside of headscale is I don't feel confident I can update ACLs without having a high likelihood of taking down the entire tailnet until I can get it fixed.
I really like a lot of Tailscale, but I just finished implementing it for my company using headscale (I couldn't get the funding to buy from Tailscale). This is across ~200 machines.
I'll be honest: If I could do it again, I'd use Nebula. The primary issues I have are that Tailscale has a lot of magic which I can see some cases it being nice, but it does make some of the routing and firewalling I'm doing on machines, and in particular the thing where it sets up Tailscale routes to network routes as higher priority than local interfaces leads to problems in my environment.
The other thing is just Headscale itself, it works quite well but does have some rough edges. It's entirely too easy to kill your whole mesh by flubbing an ACL, and currently restarting headscale to pick up ACL changes is taking 3-5 minutes.
I do, however, really prefer the Tailscale ACLs over Nebula's.
One thing that led me to Tailscale was the ability for it to relay around network routing problems, and it looks like Nebula has added that since I started. Around the time I was evaluating Nebula vs. Tailscale we had a ~1 day network routing issue where some of my users were blackhole routed in Comcast, and Tailscale just worked around it.
Tailscale is great, but for anything more than toy uses, particularly business uses, where it's a critical part of your infra, you should consider paying Tailscale or using Nebula. My biggest reasons for saying this are: Headscale config errors (including ACL issues) will take down the whole Tailnet until you can get it corrected, setting up extra "relay" nodes is fairly likely and somewhat "hard" (especially without a dedicated IP), and headscale can take quite a few resources. Data point: I recently set up a ~200 node Tailnet with headscale and in retrospect wish I had gone with Nebula. Tailscale's "magic" can be nice, but it can also lead to network weirdness. For example, I can't seem to use the tailnet to route traffic between sites without turning on "accept-routes", but turning that on causes traffic for local ethernet segments on those nodes to be routed over the Tailnet.
Reasons I went with Headscale/Tailscale over Nebula: We could enforce periodic re-logins on user workstations, Tailscale was good at routing around networking problems (Nebula has since added similar functionality), Tailscale's self-service is really nice (A user can login from any of their devices using OIDC, Nebula you have to generate a cert).
Tailscale and Headscale are both fantastic, just beware of the limitations.
Having recently put some work to essentially sell headscale-as-a-service (to clients that for various reasons wouldn't want to pay tailscale anyway even if they found the service great), about only issues between tailscale and headscale are that headscale got a bit of cruft regarding internal models that are currently being worked on, and for practical purposes it shows up in a bit harder time handling ACLs and no tailnet-peering support.
Mostly it is the "adding an ACL can take down my tailnet" issue. I had hoped to use Tailscale as an overlay network, starting to route our internal traffic over it for some things, but I've lost my tailnet so many times because of issues with headscale ACLs taking it down. This is largely a headscale issue.
Largely my issues are running ~200 nodes via headscale. Don't get me wrong, headscale is fantastic software. But it's not up for having our production networking rely on it. I tried and tried to get funding to by Tailscale, but it just wasn't in the cards with the economy as it is right now.
Headscale seems even better! They've taken what tailscale has done and improved it even more by allowing it to be a completely self hosted and private solution.
This feature is a delight to use. I've tested a few web applications, APIs, and webhooks using it over the last month or two and only experienced a handful of glitches even before it was in beta.
I like the idea of consolidating all my network ACLs with a single configuration file with Tailscale, but I don't like being wedded to a proprietary platform for my personal use. Hopefully headscale gets a similar feature, perhaps minus Tailscale's DNS management.
I recently switched to Tailscale, and it was magic.
It's very easy to recommend it to any non-tech user as well who wants to "connect to home". Well worth paying for in those cases.
But, Tailscale had so much magic, that I didn't want to be solely dependant on it, remembering how Docker is turning out.
Finding Headscale was a great discovery, and nice that the Tailscale clients already maintained can connect to a separate open-source project that lets you run your own server.
There's really no redundancy. If your headscale node goes down it can take your tailnet with it. In particular, if you drop in some ACLs that have issues, your entire tailnet can drop until you get it fixed. The recent (0.21) "configtest" can help there, but it still feels a bit brittle.
Headscale can use a lot of system resources. ~100 nodes can saturate a t3a.small instance in cpu time and disc access. Reducing the update frequency can help, but there are hard limits here. I'm imagining much of this is database updates to sqlite, but I haven't tried switching to an external postgres server yet to see how much of the load is database related.
reply