> before systemd negatively affected the quality of Debian
Is there any publication quantifying this?
I followed the whole debacle with interest, and my personal experience with my servers was the exact opposite: adopting systemd improved reliability and made administration significantly easier. It's sad that this was politicized by a small part of the community, but the end result was worth it.
> The systemd haters are right about their complaints, but there's a
ton that systemd does right
Yes. A fair and balanced technical analysis of systemd would have to
concede it does a lot of good things, and well - if you subscribe to
its paradigm [1].
My beef was with the disruption and bonfire of social capital caused
by a heavy-handed, undemocratic, vocal group imposing upon others. It
was an ugly episode and seemed to go against all the values I believed
Debian stood for. And it came from all sides. The personal attacks on
Poettering were inexcusable.
[1] I don't. I believe it's a solution that trades away system
security and reliability to get userspace convenience.
> Well, I think Lennart misjudged how much trouble his projects caused. And especially Systemd was nothing you could easily avoid.
Think about your second sentence a bit, and why it's true: major Linux distributions adopted systemd. Did they do that because it really caused a bunch of trouble or because a deliberative review process showed that it solved a number of hard problems?
A small percentage of people having very loud emotional reactions doesn't mean that it actually caused large numbers of problems and at least in the enterprise space it's been really useful for cleaning up long-running reliability and security problems by removing thousands of lines of SysV kludge and manual work recovering from problems.
> I feel like there must have been a lot of bullying to get systemd across so many systems.
I followed the Debian discussions when they debated init changing.
It was not pretty, there was a lot of things, but I don't remember any bullying. It was simply a very long uphill battle from a sort of loose group against the majority of maintainers that wished to stop worrying about 99% of init script problems.
Systemd is not perfect, never was, but it gets shit done, whereas no other project does in this regard. (A lot of people wanted to tackle some parts of what systemd does, but that was late and insufficient.)
And, all in all, there is always room for a systemd2 in Rust, distros would switch to it in a heartbeat, if it would be better.
I did. I'm still not happy with the way it was forced in, and how the predictions of teething pains of a from scratch init system being developed by inexperienced developers came true. I still remember the growing pains from PulseAudio, the previous project by the same developers.
But SystemD was a response to the definite problems of past init systems. It might not be the optimal UNIX philosophy architecture, but it's grown into something stable and powerful.
At the end of the day, as a working IT or SWEng professional, you just want to get shit done, so you learn a new abstraction for the zillionth time, put it to practice, and get on with the problem you were trying to solve. Having moved from MS-DOS, to OS/2, to Slackware, to RedHat, then Debian (and staying on Debian for decades, for desktop and server), I trust the Debian devs to put out the most stable and reliable OS I've ever seen.
To be fair, Lennart Poettering could probably end war and famine forever in a single day, and some people would still find something wrong with that.
On the other hand a certain amount of skepticism and criticism is very much in order. (I am saddened, though, by the way online discussions so easily deteriorate into name-calling and bitter ranting.)
Just to be clear, I was highly skeptical of systemd initially, now I have two systems running Debian Jessie, and quite honestly I haven't noticed much of a difference one way or the other. So at least now I am more confident that we are not all going to die because of systemd. On the other hand, I wish there had been more initial coordination / portability work to make sure stuff like GNOME keeps working smoothly on *BSD as well.
> everything wrong with the anti-systemd crowd in one sentence
Everything wrong with pro-systemd crowd in one sentence. Namely, ignoring real issues and focusing on strawmen.
Look, systemd as such seems like a decent idea. The real problem I have with systemd is the attitude of the people who run it.
Off the top of my head, issues I've seen with systemd:
* At some point there was no way to say, "bring up this service once the network is up". I mean, there was technically a "network" target, but it considered "localhost" as a network. So if, say, you have automounted NFS, the automounter starts running before your main interface has DHCP. This setup been a widely-used configuration for decades; the systemd maintainers didn't care.
* At some point, if you typed "reboot ; exit" in an ssh session, the "reboot" would hard-kill your ssh session and your shell before the "exit" would be run; so your ssh connection would then hang until the machine came up again and refused a TCP resend request.
* The whole thing with systemd reading the kernel's command-line, deciding that "debug" was meant for it, and spamming the logs making it completely unable to boot; and forcing the kernel to introduce a work-around to hide "debug" from systemd.
* The whole interface renaming thing that's happening in Debian now; every time we've upgraded our test systems from jessie to stretch, and then stretch to buster, we've had to spend dozens of man-hours figuring out why our network configurations aren't working.
The problem here isn't so much that there are these sorts of bugs; the problem is that there seems to be an attitude of, "Well my set-up works; yours doesn't matter." That's not the attitude such a core piece of infrastructure should have.
And your response has exactly the same attitude. "systemd is fine; anyone who doesn't like it must be a raving lunatic and can be safely ignored."
Nobody should be threatened with violence over a piece of software, and the comment you responded to is absolutely out of line. But the reason people are becoming raving lunatics is because they're not being listened to.
> SystemD was selected by the Technical committee, which was very far from unanimous in the decision.
I didn't know that. I thought the poll was open to anyone to vote for the relevant time window.
> Debian broke their social contract, showing that their priority was no longer with their users.
Isn't this a little bold claim? AFAICS, the poll has spotted the potential problems and general grievances pretty well (like binary logs, etc.), and concluded that these reservations should be addressed before systemd has included in Debian?
I'm not trying to defend systemd here. As I said in my comments, I disliked it first, and then I learnt and got used to it. My teammates has shown bigger reactions to it, and then they liked it too, after we configured it to behave the way we like. I didn't migrate to systemd voluntarily. It fell from above on me in a pretty hard way.
> That's good that you didn't experience any bumps in the road, but did you notice the supposed benefits of systemd yet? Namely faster startup times?
I have, for what it's worth, but it's worth noting that the biggest benefits of systemd are invisible to end-users. Startup time is the one mentioned most frequently because it's one of the few that's directly observable by end-users, not because it's the most important.
For example, systemd makes it much easier to write services (both system and user services), and it definitely has delivered on this promise, since I take advantage of it frequently. Most users still don't have a need for this, though for the ones that do, it's now several orders of magnitude easier to write basic services.
But more importantly, systemd is far easier to maintain. That's the real reason distros like Arch switched so rapidly[0]; it wasn't part of some vast systemd conspiracy. They were willing to support sysvinit as long as there were people to maintain them, but it turns out that almost nobody wanted to maintain it. And since it's a volunteer/community-run distro, you can't exactly force people to maintain code they don't want to.
There was one guy who posted to the mailing list and was very adamant about providing an AUR package for it, but last I checked, it didn't have much traction.
[0] Arch switched back in 2012, long before most other distros. And they deprecated sysvinit within less than a year - far faster than they originally planned - for the above reasons.
> Debian picked up systemd because it was a popular decision, not because it was better. As far as i'm aware there was nothing broken in sysvinit that systemd fixed; there was just a vocal minority that wanted some new features and insisted it be shipped as default.
I don't feel that's a fair summary of the lengthy debate had about this. There is a long page here [1] listing the reasons for selecting systemd. There is also [2], a very good summary by Russ Allberry of the different init systems being suggested (including staying with sysvinit). Debian has rarely been accused of taking a major technical decision because it is hip.
> My understanding (I believe detailed in the article) is that Debian, at least, chose systemd because they were forced to
This (and much of the article) is incorrect. Debian went through an extensive evaluation process at the time, over the course of 4 months, considering several different init systems, and chose systemd based on this detailed technical evaluation. That evaluation also considered sysvinit, openrc, and upstart. All of that evaluation occurred publicly, and is archived on Debian's mailing lists and bug-tracking system.
> Can someone share the reasons why systemd generate so much hate ?
A handful of improvements came tied to a bunch of new constraints and limitations that were not mandatory to get those improvements. This annoys the people who deal in those less visible areas and got the short end of the stick.
A large chunk of what systemd does could have been implemented without also sprawling it's tentacles and cross-dependencies into everything. Had they rolled out that featureset first and given the rest of the ecosystem some time to evolve in response adoption would have gone much smoother.
> systemd has been a net positive for the linux ecosystem.
You're presuming to speak for an awful lot of people there, on a topic that would be difficult to measure.
> since the beginning of systemd people have moaned
> it's quite annoying that the armchair linux experts complain
Now you're overgeneralizing, and doing so in a dismissive and patronizing way.
Here are a few examples of problems I have with systemd:
System shutdown/reboot is now unreliable. Sometimes it will be just as quick as it was before systemd arrived, but other times, systemd will decide that something isn't to its liking, and block shutdown for somewhere between 30 seconds and 10 minutes, waiting for something that will never happen. The thing in question might be different from one session to the next, and from one systemd version to the next; I can spend hours or days tracking down the process/mount/service in question and finding a workaround, only to have systemd hang on something else the next day. It offers no manual skip option, so unless I happen to be working on a host with systemd's timeouts reconfigured to reduce this problem, I'm stuck with either forcing a power-off or having my time wasted.
Something about systemd's meddling with cgroups broke the lxc control commands a few years back. To work around the problem, I have to replace every such command I use with something like `systemd-run --quiet --user --scope --property=Delegate=yes <command>`. That's a PITA that I'm unlikely to ever remember (or want to type) so I effectively cannot manage containers interactively without helper scripts any more. It's also a new systemd dependency, so those helper scripts now also need checks for cgroup version and systemd presence, and a different code path depending on the result. Making matters worse, that systemd-run command occasionally fails even when I do everything "right". What was once simple and easy is now complex and unreliable.
At some point, Lennart unilaterally decided that all machines accessed over a network must have a domain name. Subsequently, every machine running a distro that had migrated to systemd-resolved was suddenly unable to resolve its hostname-only peers on the LAN, despite the DNS server handling them just fine. Finding the problem, figuring out the cause, and reconfiguring around it wasn't the end of the world, but it did waste more of my time. Repeating that experience once or twice more when systemd behavior changed again and again eventually drove me to a policy of ripping out systemd-resolved entirely on any new installation. (Which, of course, takes more time.) I think this behavior may have been rolled back by now, but sadly, I'll never get my time back.
There are more examples, but I'm tired of re-living them and don't really want to write a book. I hope these few are enough to convey my point:
Systemd has been a net negative in my experience. It has made my life markedly worse, without bringing anything I needed. Based on conversations, comments, and bug reports I've seen over the years, I get the impression that many others have had a similar experience, but don't bother speaking up about it any more, because they're tired of being dismissed, ignored, or shouted down, just as I am.
I would welcome a reliable, minimal, non-invasive, dependency-based init. Systemd is not it.
> Has anyone had positive experiences with systemd?
I was a relatively early adopter to CentOS 7, and I have yet to run into an issue that I could pin on systemd across any of the Linux systems I use or maintain. Then again, my requirements tend to be pretty "normal" and not terribly taxing - at most I crank out a couple of server-specific service and timer files and get on with my life.
I totally understand where some of the critics of systemd are coming from. No matter how you slice it, more lines of code is invariably going to result in more bugs. And the development team does seem a bit myopic at times - though nothing deserving the death threats they get.
However, there were some clear disadvantages to the old-school sysv init, and I thought that it was important that Linux settle on a new standard init system and do it without yet another instance of fragmentation happening, like with rpm vs deb or Flatpak vs Snap or countless other examples. Systemd happened to be it, but honestly I didn't even care what the init system was as long as something finally "won" and didn't introduce yet another schism. In fact, I highly suspect that if something else had won people would have doubtless found an axe to grind with that one too.
> Just because there's a vocal minority that complains very loudly doesn't mean it doesn't work for most users out there.
Yes, obviously.
> The distribution maintainers have spoken; systemd is a vastly superior to any alternative out there.
This seems like a logical leap. Sure, distros have spoken -- but that doesn't speak to whether or not systemd is generally superior. It only speaks to it being acceptable for distros.
> People complaining about systemd is like people complaining that these newfangled LED lights are not producing enough heat.
I don't see that parallel at all. But I guess it doesn't matter.
What I do know is that I've used systemd a lot, and it certainly has its benefits. But it also has its drawbacks. Personally, the tradeoff is not acceptable to me.
This is why systemd has spurred me to stop using Linux in favor of BSD.
> I think some people are just irrationally upset about perceived threats to something that is very important to them.
Some opponents of systemd seem to think that a move to systemd is an irreversible move in the wrong direction -- that once a distro moves to systemd, there's no going back, because systemd is large, complex, and monolithic, unlike the simple and modular SysV init system. If one is strongly attached to Debian, and feels that the distro is moving irreversibly in the wrong direction, I can see why one would strongly oppose such a move.
> I wouldn't be against systemd if it were just an init system. The problem is that it's not. That's the end of my argument.
All of systemd's functionality, as far as I'm concerned, is rationally motivated by wanting to orchestrate and instrument system startup in a sensible way. What functionality do you think systemd should not have that it has today? What is a better way of achieving the result provided by that functionality?
If your perspective is that you don't care about that functionality or the related user experience, and so all of its complexity is dead weight to you, OK. That makes sense. But as a software engineer and system administrator, I find the features extremely useful, and I think many other people do too. That's why systemd has been adopted by major distributions.
I wrote a comment earlier [1] about systemd that reviews Russ Allbery's analysis of systemd [2], from when Debian was evaluating switching to systemd and upstart. Russ's comment reviews a number of different systemd functions, such as its integrated journal, and then explains the pros and cons of each. Russ even said:
Integrated daemon status. This one caught me by surprise, since the
systemd journal was functionality that I expected to dislike. But I was
surprised at how well-implemented it is, and systemctl status blew me
away. [lots more details in linked comment]
Sure, you could argue an init system "shouldn't" have an integrated journal (for example), but then you'd lose the clear benefits that Russ outlines. You wouldn't achieve the same benefits or user experience. Similarly, systemd's configuration-driven approach to service definition makes it possible to employ a wide variety of standard configuration options across all services. From my previous comment:
> I can launch my service at the appropriate time during boot with configuration as simple as:
[Unit]
Description=Demo service
[Service]
Type=forking
ExecStart=/usr/sbin/my-daemon
> Now let's say that I didn't author this daemon, but I'd like to run it with a private network, private temp folder, or a private /dev namespace. Or perhaps the daemon needs to run as root, but I want to drop all capabilities it doesn't need. It's as simple as adding these lines to the service's configuration:
> The fact that systemd supports these configuration options means that there's a simple and standard way to employ them with any service. The service itself doesn't need to support them, and needn't complicate its own daemonization logic to do so correctly. Indeed, I don't need to trust the service to daemonize or drop capabilities, since I can tell the init system do that before launching the service.
> I can drop capabilities with CapabilityBoundingSet=, or limit resource usage with CPUSchedulingPriority=, IOSchedulingPriority=, etc. I could even tell systemd to open the listening socket for me so the service doesn't need CAP_NET_BIND_SERVICE! Moving these options into the init system makes a ton of sense, because it gives administrators the ability to employ these features from outside applications, not just by enabling them within applications that bother to explicitly support them via command line arguments. Systemd better encourages the principle of least privilege: if a system daemon does not need the ability to "ptrace" other processes, or bind to ports <1024, then as the administrator I can take those away with CapabilityBoundingSet= in the unit file. Chrooting the service is as easy as RootDirectory=. This is a huge step forward compared to the world where every service must be relied upon to expose these settings, and must be trusted to implement them correctly.
> Given the extraordinary scope of systemd, what happens with the next major issue? Having to perpetually work around poorly designed software is infuriating.
Systemd doesn’t break stuff if they just feel like it. Everything is compatible if it can be, for example you can still run /etc/init.d scripts and manage them through systemd on Debian. Lingering processes are also still supported! It’s a configuration switch that most distros decided to turn on by default, because...
> Why should the onus be on the end user? Perhaps the distributions should be making choices that are less antagonistic of their users (e.g. upstart instead of systemd).
... it’s a net benefit to most users. It’s only “antagonistic” to a particular subset of powerusers perfectly capable of working around the issue but somehow more motivated to loudly complain about it on Internet.
> You're right about the tradeoffs though, and one of the tradeoffs for buying into systemd is angry users.
Fair deal if it helps with even 0.1% desktop market share.
The comment reviews Russ Allbery's analysis of systemd [1], which he wrote as part of Debian's evaluation of whether to switch to it. Russ lays out a number of detailed arguments in favor of systemd, and contrasts systemd to alternative init systems.
Is there any publication quantifying this?
I followed the whole debacle with interest, and my personal experience with my servers was the exact opposite: adopting systemd improved reliability and made administration significantly easier. It's sad that this was politicized by a small part of the community, but the end result was worth it.
reply