run0, which behaves like sudo, but works entirely differently and is not SUID. Run0 asks the services manager to create a shell or command under the target user’s ID, creating a new PTY, sending data back and forth from the originating TTY and the new PTY.
Is that even a problem? Any program can shell out to sudo, hence why you shouldn't set NOPASSWD in sudoers. Polkit takes in a request on an unprivileged interface, that request is evaluated in privileged code against the set of privilege rules, and then passed the proper capabilities if the rules allow. This includes a mechanism where it can, if desired, prompt a user to enter a password etc to prevent a rogue program silently acquiring root. But even in the worst case, the rogue program is not going to acquire any capabilities that you would not otherwise have as with sudo, and the breakpoint between privileged and unprivileged code is (in theory) more tightly defined and controlled.
I'm really starting to hate the sub-community in Linux that tries to constantly change it.
I don't want to learn a new network config alternative with every update (Ubuntu changed its net config tool again with 24.04). I don't want an immutable os. I don't want to learn to write new config files. I just want to do what I've been doing but with new packages. If there's a problem with something, just fix it. Don't throw out the whole thing.
I moved to FreeBSD and am happy for its reluctance to change. If there is any, it's usually offering something genuinely new to me as a feature and to boot I only need to learn about it if I need it.
Hardware support is much lower but it's worth it IMO. I had the same irritation with macOS. Every release breaking something essential that was part of my workflow and i didn't want to change. Eventually I did change but away from Apple.
To help myself to understand others better, I made some efforts to look and keep looking outside my bubble.
That's not simple though- humans tend to think THEY know better for all others while it's often not.
Say you don't want to have immutable distros, but many want. Android or Talos or even OS for network switches are nice examples.
Linux based solutions and products related community is wide and to keep up with evolution one needs to adapt and adopt changes.
We already see what happens for immutable/stale systems like FreeBSD - even TrueNAS is abandoning it, dying as predicted by experts since 2008-2010.
On other side - nothing stops you from keep using sudo or even choose disto vendor which doesn't use systemd, there is a choice, not thing set in stone.
Say I don't plan to switch to run0 in any near future - first it's not gonna be in any LTS disto, 2nd - it seems to be lacking LDAP based rules and very much likely other important features.
> To help myself to understand others better, I made some efforts to look and keep looking outside my bubble.
I stopped doing that when 24% of my country voted for the fascists last winter. Too many people are just wrong and I really want to shut them out. I'm past trying to work together or convincing them, I just want to wall them off. So no, I need my bubble because the outside is toxic.
> That's not simple though- humans tend to think THEY know better for all others while it's often not.
Uh no. I was just speaking for myself. Not for all others.
> Say you don't want to have immutable distros, but many want. Android or Talos or even OS for network switches are nice examples.
I use android because it's more open than iOS. Not because it's great.
Immutable also means unable to remove spyware like Google stuff. This is precisely the problem with immutability, you have to be able to trust the maker of the core 100%. Not many companies deserve trust these days.
> We already see what happens for immutable/stale systems like FreeBSD - even TrueNAS is abandoning it, dying as predicted by experts since 2008-2010.
I don't care about Truenas. They never did much for the community anyway.
> On other side - nothing stops you from keep using sudo or even choose disto vendor which doesn't use systemd, there is a choice, not thing set in stone.
Yes like I said that's what I'm doing. But I lament a stronger traditionalist movement in Linux.
I really think you need some time away from a PC. You seem to be taking this incredibly personally. You can feel how you feel about the general public, but knowingly and willfully staying in a bubble is only detrimental to yourself. You will ultimately become what you hate. A person unwilling to move with the current change of society.
> A person unwilling to move with the current change of society.
Well if society goes towards fascism like the Netherlands, then yes I'm very proud of not moving along with that.
The best remaining option is to live side by side with those people and interact with them as least as possible, and hinder them as much as possible in their attempts at legislating, like using the first chamber to block things. Mind you, this is already the status quo in the US where the democrats and republicans use whatever means at their disposal to undermine the other.
We didn't really have this phenomenon in the Netherlands because we don't have this "the winner takes all" kind of political system but parties need to collaborate. But the rise of the extreme-right makes it impossible to cooperate any longer.
This is the main reason why I'm still on Slackware. Pat is keeping the same thing for the last 20-30 years. Sure, he had to introduce some stuff, like NetworkManager or PulseAudio, to keep up with the latest software versions, but every major change is postponed as much as possible. Hell, even systemd is not there yet, and I'm pleased about that.
I love it when new and better ways are found to do things.
I love it that Linux is constantly improving and moving forward.
I’m willing to accept along the way some things seem to be mis steps (I’m looking at you snap packaging).
I love it that improved network configuration systems are being adopted because network configuration is a pain.
I love systemd and when new stuf comes out from the systemd project I think “gee I’m glad finally someone is taking a wholistic look at and fixing that messy inconsistent evolved corner of Linux and replacing it with well thought out powerful and integrated solution.”
Bring on the change, change is the best thing about computing and software. I own vintage computers but wouldn’t want to live there.
I’ve been running Ubuntu boxes in prod since 12.x — there is no “improved” way to put an ip address on an interface besides writing something to a file in /etc, but every update this file changes, or it’s format does.
It’s bullshit and I wish it would end.
Alas, keeps the consultant bucks flooding in when we have to rewrite a load of cfgmgt to go to 24.04 I guess..
I just switch every system to systemd-networkd immediately. The same .ini sytax as for service files, and dependencies are easy to handle, e.g. on one system I have two physical Interfaces eth0 and eth1, I want two vlans on eth0, and then bridge one of those vlans with eth1 and then run a DHCP client on that bridge but at the same time assign an additional IP address. This is dead simple to describe with one .ini file per vlan/bridge. Seriously the first time I feel like I'm not fighting an archaic config syntax, fixing up crap in some post-hook.d script, or give up entirely on any config language and just have a convoluted script setting up everything manually.
It's also easy to explicitly express "weird" stuff like "run DHCP client and use all the config options except the default route". Seriously a couple times I needed to do dumb shit and was like "there's no way they let you do this" but no, there's a way to do it.
And I'm pretty positive the config files will stay stable over the coming years and any new networking features will get appropriate config options in newer versions.
Eh, I could. But I'm quite sure that the method of switching to systemd-networkd also requires the same if not much more maintenance as simply changing the ifconfig template every few years...
I really don't care about interface configurations that much. It's an annoyance, but one that's quite easy to fix. I think introducing yet another network configuration here isn't the answer, but my linux fleet just run k8s anyway and network config on the linux level is quite simple before we get into cillium/istio/etc :}
Sure, I made that choice at a time where after upgrading, Ubuntu suddenly wanted netplan, debian stayed with /etc/network/interfaces and I think fedora went with networkmanager. So I though f- it, they all have systemd so I'll give networkd a spin. And never looked back.
So maybe don't just switch out of the blue right now if you've got a working setup, but maybe keep it in mind for the next time. :)
> I love it when new and better ways are found to do things.
I do, too. But I also really hate it when those new and better ways make things worse for me. Systemd does that in a couple of important ways. There are even some network-related startup stuff that I can no longer make work automatically at all. For me personally, Systemd is a regression, not an advance.
But I also recognize that the Linux world is not duty-bound to make sure it remains excellent for me, and I've pretty much given up on advocating for my needs in the Linux space. There's no point.
It's very fair to eschew change for negligible improvements.
But I've also seen the community defend terrible stuff just because.
Look at what happened with the init system. System V, fstab, etc was awful. Doing anything with a reasonable level of robustness was grotesquely obtuse and complicated. And yet it was "perfectly fine" to the greybeards. Alternative proposals were near zero.
I don't have a dog in the networkmanager/netplan fight. It could be that one is irrelevant; given history, I have a hard time trusting what I hear.
sudo has quirks for sure (which is why you see a number of alternatives).
Daemontools was pretty popular with greybeards, actually. But yeah, daemontools/runit/s6 and company have always been for handcrafted server setups, where the thing about init scripts and unit files is that they're a standard thing a package can supply and have work out of the box with minimal tweaking across distros.
Any serious challenge to systemd nowadays is probably going to have to at least offer compatibility with it. No one is going to rip it all out and start over again (again).
I so want to ditch Linux for bsd, but hardware support, both for my current laptop and future pain in searching for compatible hardware is the only thing putting me off. And my server needs cuda for my AI shenanigans, so probably no bsd there either ...
That's funny. It's like there are two camps, conservatives, and progressives. ;)
Jokes aside, I just think that life is constant change and the programming industry is a good example of that. Coding practices have improved a lot in the last few years and will continue to improve with new knowledge and new technology. Sometimes it's better to start anew from scratch than trying to adapt old code into new practices.
Btw this is not a young whippersnapper saying this. My first IT job was on FreeBSD and OpenBSD. I was a full-time FreeBSD user from 04 to ~10.
And I remember exacty this gripe back in 2011 when Debian was using one network config, RHEL another. Today I actually enjoy the progress made with systemd, and I'm that annoying co-worker who will give you crap for disabling SElinux.
Well, some things are improvements. But many are not, and are just changed for the sake of it.
I'm a fan of SELinux (and the similar mandatory access control on BSD) because it gives strong security but the user or admin keeps control. This is a much better solution than things like immutable OSes where the user can't control anything and just has to trust the developer of the OS.
What complaining. There weren't paved paths before. Whatever one person learned was different from how anyone else did it. Few of the tools had anywhere near the essential capabilities, serving o ly some tiny niche of the use cases in some tool specific limited way.
Look at all the different netsevs supported by systemd-networkd. https://www.freedesktop.org/software/systemd/man/latest/syst... This is a huge list of tools that required a huge assortment of tools to do before, few of which had even part way decent management & fewer still had good init scripts. You used to have to learn from 0 each time, with each tool. Nothing was alike, nothing was as capable, nothing was integrative.
Don't listen to these complainers, for God sake. Your life is too short to get pissed off about well built work together featureful tools being built in a mono-repo fashion.
I'm so tired of the sabotage, so tired of broadscale general refusenik attitudes. This post is absurd. There was nothing to learn before, everything was 100% special snowflake & distinct. None of it was great, all of it was limited. Systemd mono-repo is built of many small pieces, but they couple together and are 100x more learnable. What you learn today will work across whatever system you run across. It's such a a better world, and these "pry it from my cold dead fingers" attitude can keep to that path for all I care, but I wish they weren't poisoning kinds with absurd incorrect negativity & being such magnets for disdain. The world today is fantastic & you rarely see these folks with even an iota of appreciation for how good we have it, never a drop of balance. But hate sells, & unites, powerfully.
> so tired of broadscale general refusenik attitudes
You can use whatever software you like on your computers, but you're not entitled to have everybody else follow your choices because everybody else has exactly the same freedom to choose as you do. And if you want to persuade people to follow your lead, name calling is a bad strategy (much less ethnically charged name calling...)
The anti-systemd crowd dishes it out like crazy & gets minimal blowback. I'm done letting them be monsters with small minded attitudes, ragging & raging with old crusty attitudes that refuse refuse refuse.
I have heard people use refusenik numerous times & never once has it crossed my mind or seemed remotely related to any ethnic matters. There does appear to be a definition though. Alas. It's a great fucking word, makes me smile, feels well crafted & spry, & I detest giving it up, relinquishing it: but I hereby renounce my previous usage & give it up. Blast. Now there is only absence.
But I do. So I use one, and contribute to a project that tries to create one. Am I a part of some sub-community that wants constant change? Or do I just have an unusual use-case and want to support it?
The beauty of OS is that anyone can decide which tool to use, contribute to it, and even fork it.
> I don't want to learn a new network config alternative with every update (Ubuntu changed its net config tool again with 24.04).
That's really just Ubuntu's fault. Between upstart, Unity, netplan, and snapd, Ubuntu likes to go off and do its own thing for a few years before coming back to what everyone else picked in the first place
Thats fine, and lord knows we probably need a replacement to sudo.
However, sudo needs to be user friendly and fail safe with decent information as to why its failed. Something that service files historically didn't do.
But, the way it's supported also needs to change, it almost certainly needs to be decoupled from systemd's release cycle.
I hope that we have all learnt from early systemd, and that we all won't take a "lets piss on each other's chips" approach. I'm too old you you lot to start flame warring over stuff you'll never actually fucking use.
doas exists but isn't universally available but already solves this problem. sudo has too many features and permits excessive configuration, but it also has the convenience of ubiquity. Inventing a third thing tied to systemd is absurd and unnecessary.
systemd has been a net positive for the linux ecosystem. remember when you had to write bash scripts to start, stop, restart services and handle any other signals you want to send it? nowadays it's a unit file (basically just an ini file) away with relatively straightforward API. and you can actually declare startup dependencies and other useful relationships past just "prepend a number signifying when it should run globally to the front of the filename". it's provided an extensible platform with which higher level orchestration frameworks like ansible / ignition can easily templatize services or other system configuration.
since the beginning of systemd people have moaned about how complex it is and how we're reinventing the wheel. yet time and time again the people actually working on the project show that the solution they've come up with is the result of the problem they're facing on a daily basis. it's quite annoying that the armchair linux experts complain about how "lol systemd is so stupid for reinventing the wheel, give me my shell scripts back", maybe think about whether or not you have a legitimate issue not being addressed by the solution proposed or if you are just getting rage baited by a headline.
It is much more than just a re-implementation of launchd. I can't grab a link right now, but Lenart wrote a long blog post about the philosophical decisions behind systemd, and he goes into detail about what he liked from launchd and what he did not like. It is called "rethinking PID 1"
Yeah, basically I've found that the people the more vocal against systemd are either not really knowing how it works behind the scenes, and just criticizing for the sake of it (or because other people do so), or criticizing from an ideological point of view (do one thing and do it well). They see systemd as an octopus, not following the unix ideology. Which I don't really agree tbh
Well, you don't have to understand the underlying concepts that deeply to know that putting all your xzs in one basket is a bad idea.
There is no single piece of software that has given me more Microsoft EEE vibes in the past decade than systemd
The attack vector for systemd is already enormous, but the inclusion of sudo seems like the most blatant attempt I've yet seen to put the final piece of this nation state backdoor in place.
The root of the problem is sudo, and su more generally. This derives from *nix, or mainstream versions today, requiring a super user, a manifestation of the problem with monolithic kernals. A microkernal may be a stepping stone to improving this but even this is not a solution. The only solution is no kernal space, no privileged user(s), all processes negotiate independently with each other.
No the root of the problem is the large attack surface systemd is creating by tightly coupling a ton of tools together, I agree about the microkernel idea completely though.
systemd isn't a single piece of software. It's a collection of software with an unified theme, like KDE or Gnome.
Attack surface-wise, I don't think there's much difference between "sudo" being a part of the systemd package and not. Either way there's "sudo" code to be targeted, which package it's part of is just a technicality.
I like to compare Systemd to GNU. Pretty similar scale, similar "take over the world" levels of adoption in Linux (GNU's libc is in nearly everything, all the other GNU tools are ridiculously common), adds lots of attack surface to what used to be single-purpose simple tools, etc. Just about every criticism of Systemd applies just as well to GNU.
So aim your ire at the distributions who (I agree) cocked this one up. "Take a library dependency to implement basic functionality" is not a systemd mentality, it's pernicious throughout software development - see leftPad as another example.
It's one of many things to consider. Think of it as sandboxing, or attack surface reduction. Should we expose everything to everything else, or should it be on a need-to-know basis?
IIRC, xz was used by a systemd library, and that systemd library got added to sshd so it could tell systemd when it had started or something like that. SSH itself doesn't use xz.
My Ubuntu /usr/sbin/sshd already links to libz, liblzma, liblz4 and libzstd. I don't see why linking to libxz would be so outrageous. All-in-all, ldd reports 26 libraries.
They attacked the weakest link, and systemd was just a small pawn in that game. Sure, a smaller attack surface is better, but it's not like OpenSSHd has a small attack surface even without libsystemd. Not even in projects with a similar possibility of obscure "test data."
> Except for libz, they are only linked indirectly though libsystemd.
Ah, that invalidates my point re. obscure test data. Sloppy use of ldd. (I'm guessing it would be much harder making such an attack on a crypto library.)
They didn't need to use the library to make use of the systemd notify mechanism, which is simple to interface and quite a nice feature in the first place.
The issue is that it's not just ideological. "Do one thing and do it well" is important because if you want to port software to another platform, it's a lot easier to port a single dependency component over to make it work than it is to port over the entire framework.
This is a serious problem and it makes it way harder to make things cross platform.
Systemd was written specifically for Linux, hard depends on a list of features provided by the Linux kernel and leverages them to do its work. Porting it to another kernel is a rewrite. Lack of portability is in this case a design tradeoff.
Sure that is the case for systemd itself but it's not the case for most projects that happen to use things systemd provides.
There is very little benefit for most userspace software to tie itself to systemd and by extension linux when otherwise it could be portable to any unix or unix like platform. Especially when an alternative, portable solution already exists and is well established.
The argument was that this design of systemd makes it hard to write applications that are portable, so that systemd is effectively a very big net negative to the open source ecosystem, because it causes massive fragmentation.
I think this is a really old statement. I’ve used systemd since it was made default in Fedora back in 2011 (Lovelock, anyone?) - at what point am I qualified to have an opinion, after having used it personally for 13 years and professionally for 8.
systemd is scary for 3 reasons.
1. it is inscrutable. Debugging it is nearly impossible, so you had better hope you don’t get a buggy release, especially with how hostile the devs can be.
2. it is large, and growing. Lots of things it claims are optional are in reality: not really. This is fine until uou get something that really doesnt work well (systemd-resolved is consistently the largest reason I have connectivity issues, wether it be because it interferes with docker inter-container networking or because it needs to time-out when trying DNSSec to continue- or if it fights with my vpn provider for power over my resolvers etc). Due to my distro being very tied to it: I gotta keep using it and working around it.
3. The interface is irreplaceable. Why are their 13 init systems? Because init is a closed scope. To be a sucessful init you need to spawn processes and do it cheaply. Supervising processes after start? Noble, and there were implementations that could do that (CDDL licensed SMF from Solaris for example). However we have already reached the state where it will be literal man-decades of work to replace systemd as we will need to make any replacement bug compatible with systemd itself. Its the ultimate show stopper.
The implementation is the reference. Which is a large departure now from what came before.
> you can actually declare startup dependencies and other useful relationships
In theory, yes. In practice, I had a lot of trouble ordering things correctly in non-trivial cases.
> it's quite annoying that the armchair linux experts complain about how "lol systemd is so stupid for reinventing the wheel, give me my shell scripts back"
I can only speak for myself, but I don't want the abysmal sysvinit scripts back. I just want a simple process supervision suite which is true to the UNIX way of doing things. The sysvinit/systemd dichotomy is false.
Runit and s6 are both very real alternatives to systemd. They lack a ton of features, but they are a way to reliably run services. They do use shell scripts, but not for the reasons sysvinit did. They are extremely simple and have a very small attack surface as a result. Runit itself is a spiritual descendant of daemontools which predate systemd by great many years.
The problem with systemd is that it's a mediocre solution to many problems. UNIX deserved better.
In a similar vein, systems like openrc use "shell scripts" as well, but can generally be written declaratively[1]. This provides greater flexibility when it comes to creating one-shot services or services that require a little setup before running, as you can just re-define the start() function, rather than requiring one to make a separate shell script for it, or dependency tree, like you'd have to in systemd land.
"sysvinit" is an ill-defined concept, anyway, as every distribution had their own scripts and tooling around actual services, and sysvinit was generally only responsible for starting getty and launching the distribution's actual service system. How initscripts were created and how you managed them depended significantly on the distribution.
The other pieces are also pretty excellent. Ifup gave me no joy & was very limited. Systemd-networkd is a wonderful option with vast & great capabilities, that meshes well with the init process & it's style. Systemd-timesyncd is fine, I dunno, works for me. Systemd-journald is probably the weakest of the batch but mostly because it's not very ambitious; I loath that there's no way to really deal with super-active programs hogging all the logspace, short of configuring a second journald instance for that program, which ain't no joy to setup. Systemd-home is a cool set of features for portability. Systemd-nspawn is a bit ahead of its time & looks a little weird now but was a very novel & powerful thing to have built in to most systems. Systemd-resolved is a pretty good local DNS that handles mdns and much weirder cases easily. Systemd-boot is a really nice easy to work with uefi boot loaders that's worlds easier to deal with than uboot or gasp grub.
All of these are taken em or leave em. But they're good expectations to have. I love the bazaar model, but Linux used to have so few common expectations, used to manage various bits so poorly. And now there's a much better base of capabilities, which have universal patterns of management that tend to apply to them (how etc files are laid how, being dbus and maybe varlink accessible). Linux hadn't been growing; when you came to a random systems you expected enormously little and typically got it, and what extras were available were scattered/random and often not particularly high quality or capable software. Systems has extended a much bigger base of competency & capability. That we can plug in a USB drive and systemd-home can create an isolated dynamic user out of that & let that user securely run their environment there is a neat as heck expectation. That our network manager supports setting up such a wide range of bridges and tunnels and taps and tuns is fantastic, is excellent. I don't have words for people turning their noses up at this better world; this is so much more competent & capable a world than where we were, gives us so much we all can now take for granted, and it's been done smartly, more mono-repo than monolithic, such that you can do alternatives. But many of these pieces are utterly without peer. And the consistency of operation you get, the predictability of use, from being under the same umbrella, is an ergonomic wonder that is unmatched.
Is there a good overview of these pieces and what they're responsible for and how they fit together?
I had a quick skim of the docs on systemd.io and there are quite a few documents but they don't seem particularly organized and I couldn't see a good architecture overview.
It's not difficult to use some parts of systemd without using others. Is it any different than something like coreutils, another package of linux utilities that are synergistic but usable seperately? Nobody complains about coreutils being bundled. Even the rewrite in rust crowd bundle them.
It really is not. This "sudo replacement" (which under the hood is systemd-run) will apparently require quite a bit of the systemd stack to operate. That is not something you have access to in many lightweight container distros or on other non-systemd distros.
And coreutils is expected because most people use GNU/Linux. Coreutils is that GNU userland part. And either way, the majority of coreutils are GNU implementations of standard unix components. You can use most software that depends on coreutils on other *nix platforms like BSD for this reason.
Oh you do: this work completely differently from sudo, there is no suid binary involved, instead it does IPC to the systemd pid 1 and asks it to spawn you process, attached to your current terminal. So if you don't have systeme as pid 1, it'll have noone to talk to.
Whether you like that or not is for you to decide.
There are. Gentoo has actually always used the FreeDesktop reference implementation instead of sd-bus (the systemd implementation).
FreeDesktop also maintains a list of implementations (also including bindings so you have to read to find which ones are full impls vs libdbus bindings).
From the thread: "the target command is invoked in an isolated exec context, freshly forked off PID 1, without inheriting any context from the client".
Its refreshing to see that just hate systemd because "its not UNIX way" is not anymore. I see better discussions under this post and in other places around the web.
You forget, the reason systemd was originally rationalized for its insertion into our trees was "boot times are too slow". Its chameleon-like nature and ability to solve the hastily described problem du jour seems to be its only consistently touted feature.
Bash scripts that start processes are ephemeral. If it's signal handling you want, that was your program's problem. Either that or your program didn't fork itself, which is a fish of an entirely different feather.
And now we have this sprawling mess of complexity and headache.
The resulting wayland environments are more complex because wayland itself refuses to define/include features that desktop systems are expected to have. This results in a sprawling mess of competing and incompatible interfaces for those gaps that other parts of the implementations (desktop environments) now have to compensate for by including multiple implementations of the same thing based on all these different interfaces.
Wayland by itself is simplier -- it is done by "outsourcing" everything a window system should do to the window manager.
There is where the complexity kicks in.
Wayland was a mis-fire in terms of user friendliness for the first decade or so. But it is still a big step up from the mess that was a typical xserver back in 2010.
> It reduces features adds complexity...
The irony here is Wayland is part of a huge effort to decomplex an xserver into component parts. A really commendable initiative; the path forward while maintaining the X protocol probably was impractically hairy.
The Wayland protocol design had some glaring flaws, but saying that it adds complexity is unfair. It oversimplified; it would have benefited from some flexibility in providing a standard mechanism to let people inspect the buffers graphics buffers to be composited.
systemd has been a net positive for the Linux ecosystem. The existence of systemd allows a user to examine a distribution to determine whether it relies on systemd, or whether it is suitable for use.
> remember when you had to write bash scripts to start, stop, restart services
This was a really big pain, yes, but I also remember how I could `tail -f *.logs`. I remember how I didn't need to remember about `--no-pager` and `--follow`. I knew where the files were, what they were called. I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename.
The init system was probably peak of systemd, after that they started reinveting things in a more complicated way. Do we really need journald, systemd-boot, machinectl, systemd-networkd, sd-bus, systemd-resolved, systemd-nspawn? Do people actually use it all? Are there any metrics to show how many systems have it installed and in use?
And what would be the equivalent to, "Oh, I don´t know the name of the log for this process I can see in 'ps aux', let me cd into /var/log and see what filenames I can find ... or grep everything until I can find a couple of words that make some sense so I can keep digging further"?
The lack of explorability in journalctl, the "need" to keep everything locked behind their own flavor of tools and magic file types, is what makes the rest of us abhor them.
Actually no, they are prefixed by the binary name and the PID number. Technically they are prefixed with: `<date> <host> <binary>[<pid>]`.
You can then use `systemctl status <pid>` to identify the unit if you need to.
I would imagine this is configurable, and this might be the configuration chosen by my distribution (since I have not changed it myself). It would actually be nice to show the unit name instead of the binary/PID combination, though not strictly necessary.
EDIT: Ooh, systemd 239 adds `journalctl -o with-unit` to do this exact thing. There are lots of other formats you can choose from as well.
EDIT 2: Unfortunately there's no way to set this as default, you must use `-o with-unit` each time or set up a shell alias :-\
Well `-u` refers to the unit file, so I would start with `sudo systemctl status` which lists the status of active unit files. I bet I could find the unit name I'm looking for there. If not, then `sudo systemctl list-units` should have it. (and you can grep the output of both)
Systemd and Journald are less opaque than I used to think. Even if you don't want to learn the commands, all of its unit files (and the relationships between them) are available through the filesystem. Most of your unit files will be in `/etc/systemd/system`, and the active relationships between units are expressed through soft links.
I think you can also configure the journal per unit right? Don't remember how exactly, at the very least you can stream logs to a different system.
There are also services which will just use their own logging convention for some reason. Honestly it's easier just to go with journalctl for everything you can.
These absolutely SUCK as answers to that question. They entirely miss the point. They provide a specific answer to a general problem.
The problem with systemnd is it assumes that it's possible for all needs to be predicted and accounted for ahead of time. While "look around at directories and files, and grep within them" works after the fact without any special knowledge or tools. The person who wrote the log file did not need to know how someone will maybe try to access it 23 years later on a different OS. It's just a regular file that can be read by anything over any kind of channel on any os. The person finding themselves needing to read the file does not need to have any particular command installed, or installable, or runnable. It does not require the happy path in order to work.
I have a joke I always say, often self-deprecating making fun of my own self for the way I do things sometimes, but also when I'm trying to commiserate with a customer so they don't feel intimidated by "the expert" or "the engineer": "37 easy steps!"
Every answer that starts with "it's simple, just journalctl ..." is FUCKING 37 easy steps. The very name of the program itself is a trainwreck. journalctl... it takes me 18 seconds just to type it.
systemd is great for managing exquisitely washed masses of drone vms. It's utter and complete shit for direct administration, operation, development, debugging, flexibility, or custom integrations.
They may be correct answers, in that they accomplish a task.
They suck in that the proposed method for accomplishing the task is a suck-ass downgrade from the previous ways the new way proposes to displace.
I beg to differ about emotive, because absolutely the other side of this fence behaves every bit as butt-hurt when challenged as I just did. You yourself just said these crappy commands were good, as a purely unfounded assertion. They are great because you just say they are great. That is even less objective than my rant. I at least explained what exactly I find so bad. Tell me more about not emotive.
> You yourself just said these crappy commands were good, as a purely unfounded assertion. They are great because you just say they are great. That is even less objective than my rant. I at least explained what exactly I find so bad. Tell me more about not emotive.
Very well: I didn't say the commands were good or great. If you attempt a little objectivity you'll see it.
If you attempt a little objectivity, you will see that addressing the answers (vs the commands) was in my first response.
Before even going into the nature of the commands, I said that the answer does not match the question. The answers addressed details, while the details in the question were merely examples.
Missing the point is actually merely one of the at least two dimensions along which the answers suck. Thank you for reminding me about that.
> The answers addressed details, while the details in the question were merely examples.
No, the answers addressed the two scenarios in the question. That's why they don't "SUCK". They are good answers to the question as asked, and not the question as you dreamed it. If you spent ten words on asking the question of your dreams, rather than having a go at people for not answering it (whatever it is), you might have an answer.
To be clear -- you can just dump out all the logs to text and then use your standard grep toolkit on them. You can also dump them out to JSON, and use jq on them -- something you can't do with text logs on disk.
And as far as learning the options goes, learning the basics of systemd and journalctl is much easier than learning how to, say, use a source control system effectively.
(And it's definitely worth trying out new source control systems -- for example, I think most people should at least check out Jujutsu, and many people are going to like it more than Git. As simple as Jujutsu is to use, learning it takes a lot longer than learning journalctl.)
Agreed. I’ve tried to get used to it on Debian 12 for personal VMs. I just can’t. I don’t care. grep and awk are always going to be available, and I know how to use them.
It's way better than the previous status quo: now, more or less whatever distro I'm on, there's one command which answers that question directly, not a process of guessing how this system happens to be set up. You can guess and grep through most any kind of problem, but I don't see why anyone would want to do it as anything but a last resort, and to me that fits '37 easy steps' far more than the 1 consistent step in these replies (notice how none of them say 'oh, but on debian...'?).
> journalctl... it takes me 18 seconds just to type it.
Sounds like you don't use a keyboard often enough to even remember where the keys are. Not sure I would expect deep (or even shallow) knowledge on UNIX system administration from such a person.
You do have to learn a few new things, yes. But it's not too difficult. We all have to learn new things sometimes.
I spent around 15 minutes a few weeks ago learning how to do a few things with journalctl, and I came away from it with a great appreciation for its power.
systemctl status <pid> will show you the unit and helpfully the last few lines of log from it. Journalctl -u <unit name> will then show you the full logs
> Oh, I don´t know the name of the log for this process I can see in 'ps aux',
With services using journald, there's no "name of the log" because everything's in the journal, so that part isn't a problem. Rather than "is this in auth.log or syslog or thisservice.log", it'll always be in the journal.
> let me cd into /var/log and see what filenames I can find
You can filter journal entries by unit (-u) or by service identifier (-t). Often, though, I find it really useful to be able to see the adjacent entries from other services at the same time, since that can give some indication of what's happening on the system that caused an issue.
> or grep everything until I can find a couple of words that make some sense so I can keep digging further
journalctl --grep, or more conveniently, journalctl and then / to search.
> The lack of explorability in journalctl
It's explorable by dozens of different axes, and if all of those aren't sufficient, you always can get the whole thing as text and run any command you like on it, or get it in structured form and do structured queries on it.
Same. Init system is awesome, love .ini files, networkd is as simple as it can be, journald is fine. Love systemd-nspawn for simple containers as it feels more lightweight than docker. Especially for quick one-offs where I don't need isolated networking you basically have chroot on steroids with "systemd-nspawn -D /path/to/fstree".
> I could `tail -f *.logs`. I remember how I didn't need to remember about `--no-pager` and `--follow`.
journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there.
> I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename.
My memory of this time was that every single application had its own unique method of handling logs and its own unique location for storing them. So sure, once you found the logs you didn't have to Google how to find the relevant dates if the application is using logrotate as you describe. But finding the logs in the first place was always a challenge, and systemd is nice in that it provides a single place where all logs go and a single interface for navigating them.
> journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there
Yeah, this is a bit of an odd complaint. If I were this bothered by having to type those, I'd just make an alias to "journalctl --follow --nopager"` and would have forgotten about it years ago.
I think my biggest gripe with journalctl is (and this may just be because I'm novice at using it) that discovering what sort of logs are on the system feels a lot more complex.
With regular logs, I can go to /var/log, ls the dir, and get a nice list of what's being logged. There will be, for example, an `apache.log` file that has all the logs relevant to apache.
It's just journalctl -u apache -f, assuming your distribution calls the unit file "apache", and the last few messages of the logfile appear with systemctl status apache
And you don't need to worry about "is this service rotating logs appropriately?"
The important thing for me at least isn't he `-u apache` but rather knowing that `apache` exists as one of the installed services on this box.
In the old ways, that was apparent because of the presence of the log file.
Granted, this isn't the problem it used to be for us, however, it would come up because we'd have our services named things like `foo-ws` and knowing that `foo-ws` existed on a given box might be tricky were it not for the log files.
I mean "systemctl status" shows all services and their hierarchy, there's also "systemctl list-unit-files" if you want to see things that aren't part of the current target (runlevel).
It's much easier in the systemd world to see the state of the system, the state of the service, and the logs of a service because it enforces this consistency.
correct me if i’m wrong, but i believe the point that is trying to be made is;
a system user/admin has an intuition about files. saying that ‘journalctl -f -u’ (fu, indeed :) and whatever else is inherently undiscoverable, and is a.. basically orthogonal mechanism for handling what should be a simple task. i.e., viewing some logs. it’s far easier to compose and extend from files (what if i only care about the mtime of the log, for instance), than this.
look, i think systemd isn’t.. terrible. i also think it’s suffered a bit of complexity fetishisation, and it seems as though that this resulting complexity may have become invisible to you.
run0 doesn’t seem like a bad idea. but i am wincing a bit at the thought of unrestricted javascript determining access control.
I think you have me confused with someone who cares about the difference between binary and text logs. I have no pony in this race; my comment was just made to help.
With `ls -l` I can see file modification dates. How do I do it with journalctl? And why do I need to that that in the first place?
Journald would be perfect if I didn't have to know a thing about it. Work in background, move stuff from stdin/stdout to /var/log/journald/service.log, move old stuff in /var/log/journald/archive/service/2024-04/15.log.gz. I'd be happy. Why do I need this cryptic CLI I don't understand. It brings nothing but pain.
honestly that's such a niche problem to have that i dont think it's worth throwing the baby out with the bathwater. anyways you shouldn't be discovering installed services by logfile presence, it should be something like querying your package manager.
I wouldn't say it's a niche problem. I also miss being able to go to /var/log and see instantly what's going on.
There should be a command like "journalctl --top" that would list all logging services with amount of logs, oldest and latest log times, and last few lines of each, to improve that experience. Maybe there is already a way to do something similar?
I appreciate that someone's invented shoes you inflate and snap. I can see some advantages.
That said, when I'm in a hurry and going to do a thing that I've been doing every day of my life, I'm really not a big fan of having to stop what I'm doing and reexamine a tool I've used all my life to figure out how to use it. I learned this stuff decades ago and I'm not anxious to relearn how to do it for a fractional improvement, and I'm not even really 100% sure that this new "inflate and snap" shoe model is actually an improvement, except for very specific use cases.
Don’t think of it as relearning a tool, think of it as learning one tool which works for everything. You no longer have to implement your own log rotation, compression, etc. or work out which configuration each program needs to do that in the manner you expect – for example, “is it safe to rotate a log file?” requires you to know implementation details on each program.
One thing to remember is that a lot of people use the testing distribution so they would have seen it before the Debian 8 release changed the default. The debate in the community took ages but one factor in concluding it was that people were using it for years without it being anything like the more hyperbolic predictions some opponents made.
I’ve been doing this for decades, too, but I don’t miss having to deal with everyone inventing multiple log hierarchies, different rotation conventions, compression strategies, etc. not to mention all of the other things which systemd removed from daily toil around process management.
'journalctl -u service -f' takes many seconds to start showing logs (even with 4GB journal size limit which not that much). 'tail -f /var/log/service' works without a noticeable delay (be it 1kb or 100Gb log file). For me it's a huge regression.
Not my memory at all. In almost all cases the logs were in /var/log and written by syslog. Unless the whole service was in a chroot directory, or the program was written by an amateur who didn't follow logging conventions.
This and many other discussions misses the point that most people that don't like systemd have with it.
It can largely do everything it replaces, but differently, in some cases with improvements. Every year it increases its scope of 'problems' its solved.
But the issue is it hasn't _ACTUALLY_ solved a problem for me in about a decade, its only introduced problems. Its replaced things that worked perfectly for me and millions of other users for years/decades with something that... also works... but differently. Now to accomplish the _SAME_ outcome without any personal benefit I have to re-learn their new and usually opinionated way.
There was a time I could log into a system, find logs, parse dates and do everything in these discussions reletave to logs, rotation. Regardless of service because the standards were close enough and the basic tooling (tail/find/tar/grep/awk etc.) worked everywhere always, and you never had to google even if the service was one you never worked on before.
For systemd its the opposite, its almost a guarantee you have to google something along the line when interacting with new systems and services. And as soon as their newly enforced opinions and way of doing things is getting comfortable they go and replace something else that was also working... perfectly fine.
If it actively solved problems for the majority of users we wouldn't hate it so much. Instead it requires more mental overhead than anything other than XORG to deal with and constantly manage than anything else in the linux ecosystem.
Ahh the vocal minority in support makes its appearance!
I wouldn't call the opposition tiny, although the vocal portion of any group tends to be the smallest. This would be my first online comment on it so I'm certainly part of the non-vocal group you ignore, that in my experience is significantly larger than the vocal side of group you dismiss.
systemd-nspawn is actually pretty great, and is mostly just a tool that exposes functionality that systemd already needs for other reasons. IIRC it was originally developed to aid in testing systemd itself, and was initially shipped as an unsupported extra in case other people found it useful.
systemd-resolved is the only way I've ever been able to get dockerd to play nice with Tailscale DNS; it's also the best way that I've found to get a system to pick different upstream DNS servers depending on domain. The alternative is hand-rolling it with dnsmasq or something similar.
I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet, but for servers with relatively static network configurations, I greatly prefer systemd-networkd over any of the various implementations of ifupdown.
> I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet
I don't think there is one. systemd-networkd is explicitly not for dynamic environments like personal laptops or workstations; it has a scope and NetworkManager is largely outside it.
I've recently become a fan of the `-x` flag to journalctl, which adds additional context to log entries and even suggests remediations at times. I've identified and fixed several issues on my Linux systems that way. Kind of hard to get that with tail -f!
> This was a really big pain, yes, but I also remember how I could `tail -f *.logs`.
Others point out that journalctl supports `-f` and `-n` for determining how much journal to return, but for me what's great about journalctl is that I do not need to know where the program is logging to, how it rotates logs (or how it doesn't), or anything else. The program does not need special permissions to a log location in /var/log or /usr/var/log or ~/... or anything else, it just outputs using stdout and it gets captured. When I want to look at logs, I just use the same moniker that I use to control the service: its unit name. It is so incredibly nice- I do not want to go back to the bad old days of spelunking through /var/log and friends.
> --no-pager
To be clear, if you pipe journalctl somewhere, it will automatically skip the pager and just work. `--no-pager` is most useful for when you want to print some unpiped content in an automated script to avoid having a non-existent user exit the pager.
I remember, and sometimes still have to deal with, programs that _didn't_ dump their logs in /var/log, forcing me to figure out where the log are in the first place
Those old init scripts were pretty boilerplate and not that hard to write. And in the case of OpenBSD (and maybe other BSDs?) the scripts for /etc/rc.d are usually just a few lines.
With the traditional init system I knew what was going on, and if I didn't, find and grep were your friends. With systemd I'm googling everything every time. The systemctl man page is over 1,100 lines. OpenBSD's rcctl man page is 119 lines.
I'm sure there are use cases for systemd but for running a server that provides a single or a handful of services, it seems like complete overkill.
Not the parent commenter. Windows has a lot of rubbish user-level interaction which belie how good an OS NT really is. Anyone who blindly disses it because 'Windoze' and 'M$ bad' is not really worthy of any attention whatsoever.
NT isn't terrible with some very well designed systems, that is quite true. It is however good for a completely different list of reasons that *nix is. Trying to conflate the two is a pottering's errand.
I'm still trying to figure out whether I like nftables more than iptables. Both are pretty complex to manage IIRC (I've only touched nftables a couple of times though, and not recently)
I don't resent systemd for existing, but I don't want to be forced to accept Lennart's shitty design decisions and it's clear he wants me to be since he successfully agitated for other critical components to hard depend on it. I want the option of opting out, which Lennart would deny me if he could.
At the end of the day, I just run Void (or MX Linux if I want something more Debian-flavored) and get on with my life.
> systemd has been a net positive for the linux ecosystem.
You're presuming to speak for an awful lot of people there, on a topic that would be difficult to measure.
> since the beginning of systemd people have moaned
> it's quite annoying that the armchair linux experts complain
Now you're overgeneralizing, and doing so in a dismissive and patronizing way.
Here are a few examples of problems I have with systemd:
System shutdown/reboot is now unreliable. Sometimes it will be just as quick as it was before systemd arrived, but other times, systemd will decide that something isn't to its liking, and block shutdown for somewhere between 30 seconds and 10 minutes, waiting for something that will never happen. The thing in question might be different from one session to the next, and from one systemd version to the next; I can spend hours or days tracking down the process/mount/service in question and finding a workaround, only to have systemd hang on something else the next day. It offers no manual skip option, so unless I happen to be working on a host with systemd's timeouts reconfigured to reduce this problem, I'm stuck with either forcing a power-off or having my time wasted.
Something about systemd's meddling with cgroups broke the lxc control commands a few years back. To work around the problem, I have to replace every such command I use with something like `systemd-run --quiet --user --scope --property=Delegate=yes <command>`. That's a PITA that I'm unlikely to ever remember (or want to type) so I effectively cannot manage containers interactively without helper scripts any more. It's also a new systemd dependency, so those helper scripts now also need checks for cgroup version and systemd presence, and a different code path depending on the result. Making matters worse, that systemd-run command occasionally fails even when I do everything "right". What was once simple and easy is now complex and unreliable.
At some point, Lennart unilaterally decided that all machines accessed over a network must have a domain name. Subsequently, every machine running a distro that had migrated to systemd-resolved was suddenly unable to resolve its hostname-only peers on the LAN, despite the DNS server handling them just fine. Finding the problem, figuring out the cause, and reconfiguring around it wasn't the end of the world, but it did waste more of my time. Repeating that experience once or twice more when systemd behavior changed again and again eventually drove me to a policy of ripping out systemd-resolved entirely on any new installation. (Which, of course, takes more time.) I think this behavior may have been rolled back by now, but sadly, I'll never get my time back.
There are more examples, but I'm tired of re-living them and don't really want to write a book. I hope these few are enough to convey my point:
Systemd has been a net negative in my experience. It has made my life markedly worse, without bringing anything I needed. Based on conversations, comments, and bug reports I've seen over the years, I get the impression that many others have had a similar experience, but don't bother speaking up about it any more, because they're tired of being dismissed, ignored, or shouted down, just as I am.
I would welcome a reliable, minimal, non-invasive, dependency-based init. Systemd is not it.
Systems isn't blocking the shutdown. Systems is being blocked by a service. Since you are saying that you can't track down the exact service, you must be using a container runtime like docker or LXC which is not transmitting the kill signal properly to the container to shut it down. The container itself then keeps running for whatever reason. This entirely depends on your particular setup and has very little to do with systemd.
When it comes to the shutdown issue, I'd argue this is a side-effect of systemd bringing some consistency and correctness to what used to be the wild west, and is highlighting some issue that was before overlooked.
Granted, in a lot of cases the issue probably wasn't a big deal (the system is being shut down, the user is already logged out, do you really care that a system background process is being shut down cleanly?) but from systemd's perspective there's no difference between that and an actually business-critical process that should absolutely be allowed to terminate cleanly before unmounting the filesystem and powering off the machine.
> systemd has been a net positive for the linux ecosystem. remember when you had to write bash scripts to start, stop, restart services and handle any other signals you want to send it?
Systemd was meant to replace Ubuntu's upstart, not sysv init. And it indeed systemd was better designed than upstart.
Sysv init is a horrendous hack. People should NOT use that. Upstart had limitations but is okay, and certainly much better than sysv init. Systemd is okay too (but has much more features).
Also as an historical note, both upstart and system was inspired by Apple's launchd.
NixOS was an early adopter of upstart but it later migrated to systemd (alongside Ubuntu itself)
"There’s a new tool in systemd, called “run0”. Or actually, it’s not a new tool, it’s actually the long-existing tool “systemd-run”, but when invoked under the “run0” name (via a symlink)".
systemd-run is very useful to run tasks with specific cgroups settings, or at a specific time. It asks for password whenever needed.
Lennart's toots suggest they are replacing a complex SUID binary with an already existing component (systemd-run) with better workflow (service manager handling the elevated context), which sounds like a sane move to me.
It will come with `filectl` for all your file operations, so you will no longer need `cd`, `pwd`, `touch`, `rm`, `mkdir`, `cat`, `grep`, `find`, etc. Instead you do everything through `filectl` commands.
This will deprecate many commands from GNU coreutils, which is a good thing because replacing things is always good.
Then, since programs are just files, and filesystem will be part of systemd, any program you want to use will obviously have to go through systemd as well, meaning they will need to be a service unit of type `oneshot`, because this way we keep everything well integrated together.
Don't worry tho, you only write the unit files once and they work forever. The only thing you need to remember is that, instead of `cargo build` you'll need to use `filectl exec -u cargo build` (`filectl exec -u` is only 3 words, so you don't have the right to ever complain about this tiny little change).
Anyone who doesn't like these changes is stuck in the past.
> instead of `cargo build` you'll need to use `filectl exec -u cargo build`
You joke, but I already have to do this with lxc commands, and the systemd-compatible version of those commands is even longer than you imagined. See my other comment for details.
> The only thing you need to remember is that, instead of `cargo build` you'll need to use `filectl exec -u cargo build`
No, you forgot about `buildctl` which compiles any language into systemd bytecode, that runs on the systemd VM. At long last, write once, run anywhere!
> You will no longer need `cd`, `pwd`, `touch`, `rm`, `mkdir`, `cat`, `grep`, `find`, etc. Instead you do everything through `filectl` commands.
This is not as ridiculous as it sounds. Arguably, the file system is more of an exception, because it is directly exposed by the kernel. But, for example, to manage the files in a tar archive you do everything with the `tar` command and to manage a git repository you do everything with the `git` command.
And one day you boot into the single mode and can't execute a file because you you can't connect to systemd D-BUS error 0xDEADBEAF contact your systemd admnistrator
I mean, it already has replaced the shell in a lot of ways. Systemd is basically the one reason why it's feasible to banish POSIX shells to where they belong, albeit still not usually practical.
I don't understand the point of creating an entirely new shell inheriting almost nothing. Seems like that would cause a lot of issues (i.e., sudo make install)
Why ? I guess you do not get this unless you have systemd, that is fine by me. If I would use anything outside of sudo I would go to doas.
My home distro does not have systemd, so I guess I may never see run0.
I have been evaluating the various BSDs for a few years in case my distro is forced to use systemd and friends. The distro I use is fighting the good fight against large monolithic tools, but it is having a hard time trying to avoid these monolithic tools.
With Linux slowly forcing systemd, Wayland, various Desktop Environments and now this, my move to a BSD may happen sooner that later.
Curious, I wonder how close run0 is to how Windows raise auths ? With LP now working for Microsoft, is he cloning tools from Windows for use in Linux ?
This is just a complete disaster. Ever since libxz has shown how bloated and spaghetti code systemd is, now they want to make it seem like they're going to have "security" in mind. What a joke.
Let's assume for a moment that it is lower risk than sudo (which is the problem is it addressing), why isn't it also called 'sudo', designed to behave the same as the thing it is replacing, so that anyone (and any scripts) that currently use sudo can carry on and be oblivious to the security benefits this new implementation offers?
I'd instead like to see a post saying something like 'on systemd based systems, a more secure implementation of sudo is provided', and all the clever whatever it is happens behind the scenes, and frankly i'll never need to know about it.
Because there already is jq-go and jq-python. One of them is called jq in linux the other is called jq in MacOS. They are not compatible and you find out you're using the other one after 6 hours of screaming at the computer.
calling it sudo would create the expectations that all the options, config, and usage is exactly the same.
It would appear that it's _functionally_ the same, but using a different mechanism and with a new name so they don't want to be locked down to all the other stuff.
Presumably because it uses different options and other different stuff, it has a different name. However, it might be useful to have a command "sudo" which emulates the options of sudo so that you can still use the same "sudo" command on systemd-based systems as well as on non-systemd-based systems. I don't really know how well that would work, though.
Maybe you've not understood. I'm asking, why does it have different options? Is there a reason for that? If not, make it compatible, so a drop in replacement. Otherwise we're just spamming new commands at people, and the old sudo will probably end up living on alongside the new on systemd based systems just to keep scripts and the like working.
I thought this was a security fix? Let's say I find a problem in ssh, I don't update it, call it 'newssh' and tell people to use an alias to point ssh to newssh do i? I just fix the app.
This is playing on the difference between hoping that sudo does the right thing juggling setuid and capabilities, and having a strict IPC boundary between privilege levels.
It sounds like a great use of systemd, for those who want to use it.
There's like 3 components involved in making setuid safe (the kernel, the dynamic loader, and your exec), and at least one of them wasn't doing its job correctly (the dynamic loader). IPC by definition involves a superset of these components.
There's no reason to think that if you can't make a simple setuid binary safe, you can make IPC safe. IPC is an order of magnitude more involved. Specially because in order to gain any effective security you need a 3 way IPC (1st level = the client, which is completely untrusted; 2nd level = the request parser, which is trusted but runs without elevated permissions; 3rd level = the actual elevator process, which must run with elevated permissions).
It's not clear why the request parser would have to be trusted. I assume you're just speaking about the call to execve running in context? That's not much of a request parser. At the point that you tell `run0` to launch a shell, you're not calling the actual commands to the shell the request parser, right?
I also think the notion of an untrusted client is kind of a hashed out thing. As said in the post itself, `run0` is an interface to `systemd-run`. `systemd-run` as a client may be more _involved_ but it doesn't seem like that has any relevance to whether or not it's more secure. It's a separate layer for the insecurity. While sudo is a single process, if it was two processes it wouldn't all have to run as root. That by necessity means something that was previously running as root isn't, which makes it more secure -- not less, right?
The actual elevator process is systemd itself which already runs as init on every machine you'll have `run0`. But by nature it's always the top of the process tree, it seems like it's _less_ complex to have systemd-init the immediate parent process. There are fewer thing that can leak into or be inherited by the spawned process.
> It's not clear why the request parser would have to be trusted [...] I assume you're just speaking about the call to execve running in context
No, I'm talking about the part which is going to parse the command line, arguments, environment, decide whether the user is allowed the elevation or not, decide which environment, file descriptors, etc. are to be passed through, etc. All of this must NOT be in the same context as the caller, as it can simply fake all these decisions. You need to handle this from a process running in another context (suid or not).
> While sudo is a single process, if it was two processes it wouldn't all have to run as root
Yes it would ? At least one of them would need to be suid for the actual execution. But the problem is that the process which was NOT suid would be running as the same user as the caller, so by the same reason as above -- you cannot trust what it does. The only thing the non-root process would be able to do is to massage the request a bit, then forward it (IPC!) to the root/suid process which you CAN trust. We are just moving the security border, and it is not clear what would be gained by it.
In this proposal, instead of a suid binary, you have a constantly running "sudod" process (or worse, pid 1), but otherwise is the same. Everything must be IPC'd to it.
> There are fewer thing that can leak into or be inherited by the spawned process.
To have this IPC complexity just because apparently we can't figure out how to do suid without inheriting anything is bonkers.
As a trade-off you now have a user-accessible IPC system with the _gazillion_ possible vulnerabilities it entails. At least before you needed root to talk to pid1..
> As a trade-off you now have a user-accessible IPC system with the _gazillion_ possible vulnerabilities it entails. At least before you needed root to talk to pid1..
Read the linked post again. This is all already available, and always has been, since forever.
> There's like 3 components involved in making setuid safe (the kernel, the dynamic loader, and your exec), and at least one of them wasn't doing its job correctly (the dynamic loader). IPC by definition involves a superset of these components
Incorrect, because nowhere in the IPC dance are these components exposed to the same untrusted environment as they are with sudo.
Yes, they are. The kernel for obvious reasons. Second, the IPC server now has to handle (and possibly pass through data) from the untrusted environment, unless you are happy with a sudo that does not even ferry stdio. Frankly, having properly working suid (the kernel does most of the job) sounds MUCH easier than having this type of APIs exposed to arbitrary users from pid1. In fact, as per Lennart's last sudo tty bug, the issue was with how sudo was exec()ing the target binary in the _target_ context (not the original context). Having sudo as a global daemon instead of a suid exec is not going to protect you against those; actually may make them worse for all I know.
I have seldom come across unix multiuser environments getting used anymore for servers. Its generally just one user on one physical machine now a days. I understand run0's promise is still useful but i would really like to see the whole unix permission system simplified for just one user who has sudo access.
NixOS may be helping multiuser make a comeback, at least it is for me and my home servers. I no longer have to containerize my apps, i can have one baremetal server with a dozen+ services, all with their own users and permissions, and i don't have to actually think about any of the separation.
Plus there’s network shares. Multiple people in my home with linux PCs, each with their own slice of the NFS pie based on user perms. Sure, it’s not secure, but these are people I live with, not state-sponsored hackers.
All that said, I’d also love a simpler single-user perm setup. For VMs, containers, etc it would be amazing
I’m not sure how “I don’t have to actually think about any of the separation” meshes with the fact that you explicitly setup multiple users and configured file and group permissions accordingly. You clearly put a lot of thought into it.
Alternatively, containers really are a no-thinking-required solution. Everything maximally isolated by default.
Containers are isolated but a far, far cry from maximally isolated. They’re still sharing a Linux Kernel with some hand waving and cgroups. The network isolation and QoS side is half-baked even in the most mature implementations.
HVM hypervisors were doing stronger, safer, better isolation than Docker was 10 years ago. They are certainly no-thinking required though which leads to the abysmal state of containerized security and performance we have currently.
> I’m not sure how “I don’t have to actually think about any of the separation” meshes with the fact that you explicitly setup multiple users and configured file and group permissions accordingly. You clearly put a lot of thought into it.
That's the thing, with NixOS you usually don't have to explicitly setup users and permissions. For most simple services, the entire setup is a single line of code in your NixOS configuration. E.g.
services.uptime-kuma.enable = true;
will make sure that your system is running an uptime-kuma instance, with its own user and all.
Some more complex software might require more configuration, but most of the time user and group setup is not part of that.
> i can have one baremetal server with a dozen+ services, all with their own users and permissions
I've used nixos and I don't really see how nixos is special apart from the declarative config. The same can/should be done with any distro and any config manager.
And unless you were running Podman in rootless mode, the same setup applies to containers too.
Sure i could do this on debian, but like, i wont. Some software comes packaged with nice scripts to provision new users for running systemd services, but a lot do not.
For me and my home network, if the default security mode is “manage users yourself”, i chmod -R 777 on all applicable files and call it a day. Nixos lets me be lazy, as all nixos modules (that I’ve ever used) have their own user setups with minimal permissions by default
No, but that's the case I've overwhelmingly seen over the decades. Anyway, are you going to redesign ssh not to require a user, for instance? I assume you wouldn't want sshd running as the putative single user.
[I'm all for replacing notions of privileges/permissions with capabilities.]
I've never understood the need for sudo(1) on single-user, physical machines: I keep a root shell (su(1)) around for admin tasks, and it's always been sufficient.
You're entering your own accounts password, not root, when you use sudo. It's a security measure to prove our shell hasn't been hijacked and to make you pause and acknowledge your running a command that may affect the entire system.
> Everything I run with sudo is logged so I know how I messed up.
FWIW, shells have a (configurable) history file. I'm not sure how it compares to sudo's logging though. I also personally perform little day to day admin tasks (I don't have as much time nor interest to toy around as I used to, and my current setup has been sufficient for about a decade).
> Nothing worse than ansible with its “sudo /tmp/whatever.sh” which hides what it’s doing.
That's a nightmare indeed; for sensitive and complex-enough tasks requiring a script, those scripts should at least be equipped with something as crude as a ``log() { printf ... | tail $logfile`` }.
in some other systems the concept has become overloaded. instead of multiple real people as users, different software with different permissions are different users. its not a bad abstraction.
Ephemeral setups (amongst which k8s) remove that need but introduce a big load of other stuff.
Having a VPS that is managed by sysadmins (users with sudo rights, authed with keys) and on which partly overlapping "deploy" users can write to small parts and maybe do a passwordless "sudo sysctl restart fooapp" but only that, is a nice and simple setup.
I manage at least seven of these. And nothing in me even considers porting this to my k8s infra.
Edit: The reason for this setup is simple and twofold: deploy is safe and clear: deployers can be confident that whatever crap they pull, the server will churn on, data will be safe, recovery is possible. And all devs/ops having their own keys and accts gives a trail, logs and makes it very easy to remove that contractor after she did her work.
Yes, we are moving more and more towards a system of immutable deployments.
That's good! We don't patch executable binaries these days: we just compile a new one from source, when we made a change. Similarly, more and more we just build new systems (or their images) from source, instead of mucking around with existing systems.
> across unix multiuser environments getting used anymore for servers
I guess it depends on the servers. I'm in academic/research computing and single-user systems are the anomaly. Part of it is having access to beefier systems for smaller slices of time, but most of it is being able to share data and collaboration between users.
If you're only used to cloud VMs that are setup for a single user or service, I guess your views would be different.
Multi-user clusters are still quite common in HPC. And I think you're not going to see a switch away from multi-user systems anytime soon. Single user systems like laptops might be a good use-case, but even the laptop I'm using now has different accounts for me and my wife (and it's a Mac).
When you have one OS that is used on devices from phones, to laptops, to servers, to HPC clusters, you're going to have this friction. Could Linux operate in a single-user mode? Of course. But does that really make sense for the other use-cases?
you could potentially create multiple containers in that machine which are single user and give to every user who needs access. CPU/Memory/GPU can be assigned in any way you want(shared/not shared). Now no user can mess up another user.
It's not "that machine" it's a cluster of dozens or hundreds of machines that is partitioned in various ways and runs batch jobs submitted via a queuing system (probably slurm).
Isn't that just reinventing multiuser operating systems? Normal Linux already has the property that no user can mess up any other user (unless they are root or have sudo rights)
Containers rely on many privilege separation systems to do what they do, they are in fact a rather extreme case of multi-user systems, but they tend to present as “single” user environs to the container’s processes.
Nope, full usermode containers (e.g.: apptainer) run under the user's own context, and furthermore under a cgroup (if we're talking HPC/SLURM at least) which restricts the user's resources to what they requested in their job file.
Hence all containers are isolated from each other, not only at process level, but at user + cgroup level too.
I think a admin would better understand the system if there was only one subsystem doing a particular type of security and not two. Two subsystems doing security would lead to more problems down the road.
For HPC, there are two different contexts where users need to be considered - interactive use and batch job processing. Users login to a cluster, write their scripts, work with files, etc. This is your typical user account stuff. But they also submit jobs here.
Second, there are the jobs users submit. These are often executed on separate nodes and the usage is managed. Here you have both user and cgroup limits in place. The cgroups make sure that the jobs on have the required resources. The user authentication makes sure that the job can read/write data as the user. This was the user can work with their data on the interactive nodes.
So the two different systems have different rationales, and both are needed. It all depends on the context.
If we forget how the current system is architected, we are looking at two problems: First problem is that Linux capabilities are also dealing with isolating processes so they have limited capabilities because the user based isolation is not enough. Second problem is that local identity has no relation to the cloud identity which is undesirable. If we remove user based authentication and rely on capabilities only with identity served by cloud or kubernetes, it could be a simpler way to do authenticating and authorization
The primary point of user-authentication is that we need to be able to read/write data and programs. So you have to have a user-level authentication mechanism someplace to be able to read and write data. cgroups are used primarily for restricting resources, so those two sets of restrictions are largely orthogonal to each other.
Second, user-authentication is almost always backed (at least on interactive nodes) by an LDAP or some other networked mechanism, so I'm not sure what "cloud" or "k8s" really adds here.
If you're trying to say that we should just run HPC jobs in the cloud, that's an option. It's not necessarily a great option from a long-term budget perspective, but it's an option.
I wonder if they might be more common than you think. You will never see someone standing up at a conference and describing this setup, but there are millions of machines out there quietly doing work which are run by people who do not speak at conferences.
Where i work, we have a lot of physical machines. The IT staff own the root account, and development teams get some sort of normal user accounts, with highly restricted sudo.
> No corporate IT department is timesharing users on a mainframe
Not a mainframe perhaps, but the sentiment besides is flat wrong, because that is how Citrix and RDS (fka Terminal Server) do app virtualization. It's an approach in widespread use both for enterprise mobile/remote access, and for thin clients in point of sale or booth applications. What's more, a *nix as the underlying infrastructure is far from unusual.
I know of at least two financial institutions that prefer this delivery model to manage the attack surface in retail settings (there is a argument sometimes made that it's cheaper; it isn't, because TCO of end-user terminal infrastructure in enterprise scales with licensing and support ratios).
You don't run any services where more than one person shares responsibility for managing that service? E.g. kubernetes. That is just one guy holding it up?
That's convenient but doesn't scale and really not too great for security for a bunch of reasons, but it can work great for smaller teams and minimize friction.
Technically not with virtual machines as the hardware is shared, though I agree, nowadays accounts and access control of the system belong to the virtualization layer below. The benefits of multiple accounts per machine are tiny and not worth the complexity for server setups.
We could significantly simplify things by getting rid of the account system. The same could be said for a lot of systems like database servers. Typically it's just one database, one user (your application server) with full access. The account system is mostly an annoyance.
For big company use cases where you want to reduce attack surface, why not spawn a second server with different credentials? Anyway big companies typically have many database servers in a cluster and the same credentials are shared by many server processes... The tendency there is literally in the opposite direction.
>> Typically it's just one database, one user (your application server) with full access
This is a terrifying way to access databases.
Super user, A Modify user (just below super but cant delegate rights) for schema changes. A read/write app user... Probably a pile of read only users who, have audit trails... You might want some admins or analytics users (who have their own scheme additions).
The words security and audit trails all spring to mind.
A simpler solution is to simply not give direct access to the database to anyone who doesn't own a large stake in the project. Expose it via a more restrictive CRUD interface with access control in the application layer.
The humans are now spawns of multithread shells and other things. Linux land is still very multiuser oriented. But it is the rise of the mschines instead.
Yeah but elevated permissions may be needed from time to time anyway. Either on the client, the baremetal server or the container. Running everything as root is even for containers not recommended. Considering how popular these have become, it's a bit of an irony that systemd isn't available on the container without considerable detours.
We use Userify which manages multiple user logins (via SSH) and sudo usage.. there are definitely many, many use cases for teams logging into remote servers, and most security frameworks (PCI-DSS, HIPAA, NIST, ISO 27000) require separate credentials for separate humans. Sudo has some issues, but it works very well and is well understood by many different tools.
It could all be simplified and map one to one to your identity provider credentials at a higher level. Having a complicated user system on the servers makes it a problem.
In fact, if factotum were implemented on Unix along with an analogue to the Plan 9 capability device, venerable programs like su and login would no longer need to be installed ‘‘setuid root.’’ — https://plan9.io/sys/doc/auth.html
In fact, according to sys/src/cmd/auth/login.c it looks like once you've logged it, you can shut the door using the capability device so then it's game over, no more hostowner for you
I haven't followed Plan 9 for ages, but I'm puzzled why Cox & co wrote "Plan 9", then. However, the point was more about the capability-oriented security in a Unix successor, and how you can use file handles as a sort of cabability without the global namespace. (They're often quoted as examples capabilities in POSIX, but that's ignoring the global namespace.)
So, you don't like a component because you ran software which requires that component, and you intentionally ran it in an environment without that component.
> The popularity of systemd encourages people to require it, which is the major problem
The usefulness of systemd encourages people to require it. People require it in cases where either there isn't an alternative, the alternative isn't maintained, or the alternative is missing functionality.
Self perpetuating growth.
Systemd integrates many functions so people default using it and add even more functionality that bring even more people into ecosystem.
Which is basically how every tech ecosystem works.
The problem is that linux is supposed to not be just_another_centrally_controlled_ecosystem, so when systemd abuses their popularity by enforcing whole ecosystem (rather then cut itself into separate pieces), that is worrying
It seems like you're attributing to perceived malice ("enforcing") what is better explained by a combination of trying to get things done and not prioritizing replaceability higher than the benefits of integration. In other words, a project that simply has different priorities.
It turns out that many other people also like seeing stuff get done more so than they care whether the project in question is trivially replaceable.
libc has multiple independent implementations, if you don't like one you can switch to another. Software that hard-depends on a specific libc is indeed to be despised.
The energy bandwidth of a charger as opposed to a gas pump is a huge issue right now.
Gas pumps can easily pump up to 10gpm [1], corresponding to 300 miles of charge in 3-5 minutes as opposed to Superchargers, which take 15 minutes to charge enough for 200 miles of driving[2].
When this is improved, it will make BEV's much more acceptable to many people.
Offtop: long mastodon/X threads are so inconvenient that I would not even consider it a use case of such platforms. Write a blog post and link it there, ffs
I am continually baffled at people's use of X-Twitter/Mastodon for this kind of writing. Thankfully I've found mastoreader, which is an equivalent of threadreaderapp.
I'm not a fan of sudo. It's does so much it needs BNF to describe it's configuration format. Who knows, maybe replacing the configuration with polkit is a good idea. Still it's a stand alone binary with one clear job to do, simple enough that one person has no trouble getting their head around it so it's not surprising it hasn't had too many problems over it's long life time.
This made me smile:
> sudo has serious problems though. It's a relatively large SUID binary, i.e. privileged code that unprivileged users can invoke from their own context. It has a complicating configuration language, loadable plugins (ldap!), hostname matches and so on and so on.
That is a bit rich coming from the author of systemd, which must be in the running for one of the largest bodies of code that must run as root. It's also a very complex piece of code. That complexity is the reason I was completely flummoxed by interactions between systemd and dll's being exploited by the XZ utils hack to attack an unrelated and uncompromised binary: openssh. Run0 is just an extension of that ball of mud. It's a stretch to believe it will be more secure than sudo in the long term, which is amusing because it appears Lennarts primary argument is it will be more secure.
Regarding your first point: some people agree, most notably the OpenBSD people who did something abput it and wrote “doas” as a replacement; which fits the most common use-cases of sudo without fanfare.
And, as LP points out, fails to solve the actual problem because it's still locked into the exact same flawed Unix model, and refuses to integrate with anything else in the system to get things done in a better more systematic way. It's just a slightly refined version of the same tired old Unix way
Feels a bit disingenuous to use that against it, especially when including half the system libs has caused one major and highly publicized security issue already
What a tiresome response, but here, I'm feeling in a loquacious mood, so I'll humor it.
First, what else would I use? I want an open source, customizable operating system that is compatible with the vast majority of developer tools and build systems and Tool chains, and that is, without a doubt, linux. I don't have a better option from a practical standpoint. And even from a design standpoint there simply aren't any better options, because unix killed off all of the other good alternatives, and now it's essentially the least worst option we have right now.
What else have we got?
Windows? Windows and key, by way of open vms, had some very interesting ideas that I think Linux would have done well to imitate, like having a Common Language runtime that lets you directly interact with the objects and functions of every application and code base on your system and communicate with them in structured data, but we are beginning to get that with dpus and things like Nu shell, and windows failed its own promise there and has a vast panoply of its own flaws. Not least being Microsoft. So that's not really better at all.
Furthermore, I was never intimating that everything Unix did was completely irredeemable and the entire system we've spent all this time and every building should be thrown out completely in favor of something entirely new. Instead I was responding to the failure, the refusal even, to develop this amazing testament to open source development and the offering system design sensibilities that drive it, among a certain crowd.
I think we can build an excellent new system on the bones of the Unix systems we have now, if nothing else because it's what we've got to start with, and it did sort of have some interesting ideas, I just think it requires actually thinking outside of the box and reanalyzing our dogmatic adherence too tired old traditions in light of everything we've learned and everything that's changed in the interim, and the good ideas from other operating systems that we can borrow, instead of just repeating the same ideas over and over, a flaw in our modern notion of operating system development which has depressed even the creators of Unix themselves.
I don't think Plan 9 was the right move exactly — I think it actually recapitulates many of the flaws and fallacies of unix, such as the idea that everything being a file is particularly useful, or that we should use plain text designed first for people to read and only secondly for a computer to process, in idiosyncratic formats, instead of structured data — but if even the original creators of Unix believe that it should be transcended (Dennis Richie), and that the original philosophy is dead (Rob Pike), maybe we should learn something from that?
Maybe it's actually okay to look at what other good ideas there may be in the problem space instead of being myopically, slavishly devoted to a sad old religion.
> Linux won on Unix merits.
First of all, even if that was the case, that doesn't mean it can't improve? Just because, let's say, it's Unixy design was better than anything else that was on the market at the time, and so it won out against the other options that were available, doesn't mean it is the best possible option — that it can't be improved or developed, or even that the philosophy motivating it wasn't fundamentally flawed and broken in some ways; all it means is that whatever improvements could be made, or whatever was broken in the philosophy to begin with, was less broken or less bad than what happened to exist in competition with it. You're making the exact same mistake that people with conservative tendencies in life me and insisting that just because something work in the past, for whatever reasons it did that, it can't possibly be developed or improved upon. You aren't even saying if it ain't broke don't fix it, you're saying if it wasn't literally the worst, don't fix it.
In fact I don't even think Unix won out in the server, workstation, academic, scientific, and supercomputing Realms as much as it did because of Timeless technical merits and design decisions based on Eternal truths, as much as the purveyors of the art of Unix would like us to think that, but instead because it happened to have features that happened to be extremely relevant merits in the particular time and place and context, technologically and economically speaking, that it came about it. Namely, that it combined extreme minimalism and portability with a reasonable level of composability, even if the composability it offered was awkward and half baked. It went out mostly because a proper level of composability and integration, using defined IPC and structured data and well integrated families of tools with non-turn complete configuration languages was just too difficult for the hardware of the time, and so the choices were either on optic system with very little composability and flexibility and programmability, or something like unix. But that isn't the case anymore. We can have our cake and eat it too if we choose to, if we choose to not get sucked in to the North Korea like UNIX cult.
> Windows NT, by way of OpenVMS so I've heard, had some very interesting ideas that I think Linux would have done well to imitate, like having a Common Language Runtime that lets you directly interact with the objects and functions of every application and code base on your system and communicate with them in structured data
The CLR is Microsoft’s answer to the JVM. It is not among the ideas Windows NT inherited from OpenVMS. It is not part of the core OS. Most apps don’t use it. With the advent of WinRT, there is arguably less of a push for its use than there once was.
And when you talk about “communicating with structured data”, I believe you are talking about PowerShell - which apps don’t participate in by default, you need to go to the extra effort of writing PowerShell cmdlets for your app, and many developers don’t. You have to write them in PowerShell itself, or else a CLI language such as C#, F#, C++/CLI, etc. So if your app is written in C or (standard) C++ or Go or Rust or Python or Java or JavaScript or Lisp or whatever, integrating with PowerShell requires adding a new language to your project - no wonder many don’t bother
- what matters is composability (ie programs, possibly running from different computers, being able to use each other's output)
- simplicity is required for composability. This the structure less file system and the "everything is a file" design concepts.
The same merits as HTTP in the realm of protocols.
The appeal for a stricter and more principled approach are understandable, and the road one end up favouring largely depends on one's early experience I believe.
I, for one, grew up in a world where I could use either a resilient Linux system that could run a treasure trove of programs written long ago for different machines, or a more modern and principled OS (windows) where a small glitch in some non essential video driver would frequently bring down the whole thing. I went for linux, it won maybe for its merits, maybe by chance, and now I'm concerned that a whole encompassing systemd will eventually bring us back where an unrelated glitch, or backdoor, in an unrelated non essential part will bring down the whole system.
Isn't that link showing the opposite? That sudo is really large and systemd isn't? They even compare systemd with wpa_supplicant and it turns out they are the same size.
Not really. I'm not going the effort of breaking it down like he did, so just looking at total lines in the source calculated with "wc -l $(find . -type f)":
I find your comment here sadly indicative of the level of discussion around systemd from its haters.
Your metric is utterly misleading not only because you're almost certainly counting non-code files, but more crucially, because the systemd repository contains the code for *sixty-nine* entirely separate binaries, separate tools under the overall systemd project umbrella, so counting their collective code size as if it is the code base of one single gigantic tool, a single gigantic program that compiles into a single gigantic binary, with all the interweaving that implies, is just disingenuous at best. That number does not represent the size of a single attack surface area, and pretending it does is nonsensical. It's like pulling in the source of all of gnu and doing a line count. And for the record, I've actually done this properly, cloning the systemd repository and then reading the documentation to figure out what directories and source files did what, and assembled a list of the directories and source files that represented just systemd-the-init-system, and got about 240,000 lines of code.
And remember, it won't actually be systemd the init system that will be replacing sudo, it will be systemd-run, which is a separate binary, with a separate memory space, and a separate permissions model, that merely communicates with systemd the init system to get certain things done. I guarantee you it's probably smaller than the code base of sudo, and this architecture, as LP points out if you actually read his thread on mastodon, far better represents the methodology of running things with at least privilege and privilege separation and so on, because instead of having the binary that is called from unprivileged space managing transitioning itself into privileged space and then doing things, instead the binary always stays unprivileged and just communicates via a strictly defined IPC protocol that gives it no direct abikity to do anything with a process that was already privileged instead, that can decide what to do on its own.
Let me leave you with this quote:
> If you build systemd with all configuration options enabled you will build 69 individual binaries. These binaries all serve different tasks, and are neatly separated for a number of reasons. For example, we designed systemd with security in mind, hence most daemons run at minimal privileges (using kernel capabilities, for example) and are responsible for very specific tasks only, to minimize their security surface and impact. Also, systemd parallelizes the boot more than any prior solution. This parallization happens by running more processes in parallel. Thus it is essential that systemd is nicely split up into many binaries and thus processes. In fact, many of these binaries[1] are separated out so nicely, that they are very useful outside of systemd, too.
> A package involving 69 individual binaries can hardly be called monolithic. What is different from prior solutions however, is that we ship more components in a single tarball, and maintain them upstream in a single repository with a unified release cycle.
- Non code files were counted for both sudo and systemd. That's because I'm lazy, not because I think it influences the result one way or the other (I don't what effect it would have).
- Separate binaries are not separate logical entities. Pointing to separate binaries is a misdirection. systemd is a set of binaries cooperating using RPC (dbus) to yield something bigger than any single binary. The biggest hint they are interconnected is they are in one source ball for a reason: so these binaries can share a lot of code and interact in complex ways.
As for systemd-run being "a separate binary, with a separate memory space, and a separate permissions model", those things didn't protect openssh being hit with the XZ Utils hack via systemd. An unexpected interconnection is more than enough. systemd abounds with interconnections. Being tightly interconnected often gives you greater functionality, and in the case of Run0 I suspect that is going to be a big win because systemd has a lot of process isolation mechanisms build in. But complex interconnected systems achilles heel is security, and that's definitely true here.
> Separate binaries are not separate logical entities.
They're separate programs, sometimes essentially completely independent, sometimes sharing almost no code at all. For example, systemd-init does not have dependencies on all of the other programs, or vice versa. You can use the systemd-boot bootloader without using systemd's init daemon. You can use the systemd init daemon without using the systemd-boot bootloader.
Having separate release tarballs isn't some special distinction that makes things more "logically separate".
Besides, isn't this a goalpost shift of epic proportions?
> That is a bit rich coming from the author of systemd, which must be in the running for one of the largest bodies of code that must run as root
The point in indicting the size of systemd was the code that runs as root, but the wc line count is counting tons of stuff that not only doesn't all run as root, but some of it doesn't even run under Linux.
> those things didn't protect openssh being hit with the XZ Utils hack via systemd
A lot has already been said about this before, but the systemd library that they included in the Debian and Fedora patches to OpenSSH is by far one of the smallest surface areas of any of the runtime dependencies. A bit ago, when I ran libtree on sshd, I got this:
And that's just what you can see by looking at the shared objects, there might be more at runtime.
You would have to ignore mountains of rationality in order to come to the conclusion that systemd was remotely reasonably "at fault" for what happened with the xz incident. Not only that, even prior to the xz incident, systemd had already fixed the problem that lead to the xz exploit, it just wasn't shipping in Debian or Fedora yet; that's most likely why the xz backdoor had to be rushed into the next releases in the first place, because the window of opportunity was closing.
> You would have to ignore mountains of rationality in order to come to the conclusion that systemd was remotely reasonably "at fault" for what happened with the xz incident.
There's a fairly comprehensive thread about this (1), and my take away from it was not 'this had nothing to do with systemd'.
You can argue where blame lies for it, but that it involved system-d, and would not have been possible without system-d is not in dispute.
Would something else have been the cause if it hadn't been system-d? Maybe. ...but let's not play the 'what if game', because it's stupid. It was system-d.
You're stepping from 'lets argue about system-d being big and clunky' into 'nothing is ever wrong with system-d we cannot tolerate critique!' here.
> The point in indicting the size of systemd was the code that runs as root, but the wc line count is counting tons of stuff that not only doesn't all run as root, but some of it doesn't even run under Linux.
yeah fair.
...but, come on. We're arguing about the minute (irrelevant) details of a much larger point being made here.
System-d is a big complex system; it is kind of funny that something that is big and complex and part (undenyably) of a major recent security scare is part of the plan to replace sudo to be more secure and less complex.
I believe the intent is good, and the result will probably be better than the mess that sudo is now; but that's because sudo is a mess (or I personally think so anyway), not because system-d is simple, or easy, or somehow fundamentally more secure.
If you can't see the irony here, you are dancing around with your hands over you ears.
Why do you feel the need to insert this personal insult into an otherwise impersonal argument? This infuriates me. I'm not sitting here inflamed with passion to argue about systemd because I think it is great. I'm sitting here arguing about systemd because I feel someone was wrong on the Internet.
If you want to ascribe me as some "systemd apologist" because I am willing to "defend" systemd when I feel it is justified, then so be it, but it's wildly inaccurate, and framing it this way is biased in and of itself. It implies some kind of personal identification with things like init systems that I don't have. I have strong opinions, but I have those about everything.
> There's a fairly comprehensive thread about this (1), and my take away from it was not 'this had nothing to do with systemd'.
That's an enormous thread, if it really has a substantive bit can you please at least point to it? There's no chance I'm reading another thread about the xz incident.
> You can argue where blame lies for it,
Which is what I did do
> but that it involved system-d, and would not have been possible without system-d is not in dispute.
FWIW it would have been possible to backdoor Linux via liblzma5 without sshd linking to it. The actual thing that makes the sshd backdoor so incredibly stealthy is a combination of the fact that it is done in such a roundabout way (through an unexpected dependency), in a way that it would only be activated by downstream patches in distros and not upstream code. That last part is really nifty for avoiding detection.
There are still plenty of places where liblzma5 is used, with root privileges even. For example, many package managers use liblzma5. It certainly would've been possible to install a backdoor into sshd through that vector.
People are reading a bit much into the systemd link, but even if you won't accept that backdooring Linux would've been completely doable through liblzma5 without systemd, I would still contend that linking to liblzma5 is honestly a completely reasonable thing for a system library to do; heck, sshd itself already links to zlib. While 0 unused dependencies are better than 1, the picture people paint is that systemd recklessly linked in a bunch of libraries which is the only possible way a compromise of the xz project could've backdoored Linux, but the reality is a lot more nuanced than that.
> You're stepping from 'lets argue about system-d being big and clunky' into 'nothing is ever wrong with system-d we cannot tolerate critique!' here.
That's neither where this argument started nor where it is. Nothing in my single reply in this thread even remotely suggests the latter. It is a direct confrontation of two specific points.
> ...but, come on. We're arguing about the minute (irrelevant) details of a much larger point being made here.
> System-d is a big complex system; it is kind of funny that something that is big and complex
sigh. Now we're all the way back to square one. systemd is not a monolithic piece of software, it is a suite of programs.
> and part (undenyably) of a major recent security scare is part of the plan to replace sudo to be more secure and less complex.
I don't see this ironic at all considering that you don't need a bunch of roundabout logic to find security issues in sudo, they happen fairly regularly.
> I believe the intent is good, and the result will probably be better than the mess that sudo is now; but that's because sudo is a mess (or I personally think so anyway), not because system-d is simple, or easy, or somehow fundamentally more secure.
> If you can't see the irony here, you are dancing around with your hands over you ears.
Well, trust me when I say this, but I can tell you right now that I am witnessing a lot of irony.
Em, that seems an extremely generous comparison, where did you come up with that? Last I checked for example systemd relies on polkit for policies, which drags in a javascript interpreter engine. If the author thinks BNF is complex…
sudo: what, the problem is its attack surface is too big? Nothing like systemd's.
Right now systemd, PID 1 won't finish on a Debian 12 (upgraded) which I am slowly debugging. Neither journalctl nor syslog record messages that flash on the screen. Systemd goes into a dark-flash-dark loop. The developer should focus on getting systemd to work right before pointing at others' supposed deficiencies. Software need to handle error conditions and keep on plugging, especially if it replaces init.
I initially had the same reaction although having read the whole thread, the author convinced me that SUID is in fact a fatally flawed architecture and that run0's client-server architecture is correct.
Over the years I've switched from various cron daemons (anacron, cronie), sysloggers (r-syslog, syslog-ng), network managers (netifrc, NetworkManager) even ssh servers/clients (dropbear, openssh), and init systems (sysvinit, openrc) and never have I felt the need to switch to systemd despite reading some of Lennart's posts. I've used Gentoo over the years, maybe that's why.
Doas is available on Linux as a sudo alternative, I think I'll be trying that next, though I've only a limited amount of SUID binaries on my system to being with, and don't need sudo's extra features.
For the past 30+ years I’ve been the maintainer of sudo. I’m currently in search of a sponsor to fund continued sudo maintenance and development. If you or your organization is interested in sponsoring sudo, please let me know.[0]
>I’m currently in search of a sponsor to fund continued sudo maintenance and development. If you or your organization is interested in sponsoring sudo, please let me know.
His positions are mostly controversial because he challenges the way things have been done for a long time. Whenever he presents some new idea/architecture my first reaction is often confusion. Why would he change something that has worked so well for such a long time? But then I take the time to read up on the reasoning behind his ideas and then things start to make sense. Even when something isn't exactly broken, there is still room for better solutions.
There have been lots of suggestions for how to improve linux / unix for a very long time.
The first great war I remember, and I'm sure there were more before I was around, was DJB vs everyone. For the most part, I think his designs, "weird" as they were / are, are still better than almost every crackpot variation of them that's come since.
DJB is a genius, responsible for all of the non-NSA asymmetric cryptosystems, symmetric cryptosystems, and authenticated encryption algorithms supported by TLS (curve25519, chacha20, Poly1305). He's also the one who got us off of the footgun-by-design, broken-random-number-generator-will-spray-your-privatekey-everywhere nondeterministic nonce signature schemes prior to Ed25519 (the first standardized signature scheme which required deterministic nonces). Oh yeah and the only post-quantum cryptosystem that OpenSSH was comfortable shipping.
And pottering gave us pulseaudio. The gift that keeps on giving.
What I said was that the first holy war of unix I remember is the DJB vs everyone else.
As far as I can tell, as odd as DJB's designs may have seemed, they were and are ... way better than what was and still hold up today; most of the following "lets unix better" designs seem to just adopt some of DJB's designs, typically poorly.
Systemd certainly seems to have cribbed elements of daemontools et al, but seemingly none of the notion of "least privilege" ...
The great thing about unix is that there are no "wars" over these things, because everybody gets to decide for themselves.
Well at least that's how it was before systemd -- and all of DJB's unix work long predates systemd. By the time systemd came around DJB had been focusing on ECC exclusively for almost a decade.
The way I remember it is that most people didn't understand DJB and just kinda ignored his work, while a bunch of other people recognized what he was on to and integrated his ideas into software with frendlier user interfaces. For example, runit, which is PID1 for Void Linux to this day, and s6, which is PID1 for both Liminix ("NixOS-on-your-wifi-AP") and Spectrum ("Qubes for Nix"). Indeed increasing numbers of NixOS users are ditching systemd for s6.
Anyways I don't remember anything close to a "holy war".
The "people hate him because he makes new stuff" narrative is just more ego-protecting cope. Many developers of other new systems are widely respected and appreciated because their stuff works and they stay humble. Wireguard and Pipewire devs don't get hate poured on them in HN discussions because their shit works, solves problems people have, and because they know how to deal with people.
1. He gave a clear reason why it is how it is
2. He realizes it is/might be frustrating
3. even `adduser` will not allow it by default
4. The issue that it still runs the unit even with config errors has been addressed: https://github.com/systemd/systemd/commit/bb28e68477a3a39796... (~2 weeks after the issue was opened)
useradd accepts it. Since they fixed it, and the fix references that issue#, why is it still labeled "not-a-bug". Because Poettering can't be wrong and he has people like you to carry water for him.
5. useradd does allow it (as noted in a comment). 6. Local users, and the utilities that create them, are not the only source, there things like LDAP and AD.
His reason, although clear, is also plainly wrong. Such usernames although bizarre may be encountered by SystemD so it shouldn't break when it sees them. Computer programs, particularly important ones, should be conservative in what they emit and liberal with what they accept and that means not breaking when they encounter weird but technically permissible usernames. His response should have been "Golly, that's a weird username, I didn't think that was possible" and then fix the bug.
There is a certain personality type that likes to reimagine that their original thinking was not flawed, even when presented with a detail that they did not incorporate into their original thinking. If the detail had been in their awareness from the start, they would have arrived at a different position, but they are bound to a strict sense of linearity for reasons inexplicable to me except for ego protection.
Disclaimer: I know nothing about the particular bug.
Postel's Law has its tradeoffs, and its fuzzy lines are a nice place for security issues to arise.
For sure, there are limits. In this particular case, maybe we say that SystemD shouldn't support weird usernames beginning with numbers, but the other half of the law should still apply. The conservative emission would be logging an error message, not running that unit file as root.
Alternatively, if, like he says in the comments of that bug, he really means that SystemD shouldn't support systems that allow such usernames, then he should ensure SystemD won't run on such systems.
Silently doing the wrong thing is not a good thing, especially when "doing the wrong thing" is running stuff as root that wasn't supposed to run as root.
It does become a problem when you have a system service developer who
thinks the universe revolves around him, and nobody else matters, and
people sending him bug-reports are annoyances that should be ignored
rather than acknowledged and fixed. At that point, it's a problem.
But even then, system service developers don't try to 'own the whole world' so to speak and so they do need to play nicely with others. Mr. Poopering philosophy is the minute a dependencies maintainer becomes a thorn in his side - he absorbs that project into systemd. The distribution packagers follow like starving dogs on a hunt
Interesting! IANAL, but I think this should be basic functionality, ever since the recent-ish European and Californian privacy regulations. Although I think a quick e-mail to hn@ycombinator.com would suffice.
Even if you delete your account, it wouldn't really matter that much. Whole HN is probably crawled and archived on a daily basis due to a simplistic API
And another of the systemd devs, Kay Sievers, was banned from contributing to the Linux kernel due to his bad attitude and unwillingness to collaborate.
Poettering and Sievers are skilled devs with huge egos.
First of all, I like the idea. But I have questions.
What does logging look like for this?
I don't think it would be too difficult to log commands run with polkit, but is there an equivalent of sudo I/O logs? My guess is there isn't now, but to fully replace sudo it will probably need a way to record everything on the ptty it creates.
What environment variables does it forward by default? From the man page it sounds like SHELL is. What about TERM? Any others? What environment variables are set? What is PATH set to?
How are signals handled? Will a signal sent to the run0 process be propagated to the priveleged process?
What about sudoedit? How would I achieve similar functionality with run0?
Can someone explain what this is / how it works to someone who has done a considerable amount of programming but lacks this kind of operating system level knowledge?
I was under the impression that ‘sudo’ was baked into the entire system. Like ‘cd’ or ‘ps’. How exactly can you just swap out sudo? Does that involve swapping out chmod as well?
sudo, and even cd and ps you mention are simply binaries that come shipped with your distro / OS.
They, like explorer.exe on Windows, are an essential part of that OS with special privileges and roles but they are not part of the kernel, they are still simply programs. It is not developed by the people who develop the Linux kernel.
There are other Sudo alternatives such as DoAs already.
While some systems include a "cd" binary, it's basically useless since it just changes its own working directory and then exits.
Instead, "cd" commands are generally parsed and executed by your shell (/bin/sh or similar) directly so that the shell's working directory gets changed and you can run subsequent commands in the new location.
"ps" on the other hand is indeed just a normal program. Usually it reads files in /proc to figure out which processes are running.
Not only that, but it became commonly included only about 20 years ago.
I spent my first years with Linux calling ‘su’ instead.
I still run some very old distribution (e.g. RedHat 6.2) on a Pentium 1 laptop, and I downloaded the source of sudo and compiled it on it, since the sources were not even included in the extended CD set.
sudo is just another program, it's not "baked in" as such. It just ships with many (not all) distros. I remember having to install it from repos in the past on a new system: https://github.com/sudo-project/sudo
Reminds me of an assignment I had at uni to implement sudo in C. You get elevated permissions using the setuid bit, and can verify the password of $USER against /etc/passwd.
1. Parses the sudoers file to check if the current user can run the command provided.
2. If so, authenticates the user using PAM.
3. If both those pass, sets the user id to root and runs the program.
There is nothing special about it. All steps can be done by any program. In fact sudo is usually not even an installed by default package in many systems.
The only seeming magic bit is part 3, where the program sets it's user id to root. Obviously if any program could do this... That'd be unsafe.
However, unix systems allow any executable file to have their flags changed to include the setuid bit which causes the file to execute with privileges of the files owner. You'll notice that the sudo binary has this bit set and it's owned by root, which explains now the entire process.
The setuid part is where things get exploitable. Pretty much any exploit to date was abusing a library's assumption about what to sideload or what to persist after it got root rights for whatever reason.
"Good distro maintainers" (e.g. Arch Linux) try to minimize the attack surface by modifying their packages to use the capabilities flags instead (e.g. the net cap flag for ping binary).
"Bad distro maintainers" blame the end user for their own responsibility for letting this happen. They could have just uninstalled the program, right?
Well, I disagree. SystemD's new approach is that they try to reuse the seccomp sandboxes they've introduced for a while now, where root rights-given processes can even be executed in a chroot, with a fake /etc/passwd file, with fake users, with fake /dev ices etc.
As SystemD as the process #1 always has to be executed as root, I think it's a good thing that they try to offer a sandboxed alternative. Polkit is just so damn ugly with all their hacky subscriptions and policy files. If you disagree with me, I recommend you to learn more about privilege escalation exploits on POSIX systems, and how PAM, Polkit and pretty much any auth framework always said it's the users fault.
LD_PRELOAD auth bypasses are now more than 18 years old, and the CVE still works on enterprise-grade linux distributions. It literally was the reason muslc was created as an alternative to glibc. And that's older than a decade now.
Check out https://gtfobins.github.io (or the Windows NT equivalent LOLbins) if you wanna know how many binaries there are as an attack surface.
Honestly though, elevating privilege is inherently the exploitable part of any program that attempts to cover this space.
I would guess you are arguing in a reduction in complexity, which is semi-ironic as run0 is including polkit (and thus: an entire JS interpreter) among other things; while at the same time doas/sudo-rs exist as simpler implementations.
AFAIK, lots of the bloat of sudo is preventing a lot of the attacks you mention though, but I seriously doubt any privilege escalation system doesn't have any weak parts that need heavy scrutiny.
I agree very lot with what you wrote and I think what annoys me the most is the unpredictability of side effects, where maintainers of libraries push debug environment variables or similar hacks into their production libraries.
As long as this keeps happening due to the concept failure of how shared libraries are used for both active development and runtime execution, there won't be an easy fix available.
The stubborness of C++ developers not seeing that there is two kinds of different users, devs and endusers, is what also annoys me a lot. I understand their intent to make development and debugging easy.
But honestly, that should not be the job of the developer of a library and rather be part of the development toolchain. If the toolchain cannot fulfill this need, then the concept of binary distribution itself (aka FFI/C ABI/ELF) is wrong and needs to change.
The "never change C" mentality is what got us here, where hundreds of developers rewrite everything in Rust/Go/Zig, hoping that there is an end in sight...only to realize that at some point they have to build a different OS from the ground up to actually be able to really fix it.
> I was under the impression that ‘sudo’ was baked into the entire system. Like ‘cd’ or ‘ps’.
Neither of these are “baked into the entire system” for any meaningful sense of the word. `cd` is just a shell builtin and is casually reimplemented in every single shell. It’s just environment state within your session. `ps` is just a binary that (on Linux) parses stuff in /proc.
None of it is magic. Neither is sudo. It’s just a binary like any other (though in sudo’s case, it’s setups, which is how it can cross the permissions boundary).
Any linux process can run with elevated privileges (ie: as root) by setting a specific permission bit - https://en.wikipedia.org/wiki/Setuid. This is used for many things like ping and sudo.
Instead Run0 is using systemd to elevate privileges.
There is a lot that could be said, but suffice to say you can have both sudo and Run0 installed. So even if a Distro ships Run0 by default, you can always manually install sudo.
> I was under the impression that ‘sudo’ was baked into the entire system.
There's a capability baked into the entire system named 'setuid' which allows certain binaries run by a user to access things as if they were root.
For example, when a user changes their password using 'passwd' that executable gets special write access to the file containing hashes of all users' passwords. The system's security relies on passwd being coded carefully enough that it won't let one user change another user's password, no matter what input they give it.
sudo is "just" a setuid binary, which checks if the user is allowed to run things as root and if so uses its power to run them. It can be replaced.
There are a bunch of design implications resulting from the way setuid works - for example, the operating system has a special setting so if you plug in a USB drive containing a setuid binary, the setuid bit on it gets ignored. So you can't make a special version of sudo at home which doesn't check permissions, then take it to the school computer lab and have it work there.
> There's a capability baked into the entire system named 'setuid' which allows certain binaries run by a user to access things as if they were root.
It's actually "to access things as if they were the owner of the binary" (which usually is root, but that's not required).
The problem with that, is that other than the uid, the program inherits everything like a normal program. Environment variables, current directory, open file descriptors, and so on. If the program (and the dynamic linker it uses, and any library it uses including the C standard library) is not very careful, it can be tricked through these inherited things to do unexpected actions while being able to access things as if they were root (or whoever the owner of the binary is). For instance, some environment variables tell the dynamic linker to load extra libraries, or to change from where it loads libraries; these have to be ignored when running as a setuid process.
That "hack" uses reptyr to attach to the existing pty, which requires ptrace permissions.
The same "hack" can be done against sudo if you ptrace attach to the shell that started sudo.
This isn't a new issue. It's well known that if 'user1' has ptrace permissions, they can ptrace other processes for 'user1', and thus 'user1' can compromise 'user1'. If 'use1r' is also running sudo or run0 or anything else sensitive, it follows that the thing in the tweet is possible.
This would be an issue if 'user2' could take over 'user1's pty or such.
To what end? It seems Redhat made a thing that solved a lot of their problems as distro maintainers and then made it OSS. This "control" doesn't really buy them anything.
> Or in other words: the target command is invoked in an isolated exec context, freshly forked off PID 1, without inheriting any context from the client (well, admittedly, we do propagate $TERM, but that's an explicit exception, i.e. allowlist rather than denylist).
I think in practice, this is going to be an endless source of problems, so much so that it won't be adopted. The usual use case of sudo is that you have a normal shell command, making use of the environment for context in all the ways that shell commands do, but it doesn't have all the permissions it needs, so you add "sudo" as an adverb.
Sometimes it makes use of environment variables. Sometimes stdin or stdout is redirected to a file, or to something more exotic than a file. Sometimes that means it runs inside of a chroot, or a Docker container. Sometimes you care about which process group it runs in.
And sometimes the thing you're running is a complicated shell script or shell-script-like object, eg "sudo make install". In this case, you don't really know what its dependencies are. In fact this is a common enough case that, if run0 becomes widespread, I expect it'll have a flag or a set of flags that make it act exactly like sudo, and I expect people to wind up learning that they should always give run0 those flags.
And I'm kind of worried that when this breaks stuff, the systemd project is going to push forward with some plan to get rid of sudo, and not gracefully accept the feedback that this is breaking things. I'm particularly worried about this because of the whole saga of KillUsersProcesses breaking nohup and screen, which to my knowledge is still broken many years later.
I don't know what your sudo does, but mine requires the --preserve-env flag if you want the new process to have access to all your environment variables.
The thing you're saying is going to be an endless source of problems should already be an endless source of problems! (And I think I've been briefly confused by some missing environment variable once or twice so far.)
It depends on whether sudo was compiled with --disable-env-reset or not, it's on by default[1].
Also some variables are inherited regardless (e.g. DISPLAY, TERM), and some useful ones (e.g. HOME) are initialized by sudo, but I can't tell where that's done.
Even without the flag, sudo preserves a bunch of stuff. And it's not even consistent. Some implementations preserve locale setting, while others don't for example.
I think what is perhaps something to consider is how much of an attack surface sudo is and how unaware people are of the fact. Many people think they can configure sudo to be safe to use for unprivileged users, by only allowing specific things to run with it. But they don't realize all the ways it can be abused for privilege escalation. Getting rid of all that configuration removes that false sense of security, which is a good thing, it has been a huge footgun in Linux for decades. Some incompatibility is price well worth paying for that imho
I think these problems are basically negligible because the amount of people trying to "configure sudo to be safe to use for unprivileged users, by only allowing specific things to run with it" is negligible. Virtually all users of sudo are using it on their own computer which they are the sole user and ultimately the administrator of. Even in corporate contexts where the company owns the machine instead of the user, I've only ever seen cases where the use of sudo is unrestricted albeit logged. Where are these organizations where developers or syaadmins are allowed to use sudo but only with white listed commands? I don't doubt that some people are doing this, I just think it's not common.
Replacing the whole of sudo with some weird new thing to better support a niche usecase seems disconnected from reality to me.
My last job was at a UK bank. All our *nix systems were configured with a specific whitelist of commands that could be run via sudo.
We found this an enormous pain in the arse when the powers that be decided to deploy ansible everywhere, and found that none of its "become" methods would work if sudo was set up like that.
That's not what parent means. They are arguing it's not generally used to delegate partial root access in a multi-user environment, i.e by adding narrow sudoers rules to allow "some" things to be run as root to users who don't have full root access otherwise.
I tend to agree that 99% or use cases are "I already have root access but an logged in as a normal user and need root now", i.e this is just a way to gain the root access they already have. If sudo didn't work they would just be using login and doing exactly the same thing but with less convenience.
Although It's only a guess, and now I've read some other comments it's surprising what people will use it for. Adding custom sudoers rules is known to be a bit of a footgun, it's quite easy to accidentally open up a hole for privileged escalation if you aren't very careful and don't know what you are doing.
Its a common finding during pentests, but that's just unconvincing anecdote vs. anecdote. Another argument, besides misconfiguration, is the reduced attack surface by removing the huge complexity of sudo entirely. If your argument is basically that its fine because nobody is using the complexity of sudo, then I don't quite understand what your objection is to removing that complexity. You might need to manually restore some env-vars, whats the big deal?
But I suspect this would just turn into a "doing things differently is bad because its doing things differently" argument, its not a very useful conversation to have.
> Virtually all users of sudo are using it on their own computer
Nope. If I had to guess, it's in containers, like Docker. And those run in lots of places, and often in places with easy access to company's cloud account, credit card info etc.
'systemd-run except with privilege escalation' is a thing I wished for for a long time, needed in production.
Glad they finally made it, too bad it took them so long. (To be honest, it feels like it should have just been part of systemd-run in the first place.)
I mean, systemd-run could do privilege escalation from day one. It's even the default (otherwise overridable by systemd-run -pUser=<user>). I have used systemd-run --shell on countless occasions when I needed a clean root shell without any traces of the current environment.
What is being announced is merely a thin layer of cmdline syntactic sugar over an existing feature, to make it closer to sudo in usage.
Yeah we had to explicitly set that to No and enable linger to let pulseaudio keep running for users so they can continue to stream sound from their remote browsers in BrowserBox/CloudTabs. Ie at: https://puter.com/app/cloudtabs-browserbox
Your comment was really well expressed btw. Made your thoughts and emotions very clear about this. Inspiring communication skill! :)
> I'm particularly worried about this because of the whole saga of KillUsersProcesses breaking nohup and screen, which to my knowledge is still broken many years later.
For anyone curious, see "Systemd v230 kills background processes after user logs out, breaks screen, tmux" from 2016:
> And I'm kind of worried that when this breaks stuff, the systemd project is going to push forward with some plan to get rid of sudo, and not gracefully accept the feedback that this is breaking things.
Given Lennart already declared SUID concept as “bad”, I think this is the game plan all along.
Systemd: Do all the things, but not very well, and don’t listen to anyone.
SUID mechanism doesn’t always “elevate to root”. It’s a mechanism to “run as another user” and with SGID allows great flexibility in user permission management. You can allow all kinds of (responsible) user switch tricks for multi-admin servers and multi-user systems.
Focusing all of this to sudo and framing SUID as “just implemented to enable sudo” is not painting the correct picture.
Moreover, removing SUID breaks tons of mechanisms and scenarios.
Security of sudo can be debated, but evolving something already present to a better state step by step is miles better than banishing and rebuilding it and making it dependent on systemd. systemd already breaks tons of UNIX conventions and way more complicated than it should be.
When you think, it sounds like “conquering” another part of user space and making systemd more entrenched, plus systemd is already a pretty large surface area to attack already.
XZ back door reached SSHd over libaystemd. Do we need another “integrated target” to attack in Linux?
All these ideas that tie permissions to a file completely fail when files need to be accessed either over network, or inside a container.
I can see how the original authors didn't consider these cases, because they simply weren't there yet... but knowing what we know today: SUID is an awful idea.
Sorry for my ignorance, but what’s a scenario that you run a SUID/GUID binary from a network or a container?
If you access and run, it’s SSH or similar, so it works on the system scope. If it’s a container built correctly, it has its own users and isolation already, so it shouldn’t be able to fire any binary on your “base” system, and any effect is already in the container scope.
I have never had the need to SUID/GUID a non executable, and didn’t need to trigger something on the system inside a container in the last ~20 years.
> but what’s a scenario that you run a SUID/GUID binary from a network or a container?
A lot of publicly available container images require elevated permissions to simply function, not for anything extraordinary. So, the user in container needs to be a superuser. It's often even not to perform the program's main function, but because various ordinary things in Linux require elevated permissions.
> container built correctly
That's a spherical horse in vacuum. If you write code s.t. there aren't any errors, you don't need to do error handling, right? You don't get to choose how containers are built. You need to deal with all possibilities of how containers can be built.
Network filesystem? -- /usr/shared, /usr/opt and /usr/local? That's by design... very typical for cluster management software to mount these from NAS. It's also very not typical to keep these as "only text files". Pretty sure a lot of Google's stuff installs automatically into /usr/shared. I think even Go compiler and other infra at some point was being installed there by default.
Finally: the same argument as with containers. You, for some reason, are trying to fantasize the world where problems don't exist because you chose the world w/o problems. But this isn't the real world. It's a fantasy. In real world, with or without reason, programmers and other computer users will do what's possible, not what you want them to do.
The problem with this line of thinking is it gives automatic carte blanche to anyone pointing out problems to implement "solutions" to those problems with little interrogation of whether those solutions are actually better.
SUID, like any system, is flawed. Most of those flaws are balanced trade-offs; if you're addressing one you need to be aware of the severity of any counter-problems you're inevitably introducing.
Lennart is well known for criticising existing systems while simultaneously ignoring & dismissing criticism of the proposed solutions - you need to be able to weigh up both sides in a balanced way. Lennart demonstrably isn't.
Just because it has been pushed by RedHat and others are semi-forced to adapt, it doesn't mean all distros got in line to get a copy of the software and get it adopted.
Pulse has been replaced with the pipewire as soon as it arrived, for example.
> Ah the old 'we were forced to use this free thing'. Sure.
Well, I was a tech lead of a Debian derivative when Debian held the vote. I have seen and read enough. Didn't see the other "threats" thing, but since I had access to debian-devel, I was in the middle of it.
My views about systemd has not changed since then, and can be found if you search HN.
On the other hand, I have used 4-5 init systems in the last 20 years, and none of them were that aggressive and had the "we know the best" attitude, while going against all the best practices and making the same mistakes done in the past.
> Pipewire combines alsa, pulse and jack. They all have different strength.
Nope. ALSA is always there, working as the primary sink, delineating user space and hardware. I used Jack back in the day for recording, and never got to like pulse because of its backwards defaults and lassies-faire behavior about multi-channel audio (plus glitches, etc).
Pipewire is a great sound server which sits on top of ALSA, and replaces Pulse transparently, and makes everything 100x nicer along the way.
Lastly, it's Lennart Pottering. Not Lennard. :)
P.S.: It's important to understand that my views are not against the persons, but the behavior of the projects. I'd drink a nice round of beer with all of them, if I had the chance. :)
The reality Debian didnt want to or couldnt develop their own. The system people used them was simply shit. And the alternatives like Upstart were just crap.
Nobody forced Debian. I followed it live too. I remember him talking to Debian and he made a technical argument for it.
I had already switched to Arch and had already been using Systemd for years at that point.
The reality is, nobody was stepping up with better solutions. Would porting SMF have been better, maybe, but nobody was porting that.
There are distros with Systemd, often very compatible ones, and almost nobody uses them.
BSD folks for years have been hoping for the linux exodus over systemd and it has never happen.
And it has to be said a 1000x times. Systemd was never just init and it mever claimed it was. By now Systemd is just a software project that makes all kind of software that you can use with or without systemd the service manager.
The should just call it the "Linux Userland Software Group" and change their naming. Then people wouldnt get triggered by the term 'systemd'.
You can have whatever technical opinion you like about systemd. Fact is most people use it, including in very large organisations. And the other fact is nobody forced systemd on Debian. Whatever consipiricy was apread in 'Devel' (and elsewhere).
Thanks for confirming that I can have technical opinions about systemd, as an admin who touches more than a thousand physical servers. :)
I'll agree to disagree on the systemd's "we will replace anything and everything we even slightly dislike, and slowly make them dependent on systemd (the service manager) while not listening to you and your pesky experiences" attitude, and wish you more power for your future endeavors.
Setuid is a mechanism where you take a program, and mark it so it always runs as root (or some other user, but in this case root). The idea is that an unprivileged user can run a setuid program, and the program itself decides what privileges to allow.
The problem is that the user controls the program's view of the filesystem, environment variables, and other attributes, and this is an attack surface that can be used to trick it into loading and running code provided by the unprivileged user, which runs as root. For example, ordinary programs have a preamble inserted by the compiler where they load a programming-language runtime, usually from somewhere like /usr/lib; but a setuid program can't safely do this, because the user could use a chroot to replace /usr/lib with something different.
In practice, this means that writing a setuid program correctly is exceptionally difficult and error prone, can only be done in C, and imposes security requirements on the compiler flags/makefiles rather than the source code, which creates a large risk of distro- or compiler-specific vulnerabilities. In practice, sudo is the only program people allow to use the setuid mechanism, and sudo is a unique and dangerous snowflake.
One binary to rule them all, one binary to find them, one binary to bring them all and in the darkness bind them; in the Land of Lennart where the shadows lie. Bwaahahaha.
Huh. I'm not at all a fan of how Poettering operates, but it's neither the ideas nor the implementation where I'd fault him. Well, it depends on what you mean by implementation, I guess; I'm talking about the core "how does it do its thing", not the interface by which you use it.
I think Poettering has great ideas and great implementation. It's the execution and interface that are often terrible. If the square peg doesn't fit in the round hole, then he'll always say that the peg is perfect and the world just needs to chisel out the corners of the hole.
you have the wrong view point. he just have a different opinion than you.
he single handled managed to fool RH and all distros into turning Linux administration just like windows. systemctl list of services is so inspired by the atrocious windows' admin list of services (which have 3 fields supposed to describe the service, but they all just tell you the name again).
it's no wonder his reward was a job at Microsoft.
but again, he's good in all three aspects. you just disagree on building the torment Nexus that is putting Linux in the "standard certification" target for sysadmins.
I continue to be baffled at this widespread belief that Poettering somehow hoodwinked every single major Linux distro into accepting a shit product with, idk, hypnosis or something.
Is it not possible that systemd is simply better than the alternatives, and the distro owners are smart enough to notice that, instead of just wrapping themselves cultish mantras about The Unix Way and how anything which resembles a design used in Windows is bad by definition? Or could that not possibly be it and he must've used mind control magic.
just that his vision was garbage, and everyone knows. but he stood by it. and nobody was putting the same energy he was to either offer better or stop it (rejecting bad ideas also take energy. see gnome deep dive into garbage as another example)
Linux is mostly made from scraps (eg Bluetooth and wifi entire stacks) or misguided but funded things. the age of scratching own itch is mostly gone
> nobody was putting the same energy he was to either offer better or stop it
which was much easier thing to do, compared to an outsider, considering he was on Red Hat's payroll, along with the people (gnome/freedesktop crowd) he had need to convince
Yes, it has always reminded me of the old "Apple just sells all those shiny devices because they're good at marketing" trope.
As if marketing alone could do that. Poettering does seem to be, to a casual observer, kind of a dick. Arrogant, dismissive of competing products... kind of like that other guy — also kind of a dick — who supposedly had that "reality distortion field" that hoodwinked all those poor saps into buying his phones.
There's no fucking way in hell you are able do that if the user base doesn't think the product is good. To those saying it, I always reply, "It may not be the product you want, but a shitload of people disagree with you, quite obviously."
I'm not personally a huge fan of the iPhone or systemd. But they are both clearly "the best" for the largest number of people. (And that is even clearer for systemd, as it doesn't cost hundreds or thousands of dollars more then the competing products.)
For systemd and pulseaudio, the systems they were replacing legitimately had major problems. There were variants and workarounds that fixed some of these, but no holistic solution that I've ever heard of. There were just so many limitations if you maintained any degree of compatibility. People were (understandably) unwilling to start over and rearchitect something that desperately needed rearchitecting. Poettering designed and implemented replacements that were substantially better, and worked. Worked well, in fact. That's the great ideas & implementation part.
Much of this was enabled by a willingness to throw out compatibility with nearly everything. Backwards, forwards, sideways. If I were making a bold and breaking change like this, I would sacrifice compatibility but try to make up for it by bending over backwards to catch as much of the "once working, now broken" wreckage that inevitably piled up as I could, by creating shims and compatibility stubs and transition mechanisms. I'd certainly listen to people's problems and try to work out solutions.
Poettering, as far as I can tell is more of a honey badger (excuse the dated meme). He just doesn't give a shit. If your stuff doesn't work in the brave new world, then your stuff is broken and is going to have to adapt. That's the bad execution part. (Which is not to say that bending over backwards is always the right approach; it can massively increase the burden on the new system's implementer, to the point that it never happens. There's a reason why Poettering's stuff is taking over the world.)
As for bad interface, this is a lot more subjective, so it's easier to disagree. But the tools to configure and use the new system are done in the style of an isolated cathedral. The tools do a ton of stuff, but they do it all in a new way, and that's great once you learn the blessed paths and internalize the new architecture. But all of your existing knowledge is now useless, and you can't flexibly combine and extend the functionality with the usual tools you'd use (bash, grep, awk, find, sort, tee....) The main functionality of the new system is not new — none of this is really adding fundamental new capabilities, it's just improving things that were already being done. But the way you interface with that functionality is all new, even though it could have been exposed in ways at least a little more unix-like and composable. Instead, the tool author determines all the things you should be doing and gives you a way to do them. If you want more or different, then you're doing something wrong.
Normally, I'd expect something like this to die out as it rubbed up against the surrounding functionality. "Great system, but too much effort when we keep having to fix thing after thing." Surprisingly (to me), in systemd's case in particular, what has actually happened is that the cathedral just keeps expanding to swallow up enough of its surroundings to keep from being ejected.
Maybe it's sour grapes, but my guess is that this was only possible because the previous systems were so bad. esd was a nightmare. sysvinit scripts were baroque and buggy and error-prone. Sure, the first 80% was just plain simple shell scripting. But everything did the last 20% slightly differently or just punted. It was all buggy and idiosyncratic and failed intermittently. Supposedly some of the init system variants managed to corral it all together enough to do actual dependencies and get decent startup speed, but I never used such a system. And based on the quality of the init scripts I've seen from random packages, I'm guessing the successes only happened when a relatively small group of people wrote or repaired a metric shitload of init scripts by hand. And even then, systemd provides more in its base functionality set. Architecturally, it's really quite nice.
> Much of this was enabled by a willingness to throw out compatibility with nearly everything. Backwards, forwards, sideways. If I were making a bold and breaking change like this, I would sacrifice compatibility but try to make up for it by bending over backwards to catch as much of the "once working, now broken" wreckage that inevitably piled up as I could, by creating shims and compatibility stubs and transition mechanisms. I'd certainly listen to people's problems and try to work out solutions.
You do realize that systemd was the only init system that offered distributions a migration path from the sysv-rc init scripts?
daemontools, s6, openrc, upstart all did not have this. systemd was the only system caring about migration and backward compatibility...
> Poettering, as far as I can tell is more of a honey badger
As far as I know, he was the only author of an alternative init system that, for example, did actually talk to distributions to understand which problems they have. Unlike the authors of most alternatives that don't give a shit (and in turn nobody gives a shit about their init). To this day you'll find the s6 author just claim "nobody needs feature X from an init" because they themselves might not need it.
Requires root not just for the ptrace protection, but also to gain membership of the 'tty' group which gives control over all ttys. And then goes all surprised pikachu when it turns out that allows taking over ttys. Duh?
I wouldn't worry too much about that. It's a tricky piece of security-critical software, receiving its first round of outside auditing; of course it has vulnerabilities. Sudo does have the advantage of being much more battle-tested, but that will even out with time; what will matter is how secure it is two years from now.
To be fair, this is not at all Poettering’s idea. There is, for example, precedent in the form of s6-sudo[1], a utility from the s6 service supervisor, itself very much an anti-systemd project (except I believe it predates systemd?..).
And honestly I’d be okay with a suidless Unix. For example, as best as I can tell, the only reason the kernel needs to know what executable formats even are—beyond the bare minimum needed to load PID 1—is s[ug]id binaries.
I like s6! One of the key differences here is that s6-sudo builds on, rather than replaces, the standard unix permissions model.
s6-sudod listens on a unix domain socket. Unix domain sockets are just files, so they have an owner, group and mode bits. The answer to "who is potentially allowed to run a differently-privileged command?" is just `ls -l /path/to.sock`.
For finer-grained access control, a unix domain socket listener can call `getpeereuid()` or `getsockopt(..., SO_PEERCRED, ...)` to learn who it's talking to. You can build powerful – but still relatively simple, and importantly, readily-inspectable – access control policy on top of these basic unix primitives. That's what s6 does. Look at how simple rule definition is. [0]
Or, you could throw all that out the window and build something much more complex and much less inspectable, which is the systemd approach. The answer to "who is potentially allowed to run a differently-privileged command?" under `run0` is to...spend the evening reading through polkit xml rules, I guess?
I realize systemd uses D-Bus, and D-Bus uses a unix domain socket. But that socket is writable by world. We're trusting polkit and complex policy xml and probably a constellation of other services to get things right after the SO_PEERCRED check.
Maybe that's fine for desktop apps, but a reminder that we're talking about sudo here.
Complexity is the enemy of security. The complexity of the systemd ecosystem broadly writ is how we get CVEs like this polkit privesc, which took 12 years to notice [1].
Addendum: it's possible to regard systemd as dangerously complex AND sudo as dangerously complex. OpenBSD as usual had the right idea with `doas`.
Like many other things in Unix, SO_PEERCRED and getpeereid are half-implemented hacks that should not be used for security. They both only return the uid that was used at the time of calling connect(). Meaning you have to be incredibly careful what you do when creating the socket and you cannot really pass any sockets off to other processes if you want to try to do security that way because they will still inherit the wrong credentials. Also the usual complexities apply of how to interpret that when interacting with a process in a container.
I have a pretty low opinion of s6 because of things like this, you pretty much have to create a more complex system like polkit and systemd if you want this stuff to actually work. You don't have to use XML and javascipt like polkit does but you do have to do more than what s6 is trying to do. (Also, I personally don't find the "random collection of ad-hoc text files" style they do to be any less complex than systemd, but that's a different conversation)
> Meaning you have to be incredibly careful what you do when creating the socket and you cannot really pass any sockets off to other processes if you want to try to do security that way because they will still inherit the wrong credentials.
I see nothing new here beyond "handle privileged resources with care." Don't overshare. Got an open pipe to `sh` running as root? Maybe you oughtta set O_CLOEXEC on that fd before you exec and overshare with a child. Got a socket that's been peer authed? The same.
This is pretty basic unix stuff. If you stick to the basics and avoid the siren call of complexity, the security properties remain relatively easy to reason about. Most privileged resources are fds. Mind your fds.
I'm not a huge fan of sending file descriptors over sockets – maybe we agree on that part.
> Unix domain sockets are just files, so they have an owner, group and mode bits. The answer to "who is potentially allowed to run a differently-privileged command?" is just `ls -l /path/to.sock`.
Yeah, except that is not true. To quote unix(7):
On Linux, connecting to a stream socket object requires write permission on that socket; sending
a datagram to a datagram socket likewise requires write permission on that socket. POSIX does
not make any statement about the effect of the permissions on a socket file, and on some systems
(e.g., older BSDs), the socket permissions are ignored. Portable programs should not rely on
this feature for security.
So s6 just has a wide, easily exploitable security hole there. Or is not portable, contrary to its claims.
Lol okay man. Maybe if you're running FreeBSD 4.2 or HP-UX or some BSD derivative from the 90s. All unix systems from about 2000 on will honor unix domain socket permissions.
That's not an exploit, that's just a sequence of basic misunderstandings about how things work on Linux. Which would be fine, nobody knows everything, if they weren't coated with grand claims and not-so-veiled personal abuse.
The way I see it, this is actually a good thing. Superuser access should impose a tiny bit of friction in this regard, to enforce discipline where discipline is warranted.
>But enough about all that security blabla. The tool is also a lot more fun to use than sudo. For example, by default it will tint your terminal background in a reddish tone while you are operating with elevated privileges. That is supposed to act as a friendly reminder that you haven't given up the privileges yet, and marks the output of all commands that ran with privileges appropriately.
WHAT AM I READING?? Why can't the systemd developers just be normal?
I also suspect that this will interact terribly with anyone who uses a certain kind of terminal theme. Do they not know that you can have these "nice to have" features, off by default, so that anyone who wants can enable them and anyone else is never bothered by them?
A similar idea was tested in an experimental BSD clone in Berkeley in mid-1980s. (Great sorry I haven't kept link to the description, so rephrase with my own words. Maybe this was in the McKusick's book?)
No suid or sgid was allowed. A daemon started from init and listening on a socket listened for connections, checked permissions and run the specified binary with requested permissions. A caller had to interact with the started program using pipes.
It seems the complexity of passing all to pipes was why the approach was rejected. Instead, the checking of inherited environment was strengthened. "Everything new is well forgotten old."
I honestly have no clue what you mean. You can take unit files, journald, timers, and all the other neat features from my cold, dead hands. I’m not going back to writing brittle shell scripts; systemd has made my life SO much easier.
It’s really not that hard. I have found that writing robust shell scripts is pretty easy.
Don’t use bash, stick to #!/bin/sh, use shellcheck, wrap all variables in quotes, use command -v to check if a binary is available before trying to use it, and don’t use gnu specific things.
Sure all of that is easy, but control flow is a nightmare. How do you deal with cleaning up resources on failures? Idempotency? Traps? It’s a fucking mess
I don't know what to tell you other than skill issue.
I don't think shell is a great language or even that good for configuration, but it is pretty simple and the quantity of footguns are similar to JavaScript.
My issue with systemd is not the configuration file/language, that bit is pretty cool. My issue is mostly with all of the half-assed auxiliary services they keep pushing and something is not quite right with systemd itself, the complexity causes issues with weird edge cases.
Early in my career I was all for systemd, thought it was the greatest thing ever, then I got bit by the edge cases a few too many times. Meanwhile I have had zero issues with openrc or s6.
I don’t disagree about skill issue. I just don’t think you should require immense skill to implement simple control flow without shooting yourself in the foot. It’s much worse than modern JavaScript in my opinion.
I don’t buy the argument about all the auxiliary stuff. You are free to not use it.
Ah, Right. So, how would I add support for reloading the configuration of my init-Script service without restarting it, in a way that works across all major distributions and versions, without googling?
Show me your script, compare it to the one line of my unit file, and tell me that isn’t brittle.
I wish systemd would die or just be just an init system. This whole thread is people suggesting vague and non obvious solutions to things that people already knew how to do with just linux utilities now with some weird other binary.
everything other than "being an init system" is a compile time option in systemd, so your complaint is that ... other people are building OSes and turning these things on?
which OS are you working on that provides all these convenient features and large-scale integration of software with them, with code that isn't in systemd?
Between "systemd --user", Linux Capabilities, and containers there is no reason to ever have sudo, or the ability to touch the root filesystem at runtime at all, which should ideally be a signed, deterministic, and immutable image anyway. You can do anything as an unprivileged user these days without risking core system integrity and privilege separation guarantees.
Remember, malware can just alias your sudo command to one that logs your password and piggyback on your next use. If you ever use sudo, then all bets are off on sandboxing malware. Best to not have a ladder to root at all.
Sudo is a crutch for people that have not learned the last 20 years of privilege isolation tech.
So by design run0 does not do what sudo does. If I understand it correctly it has to stay a child of the daemon it's forked or does Linux provide an API to mangle the poor process table to change that since because according to POSIX the process would not part of the current session, or process group? How does this interact with (or break) shell job control since you can't forward SIGKILL. Does anything prevent the invoked process (which could also have less privileges) from holding on to the passed file descriptors e.g. the one to the callers controlling tty or is it restricted to always go through pipes/sockets/a fresh pseudo-tty?
Okay further down it explains that it always goes through a fresh pseudo-tty (at least for interactive commands?). That solves the file descriptor passing problem but not the reliable signal handling for job control since there're signals you can't catch and forward.
run0, which behaves like sudo, but works entirely differently and is not SUID. Run0 asks the services manager to create a shell or command under the target user’s ID, creating a new PTY, sending data back and forth from the originating TTY and the new PTY.
reply