As a rule, NixOS doesn't patch binaries, it causes RPATH to get baked in at build time. Also, as a rule, it doesn't have some sort of materialized FHS-like subtree for each package, it manages a complex filesystem tree where the (bit-exact) stuff from the packages is stored immutably by the hash of the full build description.
Binary patching only comes in when they're trying to get closed source binaries to run on NixOS and is fully managed by the packaging process to happen the same way on every system. These days, though, the approach of using filesystem namespacing to give packages a custom FHS-like view of the world seems to be growing more common instead of the patching.
I really like the idea of NixOS and Guix and tried NixOS for half a year on my laptop. Then I changed back to ordinary package based distributions. (Arch, Debian and Gentoo)
The issue with NixOS for me was probably quality control. When I did an update and instead of fetching the files from the binary cache it starts to build stuff on its own, I could be reasonably sure that some build error happens. Coupled with sparse documentation, I was very often at a loss at how to fix it and just had to remove the package for a while and try again a couple of days later.
Another point is NixOSes path mangling... Do we really need that? Can we not try to use namespaces and OverlayFS etc. to let each process assume it has a normal FHS file-hierachy while in real its 'root' is cobbled together from multiple package installation directories? Instead of patching the paths of every package, letting the kernel do the path calculations seems to be less intrusive.
How does an article on NixOS talk about the `rpath` issue without also mentioning the `patchelf` utility that NixOS developers created to solve this issue? It's a small tool that lets you modify ELF executables and binaries. It's also the recommended way for NixOS users to modify binaries to work properly.
Yeah, and thanks to Nix you can use PatchELF[0] to change the rpath on a binary too. Point being that for users it is not a straight-forward process to make a Linux application portable. Thanks to Linux Desktop's lackadaisical approach to userspace ABI compatibility it isn't necessarily that easy for developers either.
nixOS is built around hashing the outputs of its builds, so you can verify if a build produces the expected output given the same inputs. The reason nixOS patches binaries is so that they can actually find the shared objects they expect when they are not stored in /usr/lib. Since every build output gets its own unique path, this allows for a binary to link against two slightly different versions of the same library, which in practice is never an issue one needs to resolve. However, a more practical issue this solves is that you can have a single sysroot with two binaries that need two slightly different versions of the same library.
I've not heard of anything patched outside of paths being changed by patchelf. This is done because any dependencies are being stored inside the nix store under a hash of their dependency trees, which allows for multiple versions of the same dependency to exist without conflict. You should find that the NixOS community has no problem with producing the nix file to make NixOS support possible for your open source DAW.
NixOS changes some standard conventions of Linux filesystem layout and where tools can expect to find things. These are for good reasons and are due to the core of what NixOS is trying to achieve. For building most things those changes are relatively abstracted away (see ld wrapper script for details) and you don't have to know about them. Fiddling around with something low level like a linker is I think a forgivable situation where the abstraction leaks - it is a process that is inextricably linked to where the system puts things.
My issue is the last sentence:
So… .. turns out there’s more than one lld on NixOS. There’s pkgs.lld, the thing I have been using in the post. And then there’s pkgs.llvmPackages.bintools package, which also contains lld. And that version is actually wrapped into an rpath-setting shell script, the same way ld is.
That means that there isn't a problem. NixOS has fixed this, the system works. Except that you have to magically know which package you should be using. This is the sort of problem that I run into with Nix - it's hard to know the correct incantation.
As said by the other commenter, most of these patches are caused by NixOS not using the FHS (except for /usr/bin/env), which can be considered as a good thing in itself. In an ideal world, it would not be a problem (programs should not assume anything about the filesystem hierarchy, and should always use $PATH or their own environment variables).
The issue of integrating with other language ecosystems is indeed very problematic (and it's a very hard problem), especially with regards to the chain of trust (by the way, it's not completely broken, you can still read the original package hash in the derivation, and evaluate this derivation to get the Nix derivation hash), and with regards to the fact that including all of the packages of every ecosystem in `nixpkgs` is not doable.
There is also the (related) problem that evaluating a configuration implies evaluating `nixpkgs` itself, which requires a good amount of available memory, and this can only get worse as `nixpkgs` grows.
> Binaries are a pain, but that's always true if you aren't running the exact distro that the binary was built for.
This is actually easier to deal with on NixOS than elsewhere, because the tools for handling that case (patchelf, buildFHSUserEnv, supporting multiple versions of a lib, etc) are so well developed.
patchelf is not really widely used for solving reproducible builds issues. It's made for rewriting RPATHs which is essential for NixOS, but not something you would be seeing in other distributions except for when someone need to work around poor upstream decisions.
To be fair, patching existing binaries isn’t all that easy to begin with. Once you need to start moving around structures, a lot of bets are off. In case of NixOS, I’m sure the long nix store paths added often overflow structures and require workarounds to avoid breaking images. Hacking around Win32 PE images, I’ve run into a lot of tricky subtle issues over time...
rpath is braindamaged; no program should be using that, and a distro build should almost never be inserting such a thing into executables.
I suspect that NixOS is playing with this in order to have a relocatable install: so that is to say, so that user can install NixOS in some subdirectory of a system running some existing distro. Any subdirectory, yet so that programs can find their libraries.
If I were in this predicament, rather than perpetrating hacks to patch the rpaths in binaries, I'd fix the dynamic linker to have a better way of locating shared libraries. The linker would determine the path from which the executable is being run, calculate the sysroot location dynamically, then look for libraries in that tree. E.g. /path/to/usr/bin/program would look under /path/to/usr/lib and related places.
A possibly nice hack would be to extend the meaning of the rpath variable. Give it a syntax, like say that if it starts with @, then the rest of it denotes a relative sysroot path fragment.
E.g. the program that gets installed as /path/to/usr/bin/program would be built with an rpath of "@/usr/bin". So then the dynamic linker sees the @ and does a sysroot calculation. First it strips off the basename to get just the directory part "/path/to/usr/bin". Then it sees, hey, the suffix of "/path/to/usr/bin" matches the "/usr/bin" in the rpath. The suffix is stripped to produce "/path/to" and that path is then used as the root for the library searching. Instead of searching literally in /lib or /usr/lib or whatnot, the "/path/to" part is prefixed to every search place to look in /path/to/lib and /path/to/usr/lib.
Patching binaries is very poor; it changes their cryptographic hash like SHA-256. You want your distro to be installing bit-exact stuff from the packages, and treating it as immutable.
They don't all have to be patched, they just have to be rebuilt in a carefully-controlled environment.
Nix installs everything that it manages (libraries, applications, even managed conf files) under a path that is derived from its exact build description (it's a hash plus some extra stuff to help humans identify what the component is by inspection). This allows, e.g., two different versions of Firefox or two different versions of zlib to exist on the system at the same time. Because the exact build description for the application refers to the exact build description for its dependences, there's a certain Merkle Tree like sense to it where if you were to e.g. rebuild Firefox against a different version of zlib, your resulting Firefox build would have a new build description hash and thus a new path.
An example of what an executable path might look like is: /nix/store/v5sv61sszx301i0x6xysaqzla09nksnd-hello-2.10/bin/hello
Stuff like autotools can actually generally support all of this without patching. In the case that the user wants to run a binary built for a non-nix system outside of a container on a nix system, binary patching does become necessary, in the form of appending to the RPATH of the executable. Nix does provide tools to eliminate as much pain as possible from this process.
It is worth noting that you can still only have one version of a program on your PATH with the canonical name at once.
As far as programs invoking each other with exec: that establishes no requirement at all that the invoked program be using the same versions of libraries as the invoking program. Programs packaged with nix sometimes "purely" refer to each other by their full paths. Other times, they discover each other "impurely" by just searching the PATH.
Big problem is that just forking it isn't enough, you also have to convince every distro out there to switch to your fork to make any real difference. It becomes a truckload of boring work for what might be a trivial one-line fix in the code.
I blame the package managers we use for this, as most of them make it incredible difficult to change anything and make you a slave to whatever the distribution ships, which in turn is slave to whatever upstream ships. The effort to fix the issue just far outweighs the issues the problem is causing in the first place.
NixOS is one of the few that seems to get this right by making code changes pretty much trivial. Just fork the upstream repository, point the package to the new fork and you are done. With a Nix flake you can even make the repository itself installable directly. No need to try to convince any maintainer to include your patch. And having package names be hashes means you can switch between old, new and patched versions of a software easily, no need to worry about conflicts or having to un/reinstall all the time. Downside is the lack of binary compatibility in NixOS, meaning if you change a package deep down the dependency tree you might end up recompiling half your distribution and can no longer use the precompiled binaries (there is `patchelf` to workaround this, but haven't used that myself yet).
I didn't think NixOS patched the runtime linker (though I could definitely be wrong).
They do eliminate any ability to automatically find libraries, and the dynamic loader does not exist at the path where it exists on most other distros, which means binaries linked elsewhere will not run unmodified on nixos.
Yeah, thanks. To add to that, the Nixpkgs "standard library" includes lots of convenient tools for patching like this, and much of the common stuff is done automatically by the standard package function. The #nixos IRC channel is a good place to ask if you run into trouble when packaging something.
Generally, making NixOS packages is really delightful. Even the patching stuff is straightforward and easy to use. Now that I've learned the basics, I routinely make small packages.
With many typical packages the only thing you need to do is specify how to fetch the source, plus the list of build dependencies, and there are built-in functions for fetching from HTTPS or Git.
Yeah seems like a nightmare since so much of the auto-magic that nix does to fix binaries (shebang rewriting and ELF .so patching) does so by injecting absolute paths (to /nix/store). So all of those would need to be reshimmed for your entire system
EDIT: theres also work being done to move to content-addressed nix store, in which case changing these things would invalidate the content hash
Binary patching only comes in when they're trying to get closed source binaries to run on NixOS and is fully managed by the packaging process to happen the same way on every system. These days, though, the approach of using filesystem namespacing to give packages a custom FHS-like view of the world seems to be growing more common instead of the patching.
reply