Actually, the thing we started using nix for was reliable caching of compiled artifacts, including not just your code, but all the programs and libraries that your code depends on. It's another thing that's difficult to do in a general sense, but if you have a fairly strong notion of reproducible builds, it's possible.
I can see the utility of using Nix for managing dependencies of software you're developing, because that can be horrific to deal with and Nix makes it fairly elegant to do, depending on what ecosystems it's being used with.
Using NixOS to manage your whole system, on the other hand, I don't think I'll ever get behind. Maintaining reproducible builds of my system, while being nice to have, doesn't really solve any problem that I actually have. On the other hand, it introduced a myriad of problems that plagued me on a regular basis before I switched away from it.
The first problem I had was that it added steps to a number of tasks I had on a regular basis. For example, updating or adding new extensions to VS code. Because NixOS is immutable, you don't get to benefit from just running a single update or install command within VS code like you would on a normal system. It becomes an annoying series of steps you have to carry out for each operation, all in the name of being convenient to hypothetically down the line. Multiply this by all the tools you use and it gets tiresome.
The other issue is that you raise the barrier of entry for every new piece of technology you want to experiment with. Now, in addition to learning how to use any tooling involved with the ecosystem, you have to figure out how to get it to play nicely with Nix. This works very well for some ecosystems, like Haskell, and not well at all for others. I was just getting started with Android development at the time, and just when I thought I had it working nicely with Nix, I would run into a new problem that would need investigation and band-aids.
I wish NixOS was more convenient to use than it was in my experience, because it's nice when it works well. But in my case it created more problems than it solved.
This is why I manage every nontrivial project I do nowadays with Nix (Haskell, Go, Python, C & C++, bash ..anything)
Everything is pinned to exact source revisions. You can be relatively sure to be able to git clone and nix-shell and be off to the races.
You can even go the extra mile and provide working editor integration in your nix-shell (especially easy with emacs). So you can enable anyone to git clone, nix-shell, and open a working editor.
The biggest downside is becoming proficient in Nix isn't easy or even straightforward.
I agree. Nix is awesome in a way, but it's also a product of it's time (early/mid 2000s). It definitely solved the problem of reproducible build environments, but it didn't solve isolated environments like docker did. It also has an issue where there are soany ways to do things, it's hard to know what the best way is for your situation (especially between flakes and nonflakes).
But I'd say the biggest issue with Nix in general is it's lack of discoverability. This is evidenced in the number of third-party tools built around helping figure out what even exists (noogle.dev and search.nixos.org for example). The language is so dynamic, trying to figure out where things are coming from becomes difficult. Simply put, there's too much magic.
Nix also has a tendency to drop down into massive bash scripts to actually do stuff. I really like guix as an alternative in theory, but I haven't had much luck using it yet.
No, Nix can't do everything, it can do the one thing it does, mainly it maintains immutable paths and allows to define derivations of any number of those paths using namespaced execution of build scripts.
All this is controlled using expression in the Nix language, which is
well suited for the intended purpose. It is in important aspect the opposite of Haskell.
Nix does so well what it does, that it is easy to use it where better solution exists (looking at you NixOps)
So I agree with some parts of your criticism,
on the other hand if Nix is be used to do the sensible subset of "CI servers, AWS deployments, build servers, testing, linting, dev package management, dev system environments, and more are", that would be great.
And what that subset is depends heavily on the given circumstances.
For me, being able to derive the "CI servers, AWS deployments, build servers, testing, linting, dev package management, dev system environments" things from a single source of truth, actually does sell Nix pretty nicely.
When approaching a complex, multi-component source repository, with the requirement for dev-environments, CI builds and generation of production ready Docker-images, a careful and not over-eager Nix build, seems a sensible choice, given that the layered nature of Dockerfiles and docker-compose obscures the DAG nature of built artifacts and dependencies.
Nix also doesn't build anything by itself, it merely invokes low-level build tools in isolated environments that only contain the specified dependencies.
Sure, when using Nix one is forced to explicitly state the dependencies of each (sub-component-) build-step, that results in more code than a bunch of RUN ...
calls in Dockerfiles.
Both approaches have their sweetspots.
An investment into learning Nix is done once. And you are completely right, that refactoring to a Nix based build takes weeks in a serious legacy software project. I wonder myself, if this will be worth it. Currently, I think that might be the case, and that part of the weeks put into the build are not due to Nix and would have been necessary for Docker-based builds, too, and also one can do a lot of stuff much easier with Nix, which leads to "feature-creep" like "oh let me add this preconfigured vim/emacs/vs-code into the build env" that one would probably not do if it weren't so easy to customize everything with overlays and overrides.
But that is a good thing, and it is much better to have a capable tool and critical discussions with colleagues to limit the usage of that tool, than the other way around.
Heck, I remeber when we switched to a Maven build, and had a build with all bells and whistles. That took weeks, too, and all the generated artifacts, like source code documentation sites, were probably a waste if time to implement.
I am not sure if that proves or disproves powerful, declarative and modular build systems.
Dockerfiles and docker-compose.yamls have a learning curve too, but more importantly, if you have to invest more time per build maintenance than with a Nix based build, it is only a matter time until that costs more than the nix approach.
Nix guarantees reproducibility, which means anything can be rebuilt from scratch, but that's a very abnormal use case. If it didn't work out of the box, it's a problem with the package scripts (the "derivations"). That said, all of our software tends to bottom out in a bunch of shitty C libraries that are all delicately cobbled together with autotools and cmake, so anything that aspires to reproduce these things is going to have issues. This tends to make Nix difficult to use, because it doesn't have nearly the same investment/manpower (yet) as other package ecosystems that is necessary to wrangle these dependencies into a stable foundation that doesn't leak its underlying havoc to higher levels of the stack.
I’ve looked into Nix at the previous company I worked for for exactly those reasons. I spent a bunch of time writing minimal dependency builders and artifact reuse for a maven monorepo where I had to tread carefully and not break the build. Nix seemed like a great way to get inter-repo dependent builds and dependency modeling.
A lot of misunderstanding and overexplaining in these comments. The reality is that Nix is a complex system of many parts, under active development, and yet is perhaps the best infrastructure we have for creating deterministic, hermetic, reproducible
builds. I've worked on one of the largest Nix systems and it saves our entire dev team hours. Everything is only built once, on a large set of specialized remote builders, then shared among all of our devs, CI, and deployment infrastructure. Perhaps this is common in FAANG, but Nix enables this even for small teams and individual developers.
If thats something you would like to learn more about but are having trouble getting started, I'm happy to help. Email is in my bio. We could setup a call or chat session or whatever.
Sounds like a great reason to adopt Nix or the like! Run each tool in a sandbox appropriate to what it's supposed to do, and store the entirety of the inputs and outputs so they can be audited if the need arises.
Nix got there via reproducibility, but it's exactly what's needed to mitigate compilers with surveillance features or other backdoors.
ever use nix repl? i found my complaints about grokkability disappeared once i realized i could easily introspect my build (and all my dependencies' builds) at any level.
that + learning the like 5 or so idioms that pervade nixpkgs and you can use Nix quite successfully imo.
Yes, but that's not a problem. In traditional systems the reason to avoid rebuilding everything isn't because it takes time (it does, but not that much time); the reason to avoid rebuilding everything is the fear that, halfway through rebuilding everything, you'll discover that some shared lib is missing, or some crucial toolchain has been uninstalled, or that some updated packages rely on incompatible versions of the same dependency, and it turns into a nightmare of toolchain and dependency resolution, leaving your system in a half-upgraded mess. But if the promise of Nix is true, then that's no longer possible; you rebuild everything, all the dependencies work out fine, and you go on with your day.
Can you elaborate? From what I understand, Nix packages are mostly reproducible as they are running great lengths for that, to the point where every time in Nix land is the exact same.
Ok fair enough. My issue is, whenever a project grows beyond trivial scope, it will usually involve at least some scripting, and likely some tooling, like a process which watches the source files and performs some action when they're updated.
It's very limiting not to have a shell, and arbitrary execution environment, and the wealth of tools which have been created over the years for the *nix environment.
I participate in development of the cross-platform indie video game written in C++ called Insatia [1]. We switched our CI server to building almost all the binaries using Nix. With Nix we have proper Clang with libc++/etc for latest C++ features, cross-compiling Clang (for building Windows binaries from Linux), static linking of any 3rd party libraries (useful for portable builds), Wine-based environment for building Xbox binaries (sounds crazy, but yes, it is faster and more stable in Linux than in Windows VM), and all sorts of hacks and tricks, all laid out in .nix files in a quite comprehensible, self-documented way. Now, given a clean Linux machine with only Nix installed and our game repo checked out, I can run a single nix-build command to build our game for Windows, Linux and Xbox. It will take quite a bit of time (hours) and disk space (tens of Gb), because it builds custom compilers and temporary VMs, but it will be cached in Nix store and only has to be done once.
Before that I used giant Docker image with all sorts of build tools installed/built, plus persistent VMs which had to be maintained manually, and Nix made it so much easier. Nixpkgs provides a good foundation - it has almost all software you may need and all the utilities to modify it or add new software. Need to patch or switch to custom version some obscure dependency of a compiler building another compiler building a library you use? Override a derivation, and all the dependent stuff will be rebuilt automatically. For building software Nix can be seen as a kind of super-Docker - Nix store allows for more granular caching than just layers, so incremental improvements can be done much faster. Essentially Nix turns files and packages into values in a programming language, so instead of hacky bash scripts trying to imperatively maintain a file dump, you just compose immutable packages by writing expressions.
That said, Nix is really hard to understand at first, comparable to Haskell/monad tutorials. I remember I made a few unsuccessful attempts at it over a ~6 month period, every time becoming more desperate, and then it finally clicked after careful reading of Nix pills [2] for a few consecutive days. To me, the most interesting thing to discover was that while Nix/nixpkgs do necessarily use some "hard" concepts like fixed point, it is in fact quite "old-school" and mostly about Unix, executables, linking, string templating, contains a lot of bash scripts, etc, so it's not really another Haskell. In fact, traditional building of C/C++ software with autotools is supported in Nixpkgs better than building modern stuff like Rust or Go, due to reliance of the latter on own package managers.
I use Nix for every project once it grows enough for me to have to pin a directory. And sometimes sooner.
I don't know it that well but I've picked up on design patterns, use the repl to figure stuff out, write small abstractions when necessary..and it works pretty great. And the gains grow as my dependencies become polyglot (even C + another language like Haskell..but when you add compile-to-JS, static asset generation into the mix it's amazing)
Yeah it's hard to learn. No denying it. But every other "alternative" I've found doesn't actually approach it. So I've kind of gotten sick of reading about "alternatives" because it always feels like they don't actually solve the problem Nix solves. I always end up using the alternative+Nix. Classic example of this is the Haskell stack build tool.
I could've bemoaned complexity and various capitalist criticisms when I first ran into it at a job. But I'm so much better off for not.
It's not the functional nature of Nix that is hard to deal with. Nix's functional nature is actually its saving grace.
My difficulty with Nix is that there are important subjects that aren't taught.
I figured out how to create a usable `default.nix` and `shell.nix`, but that takes a lot of effort, and there really aren't many resources to reference.
Nix expressions use a lot of predefined functions, and I have no idea where their documentation exists, if it does at all.
Even if there is some explanation for all of the functions that I need to use, there isn't documentation for how to fit them together.
Instead of being in dependency hell on the outset (like most package managers), I find myself in dependency hell when I want to do something different. The Nix community has an irrational hatred of version numbers, meaning you have to scour the web for a derivation that gets the right version of a dependency, and "pin" it, or hope that the channel you are using has the correct version. Even when channels do have the version you want, sometimes that channel provides multiple package versions, some with the version in the name.
Names are a mess. Since every package must exist in the same namespace, but its derivation is helpfully sorted into a directory tree, the same packages will often have different names. There isn't a good way to search or browse for them either. I end up digging through github quite often, which is a lot more often than the never that I should find myself doing so.
Nix is wonderful. NixOS is by far my favorite OS, and that is why I deal with all of the rough edges, but they certainly are rough.
I like the idea of Nix. It's definitely the future of package management and build systems. But it's solving a problem we knew how to solve in the 80s: dependency hell is solved by statically linking everything. In fact in the Windows world it's still like this. If you need OpenSSL, for example, it should be in your source tree and compiled in the same build pipeline as your application.
Yeah I agree here, I used to find Nix somewhat unintuitive but I have now managed a few non-trivial projects. One being to package Xilinx ISE as a flake [0], and another to build a non-volatile Linux image using busybox and runit [1]. For the latter I still haven't quite gotten where I want but that's because I realised I needed dbus for avahi and dbus is just so ridiculously complex.
Nix is hard in the way that programming is hard. Not everyone gets over the activation energy to be successful. The ones that do don't regret the effort. Nix is complex because the problems it solves are high complexity problems that other systems don't solve. Docker is not a substitute for Nix.
The solution isn't to use a weaker tool, because the weaker tool doesn't solve your problem. It's not uncommon to see a programmer use a spreadsheet, but you wouldn't expect to see a programmer use a spreadsheet where a database is needed. And you don't see people trying to use garbage-collected languages to write operating systems, even though they are easier to use than C. It's perhaps inefficient when a tool is too powerful for what you need, but it's a fatal flaw if the tool you use is too weak for what you need.
Nix let's you control your dependencies in a way no other tool even attempts. I can pin and patch any combination of dependencies, even conflicting ones in single environment, with reproducible builds— I'm in control of every detail. I would never consider a downgrade from that, but I'm open to upgrades if something even more capable came along.
What’s powerful about nix is the language IMO. I was able to build an automatic WireGuard setup[1] with tagging that automatically works on each new machine thanks to the ability to do config as code. Just provide some basic config for each machine and the code turns it into an interface with peers.
The issue to me isn’t the language persay (it’s really a tiny surface area language, see the built in/lib functions [2]) but the tooling built around packaging is a hodgepodge mess of semi-documented workarounds (with Nixpkgs blessed ways vs used libraries) and is extremely difficult to approach and understand.
reply