Usually there are tools to extract the necessary dependency information (url and hash) from a package manager's lockfile and produce a fixed-output derivation so a Nix expression can reproduce the build environment. I know of yarn2nix, cargo2nix, and node2nix, and I wrote and maintain gradle2nix for JVM projects.
You may find something like node2nix helpful (https://github.com/svanderburg/node2nix). This converts your package.json into a Nix expression that can then be cached. You're right that it does require some setup and a bit of Nix knowledge but could yield significant benefits and take a good chunk out of that 20 minutes.
Another option might be to use pnpm instead of Yarn and cache your pnpm dependencies. pnpm actually works a bit like Nix in that it creates a pnpm-lock.yaml file with content-based hashes for the full package.json dependency tree. This enables it to quickly determine which parts of the dependency tree it needs to build and which are already available.
(Before using this for real, I'd certainly want to be able to track the exact version I'm getting of a dependency, with a hash, in a "lockfile" sort of file that I can keep in version control. That's not mentioned in the demo on the front page - but if it doesn't already have that feature, I hope it will grow it.)
I'm actually working on a blog article about our setup (or possibly a series of articles, depending on how much longer it gets), it'll be published on https://korz.dev once its done. In the meantime, here's the rough summary:
- Go projects are built with https://github.com/nix-community/gomod2nix. We generate a list of internal packages a project depends on using `go list -json` that is then passed to gomod2nix's `buildGoApplication`.
- Rust projects are built with https://github.com/cargo2nix/cargo2nix. We chose cargo2nix to get incremental builds, meaning that dependency builds can be shared between our Rust projects and that not all dependencies have to be rebuilt when adding/updating/removing dependencies from a project.
- Node projects are far trickier if you want them to be able to share dependencies and depend on one another. We have found Yarn 4 (currently release candidate, not stable yet) in combination with https://github.com/madjam002/yarnpnp2nix to work best for this. Unfortunately we have to patch some package hashes of packages that contain platform-specific binaries (such as esbuild).
- Container images are built with https://github.com/nlewo/nix2container instead of Nix's built-in docker tools, because nix2container allows configuration of layers and thus makes it easier to share layers between images and improve layer caching.
As for integrating it into our CI pipeline, this is pretty much CI system agnostic as all you have to do is run `nix build ...` inside the pipeline, generating a `result` directory or file as an artifact that, in the case of nix2container, can be pushed to your container image registry.
The lockfile in this post actually seems a bit more like a manifest, or at least that it's trying to do both things at once.
I'd expect you'd have human readable file to list your dependencies and versions in a simplistic way, and then a separate machine readable file that locks the versions for reproducibility. As it stands, the example in the article does not look pleasant to write by hand so you'd have to script something yourself.
I started building this at one point. Basically there's an accompanying manifest.toml that specifies permissions for the packages with their checksums, and then it can traverse the dependency graph finding each manifest.toml.
It also generated a manifest.lock so if manifests changed you would know about it.
Then once it built up the sandbox it would execute the build. If no dependencies require networking, for example, it gets no networking, etc.
I stopped working on it because I didn't have time, and it obviously relied on everyone doing the work of writing a manifest.toml and using my tool, plus it only supported rust and crates.io
TBH it seems really easy to solve this problem, it's very well worn territory - Browser extensions have been doing the same thing for decades. Similarly, why can I upload a package with a near-0 string distance to another package? That'd help a massive amount against typosquatting.
No one wants to implement it who also implements package managers I guess.
We're working on that for Apters (http://apters.com/). We have a simple language used to describe an individual package build, which acts as a module that you can import from another package that needs it as a build dependency. The whole thing uses content hashes for versioning, like Git, and caches any build you've already run once.
The whole thing is Open Source; we'll also have a hosted version as a service.
Spack essentially is nix with dependency resolution. spack packages can have conditional (nearly) anything: versions, options, compiler choices, dependencies (including virtual deps). The resolver can make choices about all of those.
Re: dependency management and build tooling: these days I'm using opam-monorepo (https://github.com/tarides/opam-monorepo, previously known as duniverse) to manage dependencies and it's working out pretty well for me. It vendors all dependencies into a node_modules-like directory inside the projects and writes an opam-compatible lockfile, and it builds all the dependencies using dune. For the most widely used packages that don't already use dune it uses an overlay containing dune ports at https://github.com/dune-universe/opam-overlays.
I really like this proposal. It's basically the best of all current dependency tools - you get reproducibility without needing external configuration files, without hugely long vendor import statements, and still supporting raw go-get.
(1) Use a package manager, which stores hashsums in a lock file. (2) Install your dependencies from a lock file as spec. (3) Do not trust version numbers. Trust hash sums. Do not believe in "But I set the version number!". (4) Do not rely on downloads Again, trust hash sums, not URLs. (5) Hashsums!!! (6) Wherever there is randomness as in random number generators, use a seed. If the interface does not allow to specify the seed, thtow the trash away and use another generator. Careful when concurrency is involved. It might destroy reproducibility. For example this was the case with Tensorflow. Not sure it still is. (7) Use a version control system.
I guess there's CI services that pull latest versions of dependencies (in the accepted version range), build, run tests and submit a pull request if everything seems ok.
Edit: actually no I don't know, there would be nothing to submit if no lockfile is used.
To add to this, Google spent a lot of time and effort on building a custom build system (Bazel) to make this sort of dependency tracking in a mono-repo easy. It's pretty great, and I'd rather write BUILD files than Makefiles/shell scripts any day, but the OP appears to have been focusing on extremely simple projects, where shell scripts are okay.
This really depends on the specific package manager: if you're building an application in Rust, its lockfile will contain the full tree of dependencies, locked to a specific version.
reply