Hacker Read top | best | new | newcomments | leaders | about | bookmarklet login
A packaging tutorial for Guix (www.gnu.org) similar stories update story
130.0 points by _emacsomancer_ | karma 3868 | avg karma 2.65 2018-10-10 17:35:19+00:00 | hide | past | favorite | 34 comments



view as:

Packaging GNU hello is also a focus of examples in the NixPkgs manual[0]. With this Guix tutorial, there is now a litte more pairity between two great tools.

[0] https://nixos.org/nixpkgs/manual/


Speaking of parity, while the Guix project may have some catching up to do in general, I think their documentation has been superior to Nix for a while.

Nix IMO suffers greatly from... using its own weird own language and DSL (Nix) which has its own weird semantics.

I tried NixOS and was even willing to author packages to get stuff I needed working, but I could never wrap my head around the Nix-language.

Guix however, that’s just LISP and there’s few things simpler than that in the entirety of the programming-universe.

IOW you don’t have to learn something new, or get new editor plugins or whatever.

I really want Guix to succeed. Here’s me hoping they won’t hamstring it over licensing issues like the FSF often does.


I always appreciate more documentation when it comes to Nix and Guix. Being still somewhat new to Nix and Guix I found myself looking up other people's declarations on Github because the official documentation was lacking.

I like the idea a lot, although the kind of people who don't like shell scripts for init would likely flip completely out at packages using guile lisp rather than some obscure DSL, and then parenthesis-ophobes are going to react very poorly to the libgit2 extended example ending in four closing parenthesis. A little diversity in thought might be a good thing of course.

An init system written entirely in Guile LISP is interesting to think about. I'm old enough to (barely) remember the 80s lisp machines and since all modern CS/IT progress is re-implementing stuff from four decades ago, we're about due for something like linux ported from C to clojure or another lisp with a new init and the guix packaging system written in guile-scheme-lisp. The idea of an OS written to run on a JVM instead of virtualization like vmware is intriguing. I can come up with enough bright ideas in five minutes to keep me busy for fifty years, so this daydreaming is all very unlikely to happen.


> the kind of people who don't like shell scripts for init would likely flip completely out at packages using guile lisp rather than some obscure DSL

Do you mean people used to systemd specifically or does this also include runit, s6, etc.?

> parenthesis-ophobes are going to react very poorly to the libgit2 extended example ending in four closing parentheses

Though, of course, for humans reading Lisp, the trick is not to focus on the parentheses too much, but use the indentation. And then use the parentheses as a sort of double-sanity check as you move your editor's point/cursor across them and see the other member of the pair light up (assuming a sane lisp editor/editor configuration).

(And maybe: parenthephobes?)


An init system written entirely in Guile LISP is interesting to think about

https://www.gnu.org/software/shepherd/


And Shepherd is very pleasant to interact with.

From the documentation:

> The daemon that runs in the background and is responsible for controlling the services is shepherd, while the user interface tool is called herd: it’s the command that allows you to actually herd your daemons

It seems strange the the herd controls the shepherd, but I guess it fits in well with GNU's user empowerment philosophy.

I'm very much a parenthesis-ophobe, but things like init configuration is a niche guile seems to be perfect for. It's terse enough to replace ini's, it's complex enough to provide structure like xml or json, it's an actual language so no one ends up writing their own and it's trivial to embed, like like lua.


I always thought of the `herd` command as a verb (i.e. what do shepherds do? they `herd` things [well, sheep, I suppose]).


The funny thing is that a DSL looks aesthetically pleasing but is more often than not a nightmare to work with.

Lisp itself is a DSL, it's completely unconstrained by common syntactical conventions (whitespace, block scopes, etc.) and the worst anyone can take away from that setup is that it requires typing ')' too much. Yet there are so many tools invented entirely to manipulate Lisp code so you don't need to rewrite editor plugins and what-not to get fresh syntax highlighting and completion for your brand new language.

Because it's also a programming language, it already supports functions and other constructs for abstraction and code reuse, should you need them for larger scripts with lots of boilerplate.

But tend to reject it and the already done work because of the aesthetics, and then we're going into the GitHub repos for things like Helm, Kubernetes, Terraform, etc. because the custom YAML/JSON-esque templating setup misses vital features and we're having to re-learn the same stuff over and over again.


One thing that puzzles me is that many people who complain about parenthesis are perfectly willing to deal with strange and inconvenient delimiter characters, such as semicolons and commas.

Typically commas and semicolons don't have to be matched with open-commas and open-semicolons and you won't see several of them together like ';;;;'. There is a lot less ambiguity over what is ending than something like this example (worst case) from TFA:

> (lambda _ (for-each make-file-writable (find-files "." ".*")))))))

',' and ';' are far more convenient too, the sit right next to m and l, parenthesis require finger stretching and the shift key.


I'm using paredit, so I'm never typing out closing parentheses. I'm just slurping and barfing s-expressions all over the place. I don't see parentheses but syntax units that my editor ensures remain correct.

(I can understand the frustration if I was forced not to use Emacs, which has s-expression navigation commands, and if the use of paredit were forbidden.)


It's the great advantage of Lisp syntax that it supports structural editing in a complete and unified way, and paredit enables the leap from editing code by manipulating plain lines of characters to directly editing code as hierarchical structure.

> Typically commas and semicolons don't have to be matched with open-commas and open-semicolons

This is a problem that is non-existent when you use common tooling to write Lisp, like ParEdit or Smartparens. Since nobody would write C++ in Microsoft Word, nobody writes a lot of Lisp code without auto-matching parentheses tools.


What C and its descendents have instead is numerous repetitions of '\s}\n' to end

if-then-elseif-else switch-case for while until functions methods classes

and good luck figuring out which matches which when there's more than a screen between the beginning and the end of a block


> parenthesis-ophobes are going to react very poorly to the libgit2 extended example ending in four closing parenthesis

I think parenthesis-phobia is a red herring. The delimiter bytes in s-expressions denote structure, and most s-expression users are happy to render them as parentheses glyphs. Those who aren't are welcome to use one of the myriad preprocessors/renderers which display this structure in other ways, e.g.

https://srfi.schemers.org/srfi-119/srfi-119.html#Implementat... (converts back and forth to indentation)

https://srfi.schemers.org/srfi-110/srfi-110.html#reference-i... (converts back and forth to indentation)

https://srfi.schemers.org/srfi-49/srfi-49.html#Implementatio... (converts back and forth to indentation)

https://sourceforge.net/p/readable/code/ci/develop/tree/src/... (converts to indentation; an 'unsweeten.sscm' script converts the other way)

https://github.com/mjsottile/sfsexp/blob/master/src/sexp_vis... (converts to a graphviz tree diagram)

http://www.foldr.org/~michaelw/emacs/mwe-color-box.el (shows tree structure using nested boxes of different colours)

The fact that such tools aren't widely used shows that parentheses aren't a/the problem.


Another Guile component is mcron[1] which allows cron jobs to be scheduled using Lispy syntax as well. I have, for instance:

  ;; fetch mail via isync/mbsync for mu/mu4e every 5 minutes
  (job '(next-minute (range 0 60 5)) "/home/$USER/.guix-profile/bin/mbsync -a")

  ;; run backup every hour
  (job '(next-hour (range 0 23 1)) "/home/$USER/.local/bin/borg-backup.sh")

[1]: https://www.gnu.org/software/mcron/manual/html_node/index.ht...

If you want to know why mcron as opposed to cron, you only need to look at the introduction: https://www.gnu.org/software/mcron/manual/html_node/Introduc...

I still wish there were a few more examples of actual guile cron jobs, but there is one example which illustrates the power of using guile for defining cron jobs in the 'every second Sunday' example:

  (job (lambda (current-time)
         (let* ((next-month (next-month-from current-time))
                (first-day (tm:wday (localtime next-month)))
                (second-sunday (if (eqv? first-day 0)
                                   8
                                   (- 14 first-day))))
           (+ next-month (* 24 60 60 second-sunday))))
       "my-program")

My main problem with guix, and nix, and debs (the 3 i have tried is they are great for software wrapped using automake/autoconf, but I can never figure out how to package things using a custom build system.

It's certainly doable. I contributed Envoy to Nixpkgs (built with bazel) and developed the current incarnation of the RubyGems/Bundler build infrastructure.

Guix and Nix just run scripts. I've less experience with Guix, but in Nix you can just run a bash script which stores the build products in a path given as `$out`. For example, the following is a perfectly valid Nix package, if we replaced the nonsense names and commands:

    with import <nixpkgs> {};
    runCommand "package-name-goes-here"
      {
        buildInputs = [ build tools go here ];
        envVar1 = "hello";
        envVar2 = "world";
      }
      ''
        bash command 1
        bash command 2
        mv my-amazing-output "$out"
      ''
For example, I've never written any Go, but I was able to package some Go programs by copy-pasting their install instructions into the above template and putting the relevant build tools into `buildInputs`: https://github.com/Warbo/warbo-packages/blob/master/packages...

In contrast, the automake/autoconf support is actually a very complicated tower of helper scripts, override hooks, etc. which is built up on top of the system's built in support for running bash scripts (actually it runs whatever we call `builder` with whatever we call `args`; `runCommand` sets the `builder` to be bash and the `args` to be a file containing our script).


While Guix eventually runs a builder script, the script is 1) a Guile script and 2) not directly provided by the user but the result of compilation.

Guix does not stitch shell scripts together. Builds in Guix are split into build phases, which are Guile procedures. Build system abstractions are collections of these procedures. A packager can inject additional build phases or delete existing build phases from these build systems on a package by package basis.

The `trivial-build-system` is used for packages where none of the existing build systems are applicable, not even with build phases removed or added. This works but is not used very often as it is often more convenient to start with something like the `gnu-build-system` and to replace phases by providing new procedures; this is done to benefit from build phases that the `gnu-build-system` provides, such as those that unpack the sources, patch shebangs, or validate the binaries.


Thanks for the Guix info, as I say I've not really used it. The use of "phases", etc. is what I was getting at w.r.t. Nix's complicated "autoconf/automake support" (AKA `stdenv.mkDerivation`).

As you say, Guix's use of Scheme rather than stiching shell scripts together should be more powerful (in particular, concatenating strings of bash results in a rather opaque builder; whilst a more declarative s-expression approach would make traversing and modifying the builder more feasible)


In your derivation, do

    builder = ./builder.sh
and then write a shell script named builder.sh that builds what you want.

This is documented at https://nixos.org/nix/manual/#chap-writing-nix-expressions


Guix provides build system abstractions like `gnu-build-system` for traditional build systems, but you can easily override or extend the build phases. You can run scripts with `(invoke "my-script.sh")`.

The bigger problem, in my opinion, is that packages that use custom build systems often have strange ideas about what is desirable in a build system. Sometimes they prefer to download things during the build (that's not permitted in Guix) or they will make assumptions about the layout of directories or the discovery of dependencies.

Patching away all of these invalid assumptions can be annoying, but it is possible as is demonstrated in many package definitions in the `(gnu packages bioinformatics)` module.


I recently realized that Macports package definition files, Portfiles [1], are actually just TCL. It looks declarative, but that's just because they defined the functions well (and that's how TCL syntax works)

And "Portgroups" are effectively libraries for the packaging system, so that they can easily interface with GitHub, CMake, gobject_introspection, etc...

[1] https://github.com/macports/macports-ports


It's neat how similar this looks to a Bitbake recipe (used by the Yocto Linux project), except that those are written in a DSL that's half Python and half shell-script.

Check out a similar recipe here https://github.com/openembedded/openembedded-core/blob/maste...


Recently I have gotten very interested in Guix, after 16 years of only Debian. Guix keeps surprising me, most recently its container system. This page is exactly what I was looking for to get started with my own packages.

Why are you surprised by the container system ?

Legal | privacy