futurile/guix-play
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22dd558c70
guix-play/doc/guix-cookbook.texi
Ludovic Courtès 47c1de22df
doc: cookbook: Add "Installing Guix on a Cluster" chapter. 
This is derived from the article at
<https://hpc.guix.info/blog/2017/11/installing-guix-on-a-cluster/>, with
clarifications and updates.

* doc/guix-cookbook.texi (Installing Guix on a Cluster): New chapter.
2023-01-09 17:40:53 +01:00

4074 lines
144 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

\input texinfo
@c -*-texinfo-*-
@c %**start of header
@setfilename guix-cookbook.info
@documentencoding UTF-8
@settitle GNU Guix Cookbook
@c %**end of header
@c Onion service for ci.guix.gnu.org.
@set SUBSTITUTE-TOR-URL https://4zwzi66wwdaalbhgnix55ea3ab4pvvw66ll2ow53kjub6se4q2bclcyd.onion
@copying
Copyright @copyright{} 2019, 2022 Ricardo Wurmus@*
Copyright @copyright{} 2019 Efraim Flashner@*
Copyright @copyright{} 2019 Pierre Neidhardt@*
Copyright @copyright{} 2020 Oleg Pykhalov@*
Copyright @copyright{} 2020 Matthew Brooks@*
Copyright @copyright{} 2020 Marcin Karpezo@*
Copyright @copyright{} 2020 Brice Waegeneire@*
Copyright @copyright{} 2020 André Batista@*
Copyright @copyright{} 2020 Christine Lemmer-Webber@*
Copyright @copyright{} 2021 Joshua Branson@*
Copyright @copyright{} 2022 Maxim Cournoyer@*
Copyright @copyright{} 2023 Ludovic Courtès
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A
copy of the license is included in the section entitled ``GNU Free
Documentation License''.
@end copying
@dircategory System administration
@direntry
* Guix cookbook: (guix-cookbook). Tutorials and examples for GNU Guix.
@end direntry
@titlepage
@title GNU Guix Cookbook
@subtitle Tutorials and examples for using the GNU Guix Functional Package Manager
@author The GNU Guix Developers
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage
@contents
@c *********************************************************************
@node Top
@top GNU Guix Cookbook
This document presents tutorials and detailed examples for GNU@tie{}Guix, a
functional package management tool written for the GNU system. Please
@pxref{Top,,, guix, GNU Guix reference manual} for details about the system,
its API, and related concepts.
@c TRANSLATORS: You can replace the following paragraph with information on
@c how to join your own translation team and how to report issues with the
@c translation.
This manual is also available in French (@pxref{Top,,, guix-cookbook.fr,
Livre de recettes de GNU Guix}) and German (@pxref{Top,,,
guix-cookbook.de, GNU-Guix-Kochbuch}). If you would like to translate
this document in your native language, consider joining
@uref{https://translate.fedoraproject.org/projects/guix/documentation-cookbook,
Weblate} (@pxref{Translating Guix,,, guix, GNU Guix reference manual}).
@menu
* Scheme tutorials:: Meet your new favorite language!
* Packaging:: Packaging tutorials
* System Configuration:: Customizing the GNU System
* Containers:: Isolated environments and nested systems
* Advanced package management:: Power to the users!
* Environment management:: Control environment
* Installing Guix on a Cluster:: High-performance computing.
* Acknowledgments:: Thanks!
* GNU Free Documentation License:: The license of this document.
* Concept Index:: Concepts.
@detailmenu
--- The Detailed Node Listing ---
Packaging
* Packaging Tutorial:: A tutorial on how to add packages to Guix.
System Configuration
* Auto-Login to a Specific TTY:: Automatically Login a User to a Specific TTY
* Customizing the Kernel:: Creating and using a custom Linux kernel on Guix System.
* Guix System Image API:: Customizing images to target specific platforms.
* Using security keys:: How to use security keys with Guix System.
* Connecting to Wireguard VPN:: Connecting to a Wireguard VPN.
* Customizing a Window Manager:: Handle customization of a Window manager on Guix System.
* Running Guix on a Linode Server:: Running Guix on a Linode Server
* Setting up a bind mount:: Setting up a bind mount in the file-systems definition.
* Getting substitutes from Tor:: Configuring Guix daemon to get substitutes through Tor.
* Setting up NGINX with Lua:: Configuring NGINX web-server to load Lua modules.
* Music Server with Bluetooth Audio:: Headless music player with Bluetooth output.
Containers
* Guix Containers:: Perfectly isolated environments
* Guix System Containers:: A system inside your system
Advanced package management
* Guix Profiles in Practice:: Strategies for multiple profiles and manifests.
Environment management
* Guix environment via direnv:: Setup Guix environment with direnv
Installing Guix on a Cluster
* Setting Up a Head Node:: The node that runs the daemon.
* Setting Up Compute Nodes:: Client nodes.
* Cluster Network Access:: Dealing with network access restrictions.
* Cluster Disk Usage:: Disk usage considerations.
* Cluster Security Considerations:: Keeping the cluster secure.
@end detailmenu
@end menu
@c *********************************************************************
@node Scheme tutorials
@chapter Scheme tutorials
GNU@tie{}Guix is written in the general purpose programming language Scheme,
and many of its features can be accessed and manipulated programmatically.
You can use Scheme to generate package definitions, to modify them, to build
them, to deploy whole operating systems, etc.
Knowing the basics of how to program in Scheme will unlock many of the
advanced features Guix provides --- and you don't even need to be an
experienced programmer to use them!
Let's get started!
@node A Scheme Crash Course
@section A Scheme Crash Course
@cindex Scheme, crash course
Guix uses the Guile implementation of Scheme. To start playing with the
language, install it with @code{guix install guile} and start a
@dfn{REPL}---short for @uref{https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop,
@dfn{read-eval-print loop}}---by running @code{guile} from the command line.
Alternatively you can also run @code{guix shell guile -- guile}
if you'd rather not have Guile installed in your user profile.
In the following examples, lines show what you would type at the REPL;
lines starting with ``@result{}'' show evaluation results, while lines
starting with ``@print{}'' show things that get printed. @xref{Using Guile
Interactively,,, guile, GNU Guile Reference Manual}, for more details on the
REPL.
@itemize
@item
Scheme syntax boils down to a tree of expressions (or @emph{s-expression} in
Lisp lingo). An expression can be a literal such as numbers and strings, or a
compound which is a parenthesized list of compounds and literals. @code{#true}
and @code{#false} (abbreviated @code{#t} and @code{#f}) stand for the
Booleans ``true'' and ``false'', respectively.
Examples of valid expressions:
@lisp
"Hello World!"
@result{} "Hello World!"
17
@result{} 17
(display (string-append "Hello " "Guix" "\n"))
@print{} Hello Guix!
@result{} #<unspecified>
@end lisp
@item
This last example is a function call nested in another function call. When a
parenthesized expression is evaluated, the first term is the function and the
rest are the arguments passed to the function. Every function returns the
last evaluated expression as its return value.
@item
Anonymous functions are declared with the @code{lambda} term:
@lisp
(lambda (x) (* x x))
@result{} #<procedure 120e348 at <unknown port>:24:0 (x)>
@end lisp
The above procedure returns the square of its argument. Since everything is
an expression, the @code{lambda} expression returns an anonymous procedure,
which can in turn be applied to an argument:
@lisp
((lambda (x) (* x x)) 3)
@result{} 9
@end lisp
@item
Anything can be assigned a global name with @code{define}:
@lisp
(define a 3)
(define square (lambda (x) (* x x)))
(square a)
@result{} 9
@end lisp
@item
Procedures can be defined more concisely with the following syntax:
@lisp
(define (square x) (* x x))
@end lisp
@item
A list structure can be created with the @code{list} procedure:
@lisp
(list 2 a 5 7)
@result{} (2 3 5 7)
@end lisp
@item
The @dfn{quote} disables evaluation of a parenthesized expression: the
first term is not called over the other terms (@pxref{Expression Syntax,
quote,, guile, GNU Guile Reference Manual}). Thus it effectively
returns a list of terms.
@lisp
'(display (string-append "Hello " "Guix" "\n"))
@result{} (display (string-append "Hello " "Guix" "\n"))
'(2 a 5 7)
@result{} (2 a 5 7)
@end lisp
@item
The @dfn{quasiquote} disables evaluation of a parenthesized expression
until @dfn{unquote} (a comma) re-enables it. Thus it provides us with
fine-grained control over what is evaluated and what is not.
@lisp
`(2 a 5 7 (2 ,a 5 ,(+ a 4)))
@result{} (2 a 5 7 (2 3 5 7))
@end lisp
Note that the above result is a list of mixed elements: numbers, symbols (here
@code{a}) and the last element is a list itself.
@item
Multiple variables can be named locally with @code{let} (@pxref{Local
Bindings,,, guile, GNU Guile Reference Manual}):
@lisp
(define x 10)
(let ((x 2)
(y 3))
(list x y))
@result{} (2 3)
x
@result{} 10
y
@error{} In procedure module-lookup: Unbound variable: y
@end lisp
Use @code{let*} to allow later variable declarations to refer to earlier
definitions.
@lisp
(let* ((x 2)
(y (* x 3)))
(list x y))
@result{} (2 6)
@end lisp
@item
@dfn{Keywords} are typically used to identify the named parameters of a
procedure. They are prefixed by @code{#:} (hash, colon) followed by
alphanumeric characters: @code{#:like-this}.
@xref{Keywords,,, guile, GNU Guile Reference Manual}.
@item
The percentage @code{%} is typically used for read-only global variables in
the build stage. Note that it is merely a convention, like @code{_} in C.
Scheme treats @code{%} exactly the same as any other letter.
@item
Modules are created with @code{define-module} (@pxref{Creating Guile
Modules,,, guile, GNU Guile Reference Manual}). For instance
@lisp
(define-module (guix build-system ruby)
#:use-module (guix store)
#:export (ruby-build
ruby-build-system))
@end lisp
defines the module @code{guix build-system ruby} which must be located in
@file{guix/build-system/ruby.scm} somewhere in the Guile load path. It
depends on the @code{(guix store)} module and it exports two variables,
@code{ruby-build} and @code{ruby-build-system}.
@end itemize
@quotation Going further
Scheme is a language that has been widely used to teach programming and
you'll find plenty of material using it as a vehicle. Here's a
selection of documents to learn more about Scheme:
@itemize
@item
@uref{https://spritely.institute/static/papers/scheme-primer.html, @i{A
Scheme Primer}}, by Christine Lemmer-Webber and the Spritely Institute.
@item
@uref{http://www.troubleshooters.com/codecorn/scheme_guile/hello.htm,
@i{Scheme at a Glance}}, by Steve Litt.
@item
@c There used to be a copy at mitpress.mit.edu but it vanished.
@uref{https://sarabander.github.io/sicp/,
@i{Structure and Interpretation of Computer Programs}}, by Harold
Abelson and Gerald Jay Sussman, with Julie Sussman. Colloquially known
as ``SICP'', this book is a reference.
You can also install it and read it from your computer:
@example
guix install sicp info-reader
info sicp
@end example
@end itemize
You'll find more books, tutorials and other resources at
@url{https://schemers.org/}.
@end quotation
@c *********************************************************************
@node Packaging
@chapter Packaging
@cindex packaging
This chapter is dedicated to teaching you how to add packages to the
collection of packages that come with GNU Guix. This involves writing package
definitions in Guile Scheme, organizing them in package modules, and building
them.
@menu
* Packaging Tutorial:: A tutorial on how to add packages to Guix.
@end menu
@node Packaging Tutorial
@section Packaging Tutorial
GNU Guix stands out as the @emph{hackable} package manager, mostly because it
uses @uref{https://www.gnu.org/software/guile/, GNU Guile}, a powerful
high-level programming language, one of the
@uref{https://en.wikipedia.org/wiki/Scheme_%28programming_language%29, Scheme}
dialects from the
@uref{https://en.wikipedia.org/wiki/Lisp_%28programming_language%29, Lisp family}.
Package definitions are also written in Scheme, which empowers Guix in some
very unique ways, unlike most other package managers that use shell scripts or
simple languages.
@itemize
@item
Use functions, structures, macros and all of Scheme expressiveness for your
package definitions.
@item
Inheritance makes it easy to customize a package by inheriting from it and
modifying only what is needed.
@item
Batch processing: the whole package collection can be parsed, filtered and
processed. Building a headless server with all graphical interfaces stripped
out? It's possible. Want to rebuild everything from source using specific
compiler optimization flags? Pass the @code{#:make-flags "..."} argument to
the list of packages. It wouldn't be a stretch to think
@uref{https://wiki.gentoo.org/wiki/USE_flag, Gentoo USE flags} here, but this
goes even further: the changes don't have to be thought out beforehand by the
packager, they can be @emph{programmed} by the user!
@end itemize
The following tutorial covers all the basics around package creation with Guix.
It does not assume much knowledge of the Guix system nor of the Lisp language.
The reader is only expected to be familiar with the command line and to have some
basic programming knowledge.
@node A ``Hello World'' package
@subsection A ``Hello World'' package
The ``Defining Packages'' section of the manual introduces the basics of Guix
packaging (@pxref{Defining Packages,,, guix, GNU Guix Reference Manual}). In
the following section, we will partly go over those basics again.
GNU@tie{}Hello is a dummy project that serves as an idiomatic example for
packaging. It uses the GNU build system (@code{./configure && make && make
install}). Guix already provides a package definition which is a perfect
example to start with. You can look up its declaration with @code{guix edit
hello} from the command line. Let's see how it looks:
@lisp
(define-public hello
(package
(name "hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, GNU world: An example GNU package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+)))
@end lisp
As you can see, most of it is rather straightforward. But let's review the
fields together:
@table @samp
@item name
The project name. Using Scheme conventions, we prefer to keep it
lower case, without underscore and using dash-separated words.
@item source
This field contains a description of the source code origin. The
@code{origin} record contains these fields:
@enumerate
@item The method, here @code{url-fetch} to download via HTTP/FTP, but other methods
exist, such as @code{git-fetch} for Git repositories.
@item The URI, which is typically some @code{https://} location for @code{url-fetch}. Here
the special `mirror://gnu` refers to a set of well known locations, all of
which can be used by Guix to fetch the source, should some of them fail.
@item The @code{sha256} checksum of the requested file. This is essential to ensure
the source is not corrupted. Note that Guix works with base32 strings,
hence the call to the @code{base32} function.
@end enumerate
@item build-system
This is where the power of abstraction provided by the Scheme language really
shines: in this case, the @code{gnu-build-system} abstracts away the famous
@code{./configure && make && make install} shell invocations. Other build
systems include the @code{trivial-build-system} which does not do anything and
requires from the packager to program all the build steps, the
@code{python-build-system}, the @code{emacs-build-system}, and many more
(@pxref{Build Systems,,, guix, GNU Guix Reference Manual}).
@item synopsis
It should be a concise summary of what the package does. For many packages a
tagline from the project's home page can be used as the synopsis.
@item description
Same as for the synopsis, it's fine to re-use the project description from the
homepage. Note that Guix uses Texinfo syntax.
@item home-page
Use HTTPS if available.
@item license
See @code{guix/licenses.scm} in the project source for a full list of
available licenses.
@end table
Time to build our first package! Nothing fancy here for now: we will stick to a
dummy @code{my-hello}, a copy of the above declaration.
As with the ritualistic ``Hello World'' taught with most programming languages,
this will possibly be the most ``manual'' approach. We will work out an ideal
setup later; for now we will go the simplest route.
Save the following to a file @file{my-hello.scm}.
@lisp
(use-modules (guix packages)
(guix download)
(guix build-system gnu)
(guix licenses))
(package
(name "my-hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, Guix world: An example custom Guix package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+))
@end lisp
We will explain the extra code in a moment.
Feel free to play with the different values of the various fields. If you
change the source, you'll need to update the checksum. Indeed, Guix refuses to
build anything if the given checksum does not match the computed checksum of the
source code. To obtain the correct checksum of the package declaration, we
need to download the source, compute the sha256 checksum and convert it to
base32.
Thankfully, Guix can automate this task for us; all we need is to provide the
URI:
@c TRANSLATORS: This is example shell output.
@example sh
$ guix download mirror://gnu/hello/hello-2.10.tar.gz
Starting download of /tmp/guix-file.JLYgL7
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirror.ibcp.fr/pub/gnu/hello/hello-2.10.tar.gz'...
…10.tar.gz 709KiB 2.5MiB/s 00:00 [##################] 100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
@end example
In this specific case the output tells us which mirror was chosen.
If the result of the above command is not the same as in the above snippet,
update your @code{my-hello} declaration accordingly.
Note that GNU package tarballs come with an OpenPGP signature, so you
should definitely check the signature of this tarball with `gpg` to
authenticate it before going further:
@c TRANSLATORS: This is example shell output.
@example sh
$ guix download mirror://gnu/hello/hello-2.10.tar.gz.sig
Starting download of /tmp/guix-file.03tFfb
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz.sig...
following redirection to `https://ftp.igh.cnrs.fr/pub/gnu/hello/hello-2.10.tar.gz.sig'...
….tar.gz.sig 819B 1.2MiB/s 00:00 [##################] 100.0%
/gnu/store/rzs8wba9ka7grrmgcpfyxvs58mly0sx6-hello-2.10.tar.gz.sig
0q0v86n3y38z17rl146gdakw9xc4mcscpk8dscs412j22glrv9jf
$ gpg --verify /gnu/store/rzs8wba9ka7grrmgcpfyxvs58mly0sx6-hello-2.10.tar.gz.sig /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
gpg: Signature made Sun 16 Nov 2014 01:08:37 PM CET
gpg: using RSA key A9553245FDE9B739
gpg: Good signature from "Sami Kerola <kerolasa@@iki.fi>" [unknown]
gpg: aka "Sami Kerola (http://www.iki.fi/kerolasa/) <kerolasa@@iki.fi>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 8ED3 96E3 7E38 D471 A005 30D3 A955 3245 FDE9 B739
@end example
You can then happily run
@c TRANSLATORS: Do not translate this command
@example sh
$ guix package --install-from-file=my-hello.scm
@end example
You should now have @code{my-hello} in your profile!
@c TRANSLATORS: Do not translate this command
@example sh
$ guix package --list-installed=my-hello
my-hello 2.10 out
/gnu/store/f1db2mfm8syb8qvc357c53slbvf1g9m9-my-hello-2.10
@end example
We've gone as far as we could without any knowledge of Scheme. Before moving
on to more complex packages, now is the right time to brush up on your Scheme
knowledge. @pxref{A Scheme Crash Course} to get up to speed.
@node Setup
@subsection Setup
In the rest of this chapter we will rely on some basic Scheme
programming knowledge. Now let's detail the different possible setups
for working on Guix packages.
There are several ways to set up a Guix packaging environment.
We recommend you work directly on the Guix source checkout since it makes it
easier for everyone to contribute to the project.
But first, let's look at other possibilities.
@node Local file
@subsubsection Local file
This is what we previously did with @samp{my-hello}. With the Scheme basics we've
covered, we are now able to explain the leading chunks. As stated in @code{guix
package --help}:
@example
-f, --install-from-file=FILE
install the package that the code within FILE
evaluates to
@end example
Thus the last expression @emph{must} return a package, which is the case in our
earlier example.
The @code{use-modules} expression tells which of the modules we need in the file.
Modules are a collection of values and procedures. They are commonly called
``libraries'' or ``packages'' in other programming languages.
@node @samp{GUIX_PACKAGE_PATH}
@subsubsection @samp{GUIX_PACKAGE_PATH}
@emph{Note: Starting from Guix 0.16, the more flexible Guix @dfn{channels} are the
preferred way and supersede @samp{GUIX_PACKAGE_PATH}. See next section.}
It can be tedious to specify the file from the command line instead of simply
calling @code{guix package --install my-hello} as you would do with the official
packages.
Guix makes it possible to streamline the process by adding as many ``package
declaration directories'' as you want.
Create a directory, say @file{~/guix-packages} and add it to the @samp{GUIX_PACKAGE_PATH}
environment variable:
@example
$ mkdir ~/guix-packages
$ export GUIX_PACKAGE_PATH=~/guix-packages
@end example
To add several directories, separate them with a colon (@code{:}).
Our previous @samp{my-hello} needs some adjustments though:
@lisp
(define-module (my-hello)
#:use-module (guix licenses)
#:use-module (guix packages)
#:use-module (guix build-system gnu)
#:use-module (guix download))
(define-public my-hello
(package
(name "my-hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, Guix world: An example custom Guix package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+)))
@end lisp
Note that we have assigned the package value to an exported variable name with
@code{define-public}. This is effectively assigning the package to the @code{my-hello}
variable so that it can be referenced, among other as dependency of other
packages.
If you use @code{guix package --install-from-file=my-hello.scm} on the above file, it
will fail because the last expression, @code{define-public}, does not return a
package. If you want to use @code{define-public} in this use-case nonetheless, make
sure the file ends with an evaluation of @code{my-hello}:
@lisp
; ...
(define-public my-hello
; ...
)
my-hello
@end lisp
This last example is not very typical.
Now @samp{my-hello} should be part of the package collection like all other official
packages. You can verify this with:
@example
$ guix package --show=my-hello
@end example
@node Guix channels
@subsubsection Guix channels
Guix 0.16 features channels, which is very similar to @samp{GUIX_PACKAGE_PATH} but
provides better integration and provenance tracking. Channels are not
necessarily local, they can be maintained as a public Git repository for
instance. Of course, several channels can be used at the same time.
@xref{Channels,,, guix, GNU Guix Reference Manual} for setup details.
@node Direct checkout hacking
@subsubsection Direct checkout hacking
Working directly on the Guix project is recommended: it reduces the friction
when the time comes to submit your changes upstream to let the community benefit
from your hard work!
Unlike most software distributions, the Guix repository holds in one place both
the tooling (including the package manager) and the package definitions. This
choice was made so that it would give developers the flexibility to modify the
API without breakage by updating all packages at the same time. This reduces
development inertia.
Check out the official @uref{https://git-scm.com/, Git} repository:
@example
$ git clone https://git.savannah.gnu.org/git/guix.git
@end example
In the rest of this article, we use @samp{$GUIX_CHECKOUT} to refer to the location of
the checkout.
Follow the instructions in the manual (@pxref{Contributing,,, guix, GNU Guix
Reference Manual}) to set up the repository environment.
Once ready, you should be able to use the package definitions from the
repository environment.
Feel free to edit package definitions found in @samp{$GUIX_CHECKOUT/gnu/packages}.
The @samp{$GUIX_CHECKOUT/pre-inst-env} script lets you use @samp{guix} over the package
collection of the repository (@pxref{Running Guix Before It Is
Installed,,, guix, GNU Guix Reference Manual}).
@itemize
@item
Search packages, such as Ruby:
@example
$ cd $GUIX_CHECKOUT
$ ./pre-inst-env guix package --list-available=ruby
ruby 1.8.7-p374 out gnu/packages/ruby.scm:119:2
ruby 2.1.6 out gnu/packages/ruby.scm:91:2
ruby 2.2.2 out gnu/packages/ruby.scm:39:2
@end example
@item
Build a package, here Ruby version 2.1:
@example
$ ./pre-inst-env guix build --keep-failed ruby@@2.1
/gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6
@end example
@item
Install it to your user profile:
@example
$ ./pre-inst-env guix package --install ruby@@2.1
@end example
@item
Check for common mistakes:
@example
$ ./pre-inst-env guix lint ruby@@2.1
@end example
@end itemize
Guix strives at maintaining a high packaging standard; when contributing to the
Guix project, remember to
@itemize
@item
follow the coding style (@pxref{Coding Style,,, guix, GNU Guix Reference Manual}),
@item
and review the check list from the manual (@pxref{Submitting Patches,,, guix, GNU Guix Reference Manual}).
@end itemize
Once you are happy with the result, you are welcome to send your contribution to
make it part of Guix. This process is also detailed in the manual. (@pxref{Contributing,,, guix, GNU Guix Reference Manual})
It's a community effort so the more join in, the better Guix becomes!
@node Extended example
@subsection Extended example
The above ``Hello World'' example is as simple as it goes. Packages can be more
complex than that and Guix can handle more advanced scenarios. Let's look at
another, more sophisticated package (slightly modified from the source):
@lisp
(define-module (gnu packages version-control)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix utils)
#:use-module (guix packages)
#:use-module (guix git-download)
#:use-module (guix build-system cmake)
#:use-module (gnu packages ssh)
#:use-module (gnu packages web)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages python)
#:use-module (gnu packages compression)
#:use-module (gnu packages tls))
(define-public my-libgit2
(let ((commit "e98d0a37c93574d2c6107bf7f31140b548c6a7bf")
(revision "1"))
(package
(name "my-libgit2")
(version (git-version "0.26.6" revision commit))
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/libgit2/libgit2/")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"17pjvprmdrx4h6bb1hhc98w9qi6ki7yl57f090n9kbhswxqfs7s3"))
(patches (search-patches "libgit2-mtime-0.patch"))
(modules '((guix build utils)))
;; Remove bundled software.
(snippet '(delete-file-recursively "deps"))))
(build-system cmake-build-system)
(outputs '("out" "debug"))
(arguments
`(#:tests? #true ; Run the test suite (this is the default)
#:configure-flags '("-DUSE_SHA1DC=ON") ; SHA-1 collision detection
#:phases
(modify-phases %standard-phases
(add-after 'unpack 'fix-hardcoded-paths
(lambda _
(substitute* "tests/repo/init.c"
(("#!/bin/sh") (string-append "#!" (which "sh"))))
(substitute* "tests/clar/fs.h"
(("/bin/cp") (which "cp"))
(("/bin/rm") (which "rm")))))
;; Run checks more verbosely.
(replace 'check
(lambda _ (invoke "./libgit2_clar" "-v" "-Q")))
(add-after 'unpack 'make-files-writable-for-tests
(lambda _ (for-each make-file-writable (find-files "." ".*")))))))
(inputs
(list libssh2 http-parser python-wrapper))
(native-inputs
(list pkg-config))
(propagated-inputs
;; These two libraries are in 'Requires.private' in libgit2.pc.
(list openssl zlib))
(home-page "https://libgit2.github.com/")
(synopsis "Library providing Git core methods")
(description
"Libgit2 is a portable, pure C implementation of the Git core methods
provided as a re-entrant linkable library with a solid API, allowing you to
write native speed custom Git applications in any language with bindings.")
;; GPLv2 with linking exception
(license license:gpl2))))
@end lisp
(In those cases were you only want to tweak a few fields from a package
definition, you should rely on inheritance instead of copy-pasting everything.
See below.)
Let's discuss those fields in depth.
@subsubsection @code{git-fetch} method
Unlike the @code{url-fetch} method, @code{git-fetch} expects a @code{git-reference} which takes
a Git repository and a commit. The commit can be any Git reference such as
tags, so if the @code{version} is tagged, then it can be used directly. Sometimes
the tag is prefixed with a @code{v}, in which case you'd use @code{(commit (string-append
"v" version))}.
To ensure that the source code from the Git repository is stored in a
directory with a descriptive name, we use @code{(file-name (git-file-name name
version))}.
The @code{git-version} procedure can be used to derive the
version when packaging programs for a specific commit, following the
Guix contributor guidelines (@pxref{Version Numbers,,, guix, GNU Guix
Reference Manual}).
How does one obtain the @code{sha256} hash that's in there, you ask? By
invoking @command{guix hash} on a checkout of the desired commit, along
these lines:
@example
git clone https://github.com/libgit2/libgit2/
cd libgit2
git checkout v0.26.6
guix hash -rx .
@end example
@command{guix hash -rx} computes a SHA256 hash over the whole directory,
excluding the @file{.git} sub-directory (@pxref{Invoking guix hash,,,
guix, GNU Guix Reference Manual}).
In the future, @command{guix download} will hopefully be able to do
these steps for you, just like it does for regular downloads.
@subsubsection Snippets
Snippets are quoted (i.e. non-evaluated) Scheme code that are a means of patching
the source. They are a Guix-y alternative to the traditional @file{.patch} files.
Because of the quote, the code in only evaluated when passed to the Guix daemon
for building. There can be as many snippets as needed.
Snippets might need additional Guile modules which can be imported from the
@code{modules} field.
@subsubsection Inputs
There are 3 different input types. In short:
@table @asis
@item native-inputs
Required for building but not runtime -- installing a package
through a substitute won't install these inputs.
@item inputs
Installed in the store but not in the profile, as well as being
present at build time.
@item propagated-inputs
Installed in the store and in the profile, as well as
being present at build time.
@end table
@xref{package Reference,,, guix, GNU Guix Reference Manual} for more details.
The distinction between the various inputs is important: if a dependency can be
handled as an @emph{input} instead of a @emph{propagated input}, it should be done so, or
else it ``pollutes'' the user profile for no good reason.
For instance, a user installing a graphical program that depends on a
command line tool might only be interested in the graphical part, so there is no
need to force the command line tool into the user profile. The dependency is a
concern to the package, not to the user. @emph{Inputs} make it possible to handle
dependencies without bugging the user by adding undesired executable files (or
libraries) to their profile.
Same goes for @emph{native-inputs}: once the program is installed, build-time
dependencies can be safely garbage-collected.
It also matters when a substitute is available, in which case only the @emph{inputs}
and @emph{propagated inputs} will be fetched: the @emph{native inputs} are not required to
install a package from a substitute.
@quotation Note
You may see here and there snippets where package inputs are written
quite differently, like so:
@lisp
;; The "old style" for inputs.
(inputs
`(("libssh2" ,libssh2)
("http-parser" ,http-parser)
("python" ,python-wrapper)))
@end lisp
This is the ``old style'', where each input in the list is explicitly
given a label (a string). It is still supported but we recommend using
the style above instead. @xref{package Reference,,, guix, GNU Guix
Reference Manual}, for more info.
@end quotation
@subsubsection Outputs
Just like how a package can have multiple inputs, it can also produce multiple
outputs.
Each output corresponds to a separate directory in the store.
The user can choose which output to install; this is useful to save space or
to avoid polluting the user profile with unwanted executables or libraries.
Output separation is optional. When the @code{outputs} field is left out, the
default and only output (the complete package) is referred to as @code{"out"}.
Typical separate output names include @code{debug} and @code{doc}.
It's advised to separate outputs only when you've shown it's worth it: if the
output size is significant (compare with @code{guix size}) or in case the package is
modular.
@subsubsection Build system arguments
The @code{arguments} is a keyword-value list used to configure the build process.
The simplest argument @code{#:tests?} can be used to disable the test suite when
building the package. This is mostly useful when the package does not feature
any test suite. It's strongly recommended to keep the test suite on if there is
one.
Another common argument is @code{:make-flags}, which specifies a list of flags to
append when running make, as you would from the command line. For instance, the
following flags
@lisp
#:make-flags (list (string-append "prefix=" (assoc-ref %outputs "out"))
"CC=gcc")
@end lisp
translate into
@example
$ make CC=gcc prefix=/gnu/store/...-<out>
@end example
This sets the C compiler to @code{gcc} and the @code{prefix} variable (the installation
directory in Make parlance) to @code{(assoc-ref %outputs "out")}, which is a build-stage
global variable pointing to the destination directory in the store (something like
@file{/gnu/store/...-my-libgit2-20180408}).
Similarly, it's possible to set the configure flags:
@lisp
#:configure-flags '("-DUSE_SHA1DC=ON")
@end lisp
The @code{%build-inputs} variable is also generated in scope. It's an association
table that maps the input names to their store directories.
The @code{phases} keyword lists the sequential steps of the build system. Typically
phases include @code{unpack}, @code{configure}, @code{build}, @code{install} and @code{check}. To know
more about those phases, you need to work out the appropriate build system
definition in @samp{$GUIX_CHECKOUT/guix/build/gnu-build-system.scm}:
@lisp
(define %standard-phases
;; Standard build phases, as a list of symbol/procedure pairs.
(let-syntax ((phases (syntax-rules ()
((_ p ...) `((p . ,p) ...)))))
(phases set-SOURCE-DATE-EPOCH set-paths install-locale unpack
bootstrap
patch-usr-bin-file
patch-source-shebangs configure patch-generated-file-shebangs
build check install
patch-shebangs strip
validate-runpath
validate-documentation-location
delete-info-dir-file
patch-dot-desktop-files
install-license-files
reset-gzip-timestamps
compress-documentation)))
@end lisp
Or from the REPL:
@lisp
(add-to-load-path "/path/to/guix/checkout")
,use (guix build gnu-build-system)
(map first %standard-phases)
@result{} (set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap patch-usr-bin-file patch-source-shebangs configure patch-generated-file-shebangs build check install patch-shebangs strip validate-runpath validate-documentation-location delete-info-dir-file patch-dot-desktop-files install-license-files reset-gzip-timestamps compress-documentation)
@end lisp
If you want to know more about what happens during those phases, consult the
associated procedures.
For instance, as of this writing the definition of @code{unpack} for the GNU build
system is:
@lisp
(define* (unpack #:key source #:allow-other-keys)
"Unpack SOURCE in the working directory, and change directory within the
source. When SOURCE is a directory, copy it in a sub-directory of the current
working directory."
(if (file-is-directory? source)
(begin
(mkdir "source")
(chdir "source")
;; Preserve timestamps (set to the Epoch) on the copied tree so that
;; things work deterministically.
(copy-recursively source "."
#:keep-mtime? #true))
(begin
(if (string-suffix? ".zip" source)
(invoke "unzip" source)
(invoke "tar" "xvf" source))
(chdir (first-subdirectory "."))))
#true)
@end lisp
Note the @code{chdir} call: it changes the working directory to where the source was
unpacked.
Thus every phase following the @code{unpack} will use the source as a working
directory, which is why we can directly work on the source files.
That is to say, unless a later phase changes the working directory to something
else.
We modify the list of @code{%standard-phases} of the build system with the
@code{modify-phases} macro as per the list of specified modifications, which may have
the following forms:
@itemize
@item
@code{(add-before @var{phase} @var{new-phase} @var{procedure})}: Run @var{procedure} named @var{new-phase} before @var{phase}.
@item
@code{(add-after @var{phase} @var{new-phase} @var{procedure})}: Same, but afterwards.
@item
@code{(replace @var{phase} @var{procedure})}.
@item
@code{(delete @var{phase})}.
@end itemize
The @var{procedure} supports the keyword arguments @code{inputs} and @code{outputs}. Each
input (whether @emph{native}, @emph{propagated} or not) and output directory is referenced
by their name in those variables. Thus @code{(assoc-ref outputs "out")} is the store
directory of the main output of the package. A phase procedure may look like
this:
@lisp
(lambda* (#:key inputs outputs #:allow-other-keys)
(let ((bash-directory (assoc-ref inputs "bash"))
(output-directory (assoc-ref outputs "out"))
(doc-directory (assoc-ref outputs "doc")))
;; ...
#true))
@end lisp
The procedure must return @code{#true} on success. It's brittle to rely on the return
value of the last expression used to tweak the phase because there is no
guarantee it would be a @code{#true}. Hence the trailing @code{#true} to ensure the right value
is returned on success.
@subsubsection Code staging
The astute reader may have noticed the quasi-quote and comma syntax in the
argument field. Indeed, the build code in the package declaration should not be
evaluated on the client side, but only when passed to the Guix daemon. This
mechanism of passing code around two running processes is called @uref{https://arxiv.org/abs/1709.00833, code staging}.
@subsubsection Utility functions
When customizing @code{phases}, we often need to write code that mimics the
equivalent system invocations (@code{make}, @code{mkdir}, @code{cp}, etc.)@: commonly used during
regular ``Unix-style'' installations.
Some like @code{chmod} are native to Guile.
@xref{,,, guile, Guile reference manual} for a complete list.
Guix provides additional helper functions which prove especially handy in the
context of package management.
Some of those functions can be found in
@samp{$GUIX_CHECKOUT/guix/guix/build/utils.scm}. Most of them mirror the behaviour
of the traditional Unix system commands:
@table @code
@item which
Like the @samp{which} system command.
@item find-files
Akin to the @samp{find} system command.
@item mkdir-p
Like @samp{mkdir -p}, which creates all parents as needed.
@item install-file
Similar to @samp{install} when installing a file to a (possibly
non-existing) directory. Guile has @code{copy-file} which works
like @samp{cp}.
@item copy-recursively
Like @samp{cp -r}.
@item delete-file-recursively
Like @samp{rm -rf}.
@item invoke
Run an executable. This should be used instead of @code{system*}.
@item with-directory-excursion
Run the body in a different working directory,
then restore the previous working directory.
@item substitute*
A ``@command{sed}-like'' function.
@end table
@xref{Build Utilities,,, guix, GNU Guix Reference Manual}, for more
information on these utilities.
@subsubsection Module prefix
The license in our last example needs a prefix: this is because of how the
@code{license} module was imported in the package, as @code{#:use-module ((guix licenses)
#:prefix license:)}. The Guile module import mechanism
(@pxref{Using Guile Modules,,, guile, Guile reference manual})
gives the user full control over namespacing: this is needed to avoid
clashes between, say, the
@samp{zlib} variable from @samp{licenses.scm} (a @emph{license} value) and the @samp{zlib} variable
from @samp{compression.scm} (a @emph{package} value).
@node Other build systems
@subsection Other build systems
What we've seen so far covers the majority of packages using a build system
other than the @code{trivial-build-system}. The latter does not automate anything
and leaves you to build everything manually. This can be more demanding and we
won't cover it here for now, but thankfully it is rarely necessary to fall back
on this system.
For the other build systems, such as ASDF, Emacs, Perl, Ruby and many more, the
process is very similar to the GNU build system except for a few specialized
arguments.
@xref{Build Systems,,, guix, GNU Guix Reference Manual}, for more
information on build systems, or check the source code in the
@samp{$GUIX_CHECKOUT/guix/build} and
@samp{$GUIX_CHECKOUT/guix/build-system} directories.
@node Programmable and automated package definition
@subsection Programmable and automated package definition
We can't repeat it enough: having a full-fledged programming language at hand
empowers us in ways that reach far beyond traditional package management.
Let's illustrate this with some awesome features of Guix!
@node Recursive importers
@subsubsection Recursive importers
You might find some build systems good enough that there is little to do at all
to write a package, to the point that it becomes repetitive and tedious after a
while. A @emph{raison d'être} of computers is to replace human beings at those
boring tasks. So let's tell Guix to do this for us and create the package
definition of an R package from CRAN (the output is trimmed for conciseness):
@example
$ guix import cran --recursive walrus
(define-public r-mc2d
; ...
(license gpl2+)))
(define-public r-jmvcore
; ...
(license gpl2+)))
(define-public r-wrs2
; ...
(license gpl3)))
(define-public r-walrus
(package
(name "r-walrus")
(version "1.0.3")
(source
(origin
(method url-fetch)
(uri (cran-uri "walrus" version))
(sha256
(base32
"1nk2glcvy4hyksl5ipq2mz8jy4fss90hx6cq98m3w96kzjni6jjj"))))
(build-system r-build-system)
(propagated-inputs
(list r-ggplot2 r-jmvcore r-r6 r-wrs2))
(home-page "https://github.com/jamovi/walrus")
(synopsis "Robust Statistical Methods")
(description
"This package provides a toolbox of common robust statistical
tests, including robust descriptives, robust t-tests, and robust ANOVA.
It is also available as a module for 'jamovi' (see
<https://www.jamovi.org> for more information). Walrus is based on the
WRS2 package by Patrick Mair, which is in turn based on the scripts and
work of Rand Wilcox. These analyses are described in depth in the book
'Introduction to Robust Estimation & Hypothesis Testing'.")
(license gpl3)))
@end example
The recursive importer won't import packages for which Guix already has package
definitions, except for the very first.
Not all applications can be packaged this way, only those relying on a select
number of supported systems. Read about the full list of importers in
the guix import section of the manual
(@pxref{Invoking guix import,,, guix, GNU Guix Reference Manual}).
@node Automatic update
@subsubsection Automatic update
Guix can be smart enough to check for updates on systems it knows. It can
report outdated package definitions with
@example
$ guix refresh hello
@end example
In most cases, updating a package to a newer version requires little more than
changing the version number and the checksum. Guix can do that automatically as
well:
@example
$ guix refresh hello --update
@end example
@node Inheritance
@subsubsection Inheritance
If you've started browsing the existing package definitions, you might have
noticed that a significant number of them have a @code{inherit} field:
@lisp
(define-public adwaita-icon-theme
(package (inherit gnome-icon-theme)
(name "adwaita-icon-theme")
(version "3.26.1")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnome/sources/" name "/"
(version-major+minor version) "/"
name "-" version ".tar.xz"))
(sha256
(base32
"17fpahgh5dyckgz7rwqvzgnhx53cx9kr2xw0szprc6bnqy977fi8"))))
(native-inputs (list `(,gtk+ "bin")))))
@end lisp
All unspecified fields are inherited from the parent package. This is very
convenient to create alternative packages, for instance with different source,
version or compilation options.
@node Getting help
@subsection Getting help
Sadly, some applications can be tough to package. Sometimes they need a patch to
work with the non-standard file system hierarchy enforced by the store.
Sometimes the tests won't run properly. (They can be skipped but this is not
recommended.) Other times the resulting package won't be reproducible.
Should you be stuck, unable to figure out how to fix any sort of packaging
issue, don't hesitate to ask the community for help.
See the @uref{https://www.gnu.org/software/guix/contact/, Guix homepage} for information on the mailing lists, IRC, etc.
@node Conclusion
@subsection Conclusion
This tutorial was a showcase of the sophisticated package management that Guix
boasts. At this point we have mostly restricted this introduction to the
@code{gnu-build-system} which is a core abstraction layer on which more advanced
abstractions are based.
Where do we go from here? Next we ought to dissect the innards of the build
system by removing all abstractions, using the @code{trivial-build-system}: this
should give us a thorough understanding of the process before investigating some
more advanced packaging techniques and edge cases.
Other features worth exploring are the interactive editing and debugging
capabilities of Guix provided by the Guile REPL@.
Those fancy features are completely optional and can wait; now is a good time
to take a well-deserved break. With what we've introduced here you should be
well armed to package lots of programs. You can get started right away and
hopefully we will see your contributions soon!
@node References
@subsection References
@itemize
@item
The @uref{https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html, package reference in the manual}
@item
@uref{https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org, Pjotr’s hacking guide to GNU Guix}
@item
@uref{https://www.gnu.org/software/guix/guix-ghm-andreas-20130823.pdf, ``GNU Guix: Package without a scheme!''}, by Andreas Enge
@end itemize
@c *********************************************************************
@node System Configuration
@chapter System Configuration
Guix offers a flexible language for declaratively configuring your Guix
System. This flexibility can at times be overwhelming. The purpose of this
chapter is to demonstrate some advanced configuration concepts.
@pxref{System Configuration,,, guix, GNU Guix Reference Manual} for a complete
reference.
@menu
* Auto-Login to a Specific TTY:: Automatically Login a User to a Specific TTY
* Customizing the Kernel:: Creating and using a custom Linux kernel on Guix System.
* Guix System Image API:: Customizing images to target specific platforms.
* Using security keys:: How to use security keys with Guix System.
* Connecting to Wireguard VPN:: Connecting to a Wireguard VPN.
* Customizing a Window Manager:: Handle customization of a Window manager on Guix System.
* Running Guix on a Linode Server:: Running Guix on a Linode Server
* Setting up a bind mount:: Setting up a bind mount in the file-systems definition.
* Getting substitutes from Tor:: Configuring Guix daemon to get substitutes through Tor.
* Setting up NGINX with Lua:: Configuring NGINX web-server to load Lua modules.
* Music Server with Bluetooth Audio:: Headless music player with Bluetooth output.
@end menu
@node Auto-Login to a Specific TTY
@section Auto-Login to a Specific TTY
While the Guix manual explains auto-login one user to @emph{all} TTYs (
@pxref{auto-login to TTY,,, guix, GNU Guix Reference Manual}), some
might prefer a situation, in which one user is logged into one TTY with
the other TTYs either configured to login different users or no one at
all. Note that one can auto-login one user to any TTY, but it is
usually advisable to avoid @code{tty1}, which, by default, is used to
log warnings and errors.
Here is how one might set up auto login for one user to one tty:
@lisp
(define (auto-login-to-tty config tty user)
(if (string=? tty (mingetty-configuration-tty config))
(mingetty-configuration
(inherit config)
(auto-login user))
config))
(define %my-services
(modify-services %base-services
;; @dots{}
(mingetty-service-type config =>
(auto-login-to-tty
config "tty3" "alice"))))
(operating-system
;; @dots{}
(services %my-services))
@end lisp
One could also @code{compose} (@pxref{Higher-Order Functions,,, guile,
The Guile Reference Manual}) @code{auto-login-to-tty} to login multiple
users to multiple ttys.
Finally, here is a note of caution. Setting up auto login to a TTY,
means that anyone can turn on your computer and run commands as your
regular user.
However, if you have an encrypted root partition, and thus already need
to enter a passphrase when the system boots, auto-login might be a
convenient option.
@node Customizing the Kernel
@section Customizing the Kernel
Guix is, at its core, a source based distribution with substitutes
(@pxref{Substitutes,,, guix, GNU Guix Reference Manual}), and as such building
packages from their source code is an expected part of regular package
installations and upgrades. Given this starting point, it makes sense that
efforts are made to reduce the amount of time spent compiling packages, and
recent changes and upgrades to the building and distribution of substitutes
continues to be a topic of discussion within Guix.
The kernel, while not requiring an overabundance of RAM to build, does take a
rather long time on an average machine. The official kernel configuration, as
is the case with many GNU/Linux distributions, errs on the side of
inclusiveness, and this is really what causes the build to take such a long
time when the kernel is built from source.
The Linux kernel, however, can also just be described as a regular old
package, and as such can be customized just like any other package. The
procedure is a little bit different, although this is primarily due to the
nature of how the package definition is written.
The @code{linux-libre} kernel package definition is actually a procedure which
creates a package.
@lisp
(define* (make-linux-libre* version gnu-revision source supported-systems
#:key
(extra-version #f)
;; A function that takes an arch and a variant.
;; See kernel-config for an example.
(configuration-file #f)
(defconfig "defconfig")
(extra-options %default-extra-linux-options))
...)
@end lisp
The current @code{linux-libre} package is for the 5.15.x series, and is
declared like this:
@lisp
(define-public linux-libre-5.15
(make-linux-libre* linux-libre-5.15-version
linux-libre-5.15-gnu-revision
linux-libre-5.15-source
'("x86_64-linux" "i686-linux" "armhf-linux" "aarch64-linux" "riscv64-linux")
#:configuration-file kernel-config))
@end lisp
Any keys which are not assigned values inherit their default value from the
@code{make-linux-libre} definition. When comparing the two snippets above,
notice the code comment that refers to @code{#:configuration-file}. Because of
this, it is not actually easy to include a custom kernel configuration from the
definition, but don't worry, there are other ways to work with what we do have.
There are two ways to create a kernel with a custom kernel configuration. The
first is to provide a standard @file{.config} file during the build process by
including an actual @file{.config} file as a native input to our custom
kernel. The following is a snippet from the custom @code{'configure} phase of
the @code{make-linux-libre} package definition:
@lisp
(let ((build (assoc-ref %standard-phases 'build))
(config (assoc-ref (or native-inputs inputs) "kconfig")))
;; Use a custom kernel configuration file or a default
;; configuration file.
(if config
(begin
(copy-file config ".config")
(chmod ".config" #o666))
(invoke "make" ,defconfig)))
@end lisp
Below is a sample kernel package. The @code{linux-libre} package is nothing
special and can be inherited from and have its fields overridden like any
other package:
@lisp
(define-public linux-libre/E2140
(package
(inherit linux-libre)
(native-inputs
`(("kconfig" ,(local-file "E2140.config"))
,@@(alist-delete "kconfig"
(package-native-inputs linux-libre))))))
@end lisp
In the same directory as the file defining @code{linux-libre-E2140} is a file
named @file{E2140.config}, which is an actual kernel configuration file. The
@code{defconfig} keyword of @code{make-linux-libre} is left blank here, so the
only kernel configuration in the package is the one which was included in the
@code{native-inputs} field.
The second way to create a custom kernel is to pass a new value to the
@code{extra-options} keyword of the @code{make-linux-libre} procedure. The
@code{extra-options} keyword works with another function defined right below
it:
@lisp
(define %default-extra-linux-options
`(;; https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00039.html
("CONFIG_DEVPTS_MULTIPLE_INSTANCES" . #true)
;; Modules required for initrd:
("CONFIG_NET_9P" . m)
("CONFIG_NET_9P_VIRTIO" . m)
("CONFIG_VIRTIO_BLK" . m)
("CONFIG_VIRTIO_NET" . m)
("CONFIG_VIRTIO_PCI" . m)
("CONFIG_VIRTIO_BALLOON" . m)
("CONFIG_VIRTIO_MMIO" . m)
("CONFIG_FUSE_FS" . m)
("CONFIG_CIFS" . m)
("CONFIG_9P_FS" . m)))
(define (config->string options)
(string-join (map (match-lambda
((option . 'm)
(string-append option "=m"))
((option . #true)
(string-append option "=y"))
((option . #false)
(string-append option "=n")))
options)
"\n"))
@end lisp
And in the custom configure script from the `make-linux-libre` package:
@lisp
;; Appending works even when the option wasn't in the
;; file. The last one prevails if duplicated.
(let ((port (open-file ".config" "a"))
(extra-configuration ,(config->string extra-options)))
(display extra-configuration port)
(close-port port))
(invoke "make" "oldconfig")
@end lisp
So by not providing a configuration-file the @file{.config} starts blank, and
then we write into it the collection of flags that we want. Here's another
custom kernel:
@lisp
(define %macbook41-full-config
(append %macbook41-config-options
%file-systems
%efi-support
%emulation
(@@@@ (gnu packages linux) %default-extra-linux-options)))
(define-public linux-libre-macbook41
;; XXX: Access the internal 'make-linux-libre*' procedure, which is
;; private and unexported, and is liable to change in the future.
((@@@@ (gnu packages linux) make-linux-libre*)
(@@@@ (gnu packages linux) linux-libre-version)
(@@@@ (gnu packages linux) linux-libre-gnu-revision)
(@@@@ (gnu packages linux) linux-libre-source)
'("x86_64-linux")
#:extra-version "macbook41"
#:extra-options %macbook41-config-options))
@end lisp
In the above example @code{%file-systems} is a collection of flags enabling
different file system support, @code{%efi-support} enables EFI support and
@code{%emulation} enables a x86_64-linux machine to act in 32-bit mode also.
@code{%default-extra-linux-options} are the ones quoted above, which had to be
added in since they were replaced in the @code{extra-options} keyword.
This all sounds like it should be doable, but how does one even know which
modules are required for a particular system? Two places that can be helpful
in trying to answer this question is the
@uref{https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Kernel, Gentoo
Handbook} and the
@uref{https://www.kernel.org/doc/html/latest/admin-guide/README.html?highlight=localmodconfig,
documentation from the kernel itself}. From the kernel documentation, it
seems that @code{make localmodconfig} is the command we want.
In order to actually run @code{make localmodconfig} we first need to get and
unpack the kernel source code:
@example shell
tar xf $(guix build linux-libre --source)
@end example
Once inside the directory containing the source code run @code{touch .config}
to create an initial, empty @file{.config} to start with. @code{make
localmodconfig} works by seeing what you already have in @file{.config} and
letting you know what you're missing. If the file is blank then you're
missing everything. The next step is to run:
@example shell
guix shell -D linux-libre -- make localmodconfig
@end example
and note the output. Do note that the @file{.config} file is still empty.
The output generally contains two types of warnings. The first start with
"WARNING" and can actually be ignored in our case. The second read:
@example shell
module pcspkr did not have configs CONFIG_INPUT_PCSPKR
@end example
For each of these lines, copy the @code{CONFIG_XXXX_XXXX} portion into the
@file{.config} in the directory, and append @code{=m}, so in the end it looks
like this:
@example shell
CONFIG_INPUT_PCSPKR=m
CONFIG_VIRTIO=m
@end example
After copying all the configuration options, run @code{make localmodconfig}
again to make sure that you don't have any output starting with ``module''.
After all of these machine specific modules there are a couple more left that
are also needed. @code{CONFIG_MODULES} is necessary so that you can build and
load modules separately and not have everything built into the kernel.
@code{CONFIG_BLK_DEV_SD} is required for reading from hard drives. It is
possible that there are other modules which you will need.
This post does not aim to be a guide to configuring your own kernel however,
so if you do decide to build a custom kernel you'll have to seek out other
guides to create a kernel which is just right for your needs.
The second way to setup the kernel configuration makes more use of Guix's
features and allows you to share configuration segments between different
kernels. For example, all machines using EFI to boot have a number of EFI
configuration flags that they need. It is likely that all the kernels will
share a list of file systems to support. By using variables it is easier to
see at a glance what features are enabled and to make sure you don't have
features in one kernel but missing in another.
Left undiscussed however, is Guix's initrd and its customization. It is
likely that you'll need to modify the initrd on a machine using a custom
kernel, since certain modules which are expected to be built may not be
available for inclusion into the initrd.
@node Guix System Image API
@section Guix System Image API
Historically, Guix System is centered around an @code{operating-system}
structure. This structure contains various fields ranging from the
bootloader and kernel declaration to the services to install.
Depending on the target machine, that can go from a standard
@code{x86_64} machine to a small ARM single board computer such as the
Pine64, the image constraints can vary a lot. The hardware
manufacturers will impose different image formats with various partition
sizes and offsets.
To create images suitable for all those machines, a new abstraction is
necessary: that's the goal of the @code{image} record. This record
contains all the required information to be transformed into a
standalone image, that can be directly booted on any target machine.
@lisp
(define-record-type* <image>
image make-image
image?
(name image-name ;symbol
(default #f))
(format image-format) ;symbol
(target image-target
(default #f))
(size image-size ;size in bytes as integer
(default 'guess))
(operating-system image-operating-system ;<operating-system>
(default #f))
(partitions image-partitions ;list of <partition>
(default '()))
(compression? image-compression? ;boolean
(default #t))
(volatile-root? image-volatile-root? ;boolean
(default #t))
(substitutable? image-substitutable? ;boolean
(default #t)))
@end lisp
This record contains the operating-system to instantiate. The
@code{format} field defines the image type and can be @code{efi-raw},
@code{qcow2} or @code{iso9660} for instance. In the future, it could be
extended to @code{docker} or other image types.
A new directory in the Guix sources is dedicated to images definition. For now
there are four files:
@itemize @bullet
@item @file{gnu/system/images/hurd.scm}
@item @file{gnu/system/images/pine64.scm}
@item @file{gnu/system/images/novena.scm}
@item @file{gnu/system/images/pinebook-pro.scm}
@end itemize
Let's have a look to @file{pine64.scm}. It contains the
@code{pine64-barebones-os} variable which is a minimal definition of an
operating-system dedicated to the @b{Pine A64 LTS} board.
@lisp
(define pine64-barebones-os
(operating-system
(host-name "vignemale")
(timezone "Europe/Paris")
(locale "en_US.utf8")
(bootloader (bootloader-configuration
(bootloader u-boot-pine64-lts-bootloader)
(targets '("/dev/vda"))))
(initrd-modules '())
(kernel linux-libre-arm64-generic)
(file-systems (cons (file-system
(device (file-system-label "my-root"))
(mount-point "/")
(type "ext4"))
%base-file-systems))
(services (cons (service agetty-service-type
(agetty-configuration
(extra-options '("-L")) ; no carrier detect
(baud-rate "115200")
(term "vt100")
(tty "ttyS0")))
%base-services))))
@end lisp
The @code{kernel} and @code{bootloader} fields are pointing to packages
dedicated to this board.
Right below, the @code{pine64-image-type} variable is also defined.
@lisp
(define pine64-image-type
(image-type
(name 'pine64-raw)
(constructor (cut image-with-os arm64-disk-image <>))))
@end lisp
It's using a record we haven't talked about yet, the @code{image-type} record,
defined this way:
@lisp
(define-record-type* <image-type>
image-type make-image-type
image-type?
(name image-type-name) ;symbol
(constructor image-type-constructor)) ;<operating-system> -> <image>
@end lisp
The main purpose of this record is to associate a name to a procedure
transforming an @code{operating-system} to an image. To understand why
it is necessary, let's have a look to the command producing an image
from an @code{operating-system} configuration file:
@example
guix system image my-os.scm
@end example
This command expects an @code{operating-system} configuration but how
should we indicate that we want an image targeting a Pine64 board? We
need to provide an extra information, the @code{image-type}, by passing
the @code{--image-type} or @code{-t} flag, this way:
@example
guix system image --image-type=pine64-raw my-os.scm
@end example
This @code{image-type} parameter points to the @code{pine64-image-type}
defined above. Hence, the @code{operating-system} declared in
@code{my-os.scm} will be applied the @code{(cut image-with-os
arm64-disk-image <>)} procedure to turn it into an image.
The resulting image looks like:
@lisp
(image
(format 'disk-image)
(target "aarch64-linux-gnu")
(operating-system my-os)
(partitions
(list (partition
(inherit root-partition)
(offset root-offset)))))
@end lisp
which is the aggregation of the @code{operating-system} defined in
@code{my-os.scm} to the @code{arm64-disk-image} record.
But enough Scheme madness. What does this image API bring to the Guix user?
One can run:
@example
mathieu@@cervin:~$ guix system --list-image-types
The available image types are:
- pinebook-pro-raw
- pine64-raw
- novena-raw
- hurd-raw
- hurd-qcow2
- qcow2
- uncompressed-iso9660
- efi-raw
- arm64-raw
- arm32-raw
- iso9660
@end example
and by writing an @code{operating-system} file based on
@code{pine64-barebones-os}, you can customize your image to your
preferences in a file (@file{my-pine-os.scm}) like this:
@lisp
(use-modules (gnu services linux)
(gnu system images pine64))
(let ((base-os pine64-barebones-os))
(operating-system
(inherit base-os)
(timezone "America/Indiana/Indianapolis")
(services
(cons
(service earlyoom-service-type
(earlyoom-configuration
(prefer-regexp "icecat|chromium")))
(operating-system-user-services base-os)))))
@end lisp
run:
@example
guix system image --image-type=pine64-raw my-pine-os.scm
@end example
or,
@example
guix system image --image-type=hurd-raw my-hurd-os.scm
@end example
to get an image that can be written directly to a hard drive and booted
from.
Without changing anything to @code{my-hurd-os.scm}, calling:
@example
guix system image --image-type=hurd-qcow2 my-hurd-os.scm
@end example
will instead produce a Hurd QEMU image.
@node Using security keys
@section Using security keys
@cindex 2FA, two-factor authentication
@cindex U2F, Universal 2nd Factor
@cindex security key, configuration
The use of security keys can improve your security by providing a second
authentication source that cannot be easily stolen or copied, at least
for a remote adversary (something that you have), to the main secret (a
passphrase -- something that you know), reducing the risk of
impersonation.
The example configuration detailed below showcases what minimal
configuration needs to be made on your Guix System to allow the use of a
Yubico security key. It is hoped the configuration can be useful for
other security keys as well, with minor adjustments.
@subsection Configuration for use as a two-factor authenticator (2FA)
To be usable, the udev rules of the system should be extended with
key-specific rules. The following shows how to extend your udev rules
with the @file{lib/udev/rules.d/70-u2f.rules} udev rule file provided by
the @code{libfido2} package from the @code{(gnu packages
security-token)} module and add your user to the @samp{"plugdev"} group
it uses:
@lisp
(use-package-modules ... security-token ...)
...
(operating-system
...
(users (cons* (user-account
(name "your-user")
(group "users")
(supplementary-groups
'("wheel" "netdev" "audio" "video"
"plugdev")) ;<- added system group
(home-directory "/home/your-user"))
%base-user-accounts))
...
(services
(cons*
...
(udev-rules-service 'fido2 libfido2 #:groups '("plugdev")))))
@end lisp
After re-configuring your system and re-logging in your graphical
session so that the new group is in effect for your user, you can verify
that your key is usable by launching:
@example
guix shell ungoogled-chromium -- chromium chrome://settings/securityKeys
@end example
and validating that the security key can be reset via the ``Reset your
security key'' menu. If it works, congratulations, your security key is
ready to be used with applications supporting two-factor authentication
(2FA).
@node Connecting to Wireguard VPN
@section Connecting to Wireguard VPN
To connect to a Wireguard VPN server you need the kernel module to be
loaded in memory and a package providing networking tools that support
it (e.g. @code{wireguard-tools} or @code{network-manager}).
Here is a configuration example for Linux-Libre < 5.6, where the module
is out of tree and need to be loaded manually---following revisions of
the kernel have it built-in and so don't need such configuration:
@lisp
(use-modules (gnu))
(use-service-modules desktop)
(use-package-modules vpn)
(operating-system
;; …
(services (cons (simple-service 'wireguard-module
kernel-module-loader-service-type
'("wireguard"))
%desktop-services))
(packages (cons wireguard-tools %base-packages))
(kernel-loadable-modules (list wireguard-linux-compat)))
@end lisp
After reconfiguring and restarting your system you can either use
Wireguard tools or NetworkManager to connect to a VPN server.
@subsection Using Wireguard tools
To test your Wireguard setup it is convenient to use @command{wg-quick}.
Just give it a configuration file @command{wg-quick up ./wg0.conf}; or
put that file in @file{/etc/wireguard} and run @command{wg-quick up wg0}
instead.
@quotation Note
Be warned that the author described this command as a: “[…] very quick
and dirty bash script […]”.
@end quotation
@subsection Using NetworkManager
Thanks to NetworkManager support for Wireguard we can connect to our VPN
using @command{nmcli} command. Up to this point this guide assumes that
you're using Network Manager service provided by
@code{%desktop-services}. Ortherwise you need to adjust your services
list to load @code{network-manager-service-type} and reconfigure your
Guix system.
To import your VPN configuration execute nmcli import command:
@example shell
# nmcli connection import type wireguard file wg0.conf
Connection 'wg0' (edbee261-aa5a-42db-b032-6c7757c60fde) successfully added
@end example
This will create a configuration file in
@file{/etc/NetworkManager/wg0.nmconnection}. Next connect to the
Wireguard server:
@example shell
$ nmcli connection up wg0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/6)
@end example
By default NetworkManager will connect automatically on system boot. To
change that behaviour you need to edit your config:
@example shell
# nmcli connection modify wg0 connection.autoconnect no
@end example
For more specific information about NetworkManager and wireguard
@uref{https://blogs.gnome.org/thaller/2019/03/15/wireguard-in-networkmanager/,see
this post by thaller}.
@node Customizing a Window Manager
@section Customizing a Window Manager
@cindex wm
@node StumpWM
@subsection StumpWM
@cindex stumpwm
You could install StumpWM with a Guix system by adding
@code{stumpwm} and optionally @code{`(,stumpwm "lib")}
packages to a system configuration file, e.g.@: @file{/etc/config.scm}.
An example configuration can look like this:
@lisp
(use-modules (gnu))
(use-package-modules wm)
(operating-system
;; …
(packages (append (list sbcl stumpwm `(,stumpwm "lib"))
%base-packages)))
@end lisp
@cindex stumpwm fonts
By default StumpWM uses X11 fonts, which could be small or pixelated on
your system. You could fix this by installing StumpWM contrib Lisp
module @code{sbcl-ttf-fonts}, adding it to Guix system packages:
@lisp
(use-modules (gnu))
(use-package-modules fonts wm)
(operating-system
;; …
(packages (append (list sbcl stumpwm `(,stumpwm "lib"))
sbcl-ttf-fonts font-dejavu %base-packages)))
@end lisp
Then you need to add the following code to a StumpWM configuration file
@file{~/.stumpwm.d/init.lisp}:
@lisp
(require :ttf-fonts)
(setf xft:*font-dirs* '("/run/current-system/profile/share/fonts/"))
(setf clx-truetype:+font-cache-filename+ (concat (getenv "HOME") "/.fonts/font-cache.sexp"))
(xft:cache-fonts)
(set-font (make-instance 'xft:font :family "DejaVu Sans Mono" :subfamily "Book" :size 11))
@end lisp
@node Session lock
@subsection Session lock
@cindex sessionlock
Depending on your environment, locking the screen of your session might come built in
or it might be something you have to set up yourself. If you use a desktop environment
like GNOME or KDE, it's usually built in. If you use a plain window manager like
StumpWM or EXWM, you might have to set it up yourself.
@node Xorg
@subsubsection Xorg
If you use Xorg, you can use the utility
@uref{https://www.mankier.com/1/xss-lock, xss-lock} to lock the screen of your session.
xss-lock is triggered by DPMS which since Xorg 1.8 is auto-detected and enabled if
ACPI is also enabled at kernel runtime.
To use xss-lock, you can simple execute it and put it into the background before
you start your window manager from e.g. your @file{~/.xsession}:
@example
xss-lock -- slock &
exec stumpwm
@end example
In this example, xss-lock uses @code{slock} to do the actual locking of the screen when
it determines it's appropriate, like when you suspend your device.
For slock to be allowed to be a screen locker for the graphical session, it needs to
be made setuid-root so it can authenticate users, and it needs a PAM service. This
can be achieved by adding the following service to your @file{config.scm}:
@lisp
(screen-locker-service slock)
@end lisp
If you manually lock your screen, e.g. by directly calling slock when you want to lock
your screen but not suspend it, it's a good idea to notify xss-lock about this so no
confusion occurs. This can be done by executing @code{xset s activate} immediately
before you execute slock.
@node Running Guix on a Linode Server
@section Running Guix on a Linode Server
@cindex linode, Linode
To run Guix on a server hosted by @uref{https://www.linode.com, Linode},
start with a recommended Debian server. We recommend using the default
distro as a way to bootstrap Guix. Create your SSH keys.
@example
ssh-keygen
@end example
Be sure to add your SSH key for easy login to the remote server.
This is trivially done via Linode's graphical interface for adding
SSH keys. Go to your profile and click add SSH Key.
Copy into it the output of:
@example
cat ~/.ssh/<username>_rsa.pub
@end example
Power the Linode down.
In the Linode's Storage tab, resize the Debian disk to be smaller.
30 GB free space is recommended. Then click "Add a disk", and fill
out the form with the following:
@itemize @bullet
@item
Label: "Guix"
@item
Filesystem: ext4
@item
Set it to the remaining size
@end itemize
In the Configurations tab, press "Edit" on the default Debian profile.
Under "Block Device Assignment" click "Add a Device". It should be
@file{/dev/sdc} and you can select the "Guix" disk. Save Changes.
Now "Add a Configuration", with the following:
@itemize @bullet
@item
Label: Guix
@item
Kernel:GRUB 2 (it's at the bottom! This step is @b{IMPORTANT!})
@item
Block device assignment:
@item
@file{/dev/sda}: Guix
@item
@file{/dev/sdb}: swap
@item
Root device: @file{/dev/sda}
@item
Turn off all the filesystem/boot helpers
@end itemize
Now power it back up, booting with the Debian configuration. Once it's
running, ssh to your server via @code{ssh
root@@@var{<your-server-IP-here>}}. (You can find your server IP address in
your Linode Summary section.) Now you can run the "install guix from
@pxref{Binary Installation,,, guix, GNU Guix}" steps:
@example
sudo apt-get install gpg
wget https://sv.gnu.org/people/viewgpg.php?user_id=15145 -qO - | gpg --import -
wget https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
chmod +x guix-install.sh
./guix-install.sh
guix pull
@end example
Now it's time to write out a config for the server. The key information
is below. Save the resulting file as @file{guix-config.scm}.
@lisp
(use-modules (gnu)
(guix modules))
(use-service-modules networking
ssh)
(use-package-modules admin
certs
package-management
ssh
tls)
(operating-system
(host-name "my-server")
(timezone "America/New_York")
(locale "en_US.UTF-8")
;; This goofy code will generate the grub.cfg
;; without installing the grub bootloader on disk.
(bootloader (bootloader-configuration
(bootloader
(bootloader
(inherit grub-bootloader)
(installer #~(const #true))))))
(file-systems (cons (file-system
(device "/dev/sda")
(mount-point "/")
(type "ext4"))
%base-file-systems))
(swap-devices (list "/dev/sdb"))
(initrd-modules (cons "virtio_scsi" ; Needed to find the disk
%base-initrd-modules))
(users (cons (user-account
(name "janedoe")
(group "users")
;; Adding the account to the "wheel" group
;; makes it a sudoer.
(supplementary-groups '("wheel"))
(home-directory "/home/janedoe"))
%base-user-accounts))
(packages (cons* nss-certs ;for HTTPS access
openssh-sans-x
%base-packages))
(services (cons*
(service dhcp-client-service-type)
(service openssh-service-type
(openssh-configuration
(openssh openssh-sans-x)
(password-authentication? #false)
(authorized-keys
`(("janedoe" ,(local-file "janedoe_rsa.pub"))
("root" ,(local-file "janedoe_rsa.pub"))))))
%base-services)))
@end lisp
Replace the following fields in the above configuration:
@lisp
(host-name "my-server") ; replace with your server name
; if you chose a linode server outside the U.S., then
; use tzselect to find a correct timezone string
(timezone "America/New_York") ; if needed replace timezone
(name "janedoe") ; replace with your username
("janedoe" ,(local-file "janedoe_rsa.pub")) ; replace with your ssh key
("root" ,(local-file "janedoe_rsa.pub")) ; replace with your ssh key
@end lisp
The last line in the above example lets you log into the server as root
and set the initial root password (see the note at the end of this
recipe about root login). After you have done this, you may
delete that line from your configuration and reconfigure to prevent root
login.
Copy your ssh public key (eg: @file{~/.ssh/id_rsa.pub}) as
@file{@var{<your-username-here>}_rsa.pub} and put
@file{guix-config.scm} in the same directory. In a new terminal run
these commands.
@example
sftp root@@<remote server ip address>
put /path/to/files/<username>_rsa.pub .
put /path/to/files/guix-config.scm .
@end example
In your first terminal, mount the guix drive:
@example
mkdir /mnt/guix
mount /dev/sdc /mnt/guix
@end example
Due to the way we set up the bootloader section of the guix-config.scm,
only the grub configuration file will be installed. So, we need to copy
over some of the other GRUB stuff already installed on the Debian system:
@example
mkdir -p /mnt/guix/boot/grub
cp -r /boot/grub/* /mnt/guix/boot/grub/
@end example
Now initialize the Guix installation:
@example
guix system init guix-config.scm /mnt/guix
@end example
Ok, power it down!
Now from the Linode console, select boot and select "Guix".
Once it boots, you should be able to log in via SSH! (The server config
will have changed though.) You may encounter an error like:
@example
$ ssh root@@<server ip address>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:0B+wp33w57AnKQuHCvQP0+ZdKaqYrI/kyU7CfVbS7R4.
Please contact your system administrator.
Add correct host key in /home/joshua/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/joshua/.ssh/known_hosts:3
ECDSA host key for 198.58.98.76 has changed and you have requested strict checking.
Host key verification failed.
@end example
Either delete @file{~/.ssh/known_hosts} file, or delete the offending line
starting with your server IP address.
Be sure to set your password and root's password.
@example
ssh root@@<remote ip address>
passwd ; for the root password
passwd <username> ; for the user password
@end example
You may not be able to run the above commands at this point. If you
have issues remotely logging into your linode box via SSH, then you may
still need to set your root and user password initially by clicking on
the ``Launch Console'' option in your linode. Choose the ``Glish''
instead of ``Weblish''. Now you should be able to ssh into the machine.
Hooray! At this point you can shut down the server, delete the
Debian disk, and resize the Guix to the rest of the size.
Congratulations!
By the way, if you save it as a disk image right at this point, you'll
have an easy time spinning up new Guix images! You may need to
down-size the Guix image to 6144MB, to save it as an image. Then you
can resize it again to the max size.
@node Setting up a bind mount
@section Setting up a bind mount
To bind mount a file system, one must first set up some definitions
before the @code{operating-system} section of the system definition. In
this example we will bind mount a folder from a spinning disk drive to
@file{/tmp}, to save wear and tear on the primary SSD, without
dedicating an entire partition to be mounted as @file{/tmp}.
First, the source drive that hosts the folder we wish to bind mount
should be defined, so that the bind mount can depend on it.
@lisp
(define source-drive ;; "source-drive" can be named anything you want.
(file-system
(device (uuid "UUID goes here"))
(mount-point "/path-to-spinning-disk-goes-here")
(type "ext4"))) ;; Make sure to set this to the appropriate type for your drive.
@end lisp
The source folder must also be defined, so that guix will know it's not
a regular block device, but a folder.
@lisp
(define (%source-directory) "/path-to-spinning-disk-goes-here/tmp") ;; "source-directory" can be named any valid variable name.
@end lisp
Finally, inside the @code{file-systems} definition, we must add the
mount itself.
@lisp
(file-systems (cons*
...<other drives omitted for clarity>...
source-drive ;; Must match the name you gave the source drive in the earlier definition.
(file-system
(device (%source-directory)) ;; Make sure "source-directory" matches your earlier definition.
(mount-point "/tmp")
(type "none") ;; We are mounting a folder, not a partition, so this type needs to be "none"
(flags '(bind-mount))
(dependencies (list source-drive)) ;; Ensure "source-drive" matches what you've named the variable for the drive.
)
...<other drives omitted for clarity>...
))
@end lisp
@node Getting substitutes from Tor
@section Getting substitutes from Tor
Guix daemon can use a HTTP proxy to get substitutes, here we are
configuring it to get them via Tor.
@quotation Warning
@emph{Not all} Guix daemon's traffic will go through Tor! Only
HTTP/HTTPS will get proxied; FTP, Git protocol, SSH, etc connections
will still go through the clearnet. Again, this configuration isn't
foolproof some of your traffic won't get routed by Tor at all. Use it
at your own risk.
Also note that the procedure described here applies only to package
substitution. When you update your guix distribution with
@command{guix pull}, you still need to use @command{torsocks} if
you want to route the connection to guix's git repository servers
through Tor.
@end quotation
Guix's substitute server is available as a Onion service, if you want
to use it to get your substitutes through Tor configure your system as
follow:
@lisp
(use-modules (gnu))
(use-service-module base networking)
(operating-system
…
(services
(cons
(service tor-service-type
(tor-configuration
(config-file (plain-file "tor-config"
"HTTPTunnelPort 127.0.0.1:9250"))))
(modify-services %base-services
(guix-service-type
config => (guix-configuration
(inherit config)
;; ci.guix.gnu.org's Onion service
(substitute-urls
"@value{SUBSTITUTE-TOR-URL}")
(http-proxy "http://localhost:9250")))))))
@end lisp
This will keep a tor process running that provides a HTTP CONNECT tunnel
which will be used by @command{guix-daemon}. The daemon can use other
protocols than HTTP(S) to get remote resources, request using those
protocols won't go through Tor since we are only setting a HTTP tunnel
here. Note that @code{substitutes-urls} is using HTTPS and not HTTP or
it won't work, that's a limitation of Tor's tunnel; you may want to use
@command{privoxy} instead to avoid such limitations.
If you don't want to always get substitutes through Tor but using it just
some of the times, then skip the @code{guix-configuration}. When you
want to get a substitute from the Tor tunnel run:
@example
sudo herd set-http-proxy guix-daemon http://localhost:9250
guix build \
--substitute-urls=@value{SUBSTITUTE-TOR-URL} @dots{}
@end example
@node Setting up NGINX with Lua
@section Setting up NGINX with Lua
@cindex nginx, lua, openresty, resty
NGINX could be extended with Lua scripts.
Guix provides NGINX service with ability to load Lua module and specific
Lua packages, and reply to requests by evaluating Lua scripts.
The following example demonstrates system definition with configuration
to evaluate @file{index.lua} Lua script on HTTP request to
@uref{http://localhost/hello} endpoint:
@example
local shell = require "resty.shell"
local stdin = ""
local timeout = 1000 -- ms
local max_size = 4096 -- byte
local ok, stdout, stderr, reason, status =
shell.run([[/run/current-system/profile/bin/ls /tmp]], stdin, timeout, max_size)
ngx.say(stdout)
@end example
@lisp
(use-modules (gnu))
(use-service-modules #;… web)
(use-package-modules #;… lua)
(operating-system
;; …
(services
;; …
(service nginx-service-type
(nginx-configuration
(modules
(list
(file-append nginx-lua-module "/etc/nginx/modules/ngx_http_lua_module.so")))
(lua-package-path (list lua-resty-core
lua-resty-lrucache
lua-resty-signal
lua-tablepool
lua-resty-shell))
(lua-package-cpath (list lua-resty-signal))
(server-blocks
(list (nginx-server-configuration
(server-name '("localhost"))
(listen '("80"))
(root "/etc")
(locations (list
(nginx-location-configuration
(uri "/hello")
(body (list #~(format #f "content_by_lua_file ~s;"
#$(local-file "index.lua"))))))))))))))
@end lisp
@node Music Server with Bluetooth Audio
@section Music Server with Bluetooth Audio
@cindex mpd
@cindex music server, headless
@cindex bluetooth, ALSA configuration
MPD, the Music Player Daemon, is a flexible server-side application for
playing music. Client programs on different machines on the network ---
a mobile phone, a laptop, a desktop workstation --- can connect to it to
control the playback of audio files from your local music collection.
MPD decodes the audio files and plays them back on one or many outputs.
By default MPD will play to the default audio device. In the example
below we make things a little more interesting by setting up a headless
music server. There will be no graphical user interface, no Pulseaudio
daemon, and no local audio output. Instead we will configure MPD with
two outputs: a bluetooth speaker and a web server to serve audio streams
to any streaming media player.
Bluetooth is often rather frustrating to set up. You will have to pair
your Bluetooth device and make sure that the device is automatically
connected as soon as it powers on. The Bluetooth system service
returned by the @code{bluetooth-service} procedure provides the
infrastructure needed to set this up.
Reconfigure your system with at least the following services and
packages:
@lisp
(operating-system
;; …
(packages (cons* bluez bluez-alsa
%base-packages))
(services
;; …
(dbus-service #:services (list bluez-alsa))
(bluetooth-service #:auto-enable? #t)))
@end lisp
Start the @code{bluetooth} service and then use @command{bluetoothctl}
to scan for Bluetooth devices. Try to identify your Bluetooth speaker
and pick out its device ID from the resulting list of devices that is
indubitably dominated by a baffling smorgasbord of your neighbors' home
automation gizmos. This only needs to be done once:
@example
$ bluetoothctl
[NEW] Controller 00:11:22:33:95:7F BlueZ 5.40 [default]
[bluetooth]# power on
[bluetooth]# Changing power on succeeded
[bluetooth]# agent on
[bluetooth]# Agent registered
[bluetooth]# default-agent
[bluetooth]# Default agent request successful
[bluetooth]# scan on
[bluetooth]# Discovery started
[CHG] Controller 00:11:22:33:95:7F Discovering: yes
[NEW] Device AA:BB:CC:A4:AA:CD My Bluetooth Speaker
[NEW] Device 44:44:FF:2A:20:DC My Neighbor's TV
@dots{}
[bluetooth]# pair AA:BB:CC:A4:AA:CD
Attempting to pair with AA:BB:CC:A4:AA:CD
[CHG] Device AA:BB:CC:A4:AA:CD Connected: yes
[My Bluetooth Speaker]# [CHG] Device AA:BB:CC:A4:AA:CD UUIDs: 0000110b-0000-1000-8000-00xxxxxxxxxx
[CHG] Device AA:BB:CC:A4:AA:CD UUIDs: 0000110c-0000-1000-8000-00xxxxxxxxxx
[CHG] Device AA:BB:CC:A4:AA:CD UUIDs: 0000110e-0000-1000-8000-00xxxxxxxxxx
[CHG] Device AA:BB:CC:A4:AA:CD Paired: yes
Pairing successful
[CHG] Device AA:BB:CC:A4:AA:CD Connected: no
[bluetooth]#
[bluetooth]# trust AA:BB:CC:A4:AA:CD
[bluetooth]# [CHG] Device AA:BB:CC:A4:AA:CD Trusted: yes
Changing AA:BB:CC:A4:AA:CD trust succeeded
[bluetooth]#
[bluetooth]# connect AA:BB:CC:A4:AA:CD
Attempting to connect to AA:BB:CC:A4:AA:CD
[bluetooth]# [CHG] Device AA:BB:CC:A4:AA:CD RSSI: -63
[CHG] Device AA:BB:CC:A4:AA:CD Connected: yes
Connection successful
[My Bluetooth Speaker]# scan off
[CHG] Device AA:BB:CC:A4:AA:CD RSSI is nil
Discovery stopped
[CHG] Controller 00:11:22:33:95:7F Discovering: no
@end example
Congratulations, you can now automatically connect to your Bluetooth
speaker!
It is now time to configure ALSA to use the @emph{bluealsa} Bluetooth
module, so that you can define an ALSA pcm device corresponding to your
Bluetooth speaker. For a headless server using @emph{bluealsa} with a
fixed Bluetooth device is likely simpler than configuring Pulseaudio and
its stream switching behavior. We configure ALSA by crafting a custom
@code{alsa-configuration} for the @code{alsa-service-type}. The
configuration will declare a @code{pcm} type @code{bluealsa} from the
@code{bluealsa} module provided by the @code{bluez-alsa} package, and
then define a @code{pcm} device of that type for your Bluetooth speaker.
All that is left then is to make MPD send audio data to this ALSA
device. We also add a secondary MPD output that makes the currently
played audio files available as a stream through a web server on port
8080. When enabled a device on the network could listen to the audio
stream by connecting any capable media player to the HTTP server on port
8080, independent of the status of the Bluetooth speaker.
What follows is the outline of an @code{operating-system} declaration
that should accomplish the above-mentioned tasks:
@lisp
(use-modules (gnu))
(use-service-modules audio dbus sound #;… etc)
(use-package-modules audio linux #;… etc)
(operating-system
;; …
(packages (cons* bluez bluez-alsa
%base-packages))
(services
;; …
(service mpd-service-type
(mpd-configuration
(user "your-username")
(music-dir "/path/to/your/music")
(address "192.168.178.20")
(outputs (list (mpd-output
(type "alsa")
(name "MPD")
(extra-options
;; Use the same name as in the ALSA
;; configuration below.
'((device . "pcm.btspeaker"))))
(mpd-output
(type "httpd")
(name "streaming")
(enabled? #false)
(always-on? #true)
(tags? #true)
(mixer-type 'null)
(extra-options
'((encoder . "vorbis")
(port . "8080")
(bind-to-address . "192.168.178.20")
(max-clients . "0") ;no limit
(quality . "5.0")
(format . "44100:16:1"))))))))
(dbus-service #:services (list bluez-alsa))
(bluetooth-service #:auto-enable? #t)
(service alsa-service-type
(alsa-configuration
(pulseaudio? #false) ;we don't need it
(extra-options
#~(string-append "\
# Declare Bluetooth audio device type \"bluealsa\" from bluealsa module
pcm_type.bluealsa @{
lib \"" #$(file-append bluez-alsa "/lib/alsa-lib/libasound_module_pcm_bluealsa.so") "\"
@}
# Declare control device type \"bluealsa\" from the same module
ctl_type.bluealsa @{
lib \"" #$(file-append bluez-alsa "/lib/alsa-lib/libasound_module_ctl_bluealsa.so") "\"
@}
# Define the actual Bluetooth audio device.
pcm.btspeaker @{
type bluealsa
device \"AA:BB:CC:A4:AA:CD\" # unique device identifier
profile \"a2dp\"
@}
# Define an associated controller.
ctl.btspeaker @{
type bluealsa
@}
"))))))
@end lisp
Enjoy the music with the MPD client of your choice or a media player
capable of streaming via HTTP!
@c *********************************************************************
@node Containers
@chapter Containers
The kernel Linux provides a number of shared facilities that are
available to processes in the system. These facilities include a shared
view on the file system, other processes, network devices, user and
group identities, and a few others. Since Linux 3.19 a user can choose
to @emph{unshare} some of these shared facilities for selected
processes, providing them (and their child processes) with a different
view on the system.
A process with an unshared @code{mount} namespace, for example, has its
own view on the file system --- it will only be able to see directories
that have been explicitly bound in its mount namespace. A process with
its own @code{proc} namespace will consider itself to be the only
process running on the system, running as PID 1.
Guix uses these kernel features to provide fully isolated environments
and even complete Guix System containers, lightweight virtual machines
that share the host system's kernel. This feature comes in especially
handy when using Guix on a foreign distribution to prevent interference
from foreign libraries or configuration files that are available
system-wide.
@menu
* Guix Containers:: Perfectly isolated environments
* Guix System Containers:: A system inside your system
@end menu
@node Guix Containers
@section Guix Containers
The easiest way to get started is to use @command{guix shell} with the
@option{--container} option. @xref{Invoking guix shell,,, guix, GNU
Guix Reference Manual} for a reference of valid options.
The following snippet spawns a minimal shell process with most
namespaces unshared from the system. The current working directory is
visible to the process, but anything else on the file system is
unavailable. This extreme isolation can be very useful when you want to
rule out any sort of interference from environment variables, globally
installed libraries, or configuration files.
@example
guix shell --container
@end example
It is a bleak environment, barren, desolate. You will find that not
even the GNU coreutils are available here, so to explore this deserted
wasteland you need to use built-in shell commands. Even the usually
gigantic @file{/gnu/store} directory is reduced to a faint shadow of
itself.
@example sh
$ echo /gnu/store/*
/gnu/store/@dots{}-gcc-10.3.0-lib
/gnu/store/@dots{}-glibc-2.33
/gnu/store/@dots{}-bash-static-5.1.8
/gnu/store/@dots{}-ncurses-6.2.20210619
/gnu/store/@dots{}-bash-5.1.8
/gnu/store/@dots{}-profile
/gnu/store/@dots{}-readline-8.1.1
@end example
@cindex exiting a container
There isn't much you can do in an environment like this other than
exiting it. You can use @key{^D} or @command{exit} to terminate this
limited shell environment.
@cindex exposing directories, container
@cindex sharing directories, container
@cindex mapping locations, container
You can make other directories available inside of the container
environment; use @option{--expose=DIRECTORY} to bind-mount the given
directory as a read-only location inside the container, or use
@option{--share=DIRECTORY} to make the location writable. With an
additional mapping argument after the directory name you can control the
name of the directory inside the container. In the following example we
map @file{/etc} on the host system to @file{/the/host/etc} inside a
container in which the GNU coreutils are installed.
@example sh
$ guix shell --container --share=/etc=/the/host/etc coreutils
$ ls /the/host/etc
@end example
Similarly, you can prevent the current working directory from being
mapped into the container with the @option{--no-cwd} option. Another
good idea is to create a dedicated directory that will serve as the
container's home directory, and spawn the container shell from that
directory.
@cindex hide system libraries, container
@cindex avoid ABI mismatch, container
On a foreign system a container environment can be used to compile
software that cannot possibly be linked with system libraries or with
the system's compiler toolchain. A common use-case in a research
context is to install packages from within an R session. Outside of a
container environment there is a good chance that the foreign compiler
toolchain and incompatible system libraries are found first, resulting
in incompatible binaries that cannot be used by R. In a container shell
this problem disappears, as system libraries and executables simply
aren't available due to the unshared @code{mount} namespace.
Let's take a comprehensive manifest providing a comfortable development
environment for use with R:
@lisp
(specifications->manifest
(list "r-minimal"
;; base packages
"bash-minimal"
"glibc-locales"
"nss-certs"
;; Common command line tools lest the container is too empty.
"coreutils"
"grep"
"which"
"wget"
"sed"
;; R markdown tools
"pandoc"
;; Toolchain and common libraries for "install.packages"
"gcc-toolchain@@10"
"gfortran-toolchain"
"gawk"
"tar"
"gzip"
"unzip"
"make"
"cmake"
"pkg-config"
"cairo"
"libxt"
"openssl"
"curl"
"zlib"))
@end lisp
Let's use this to run R inside a container environment. For convenience
we share the @code{net} namespace to use the host system's network
interfaces. Now we can build R packages from source the traditional way
without having to worry about ABI mismatch or incompatibilities.
@example sh
$ guix shell --container --network --manifest=manifest.scm -- R
R version 4.2.1 (2022-06-23) -- "Funny-Looking Kid"
Copyright (C) 2022 The R Foundation for Statistical Computing
@dots{}
> e <- Sys.getenv("GUIX_ENVIRONMENT")
> Sys.setenv(GIT_SSL_CAINFO=paste0(e, "/etc/ssl/certs/ca-certificates.crt"))
> Sys.setenv(SSL_CERT_FILE=paste0(e, "/etc/ssl/certs/ca-certificates.crt"))
> Sys.setenv(SSL_CERT_DIR=paste0(e, "/etc/ssl/certs"))
> install.packages("Cairo", lib=paste0(getwd()))
@dots{}
* installing *source* package 'Cairo' ...
@dots{}
* DONE (Cairo)
The downloaded source packages are in
'/tmp/RtmpCuwdwM/downloaded_packages'
> library("Cairo", lib=getwd())
> # success!
@end example
Using container shells is fun, but they can become a little cumbersome
when you want to go beyond just a single interactive process. Some
tasks become a lot easier when they sit on the rock solid foundation of
a proper Guix System and its rich set of system services. The next
section shows you how to launch a complete Guix System inside of a
container.
@node Guix System Containers
@section Guix System Containers
The Guix System provides a wide array of interconnected system services
that are configured declaratively to form a dependable stateless GNU
System foundation for whatever tasks you throw at it. Even when using
Guix on a foreign distribution you can benefit from the design of Guix
System by running a system instance as a container. Using the same
kernel features of unshared namespaces mentioned in the previous
section, the resulting Guix System instance is isolated from the host
system and only shares file system locations that you explicitly
declare.
A Guix System container differs from the shell process created by
@command{guix shell --container} in a number of important ways. While
in a container shell the containerized process is a Bash shell process,
a Guix System container runs the Shepherd as PID 1. In a system
container all system services (@pxref{Services,,, guix, GNU Guix
Reference Manual}) are set up just as they would be on a Guix System in
a virtual machine or on bare metal---this includes daemons managed by
the GNU@tie{}Shepherd (@pxref{Shepherd Services,,, guix, GNU Guix
Reference Manual}) as well as other kinds of extensions to the operating
system (@pxref{Service Composition,,, guix, GNU Guix Reference Manual}).
The perceived increase in complexity of running a Guix System container
is easily justified when dealing with more complex applications that
have higher or just more rigid requirements on their execution
contexts---configuration files, dedicated user accounts, directories for
caches or log files, etc. In Guix System the demands of this kind of
software are satisfied through the deployment of system services.
@node A Database Container
@subsection A Database Container
A good example might be a PostgreSQL database server. Much of the
complexity of setting up such a database server is encapsulated in this
deceptively short service declaration:
@lisp
(service postgresql-service-type
(postgresql-configuration
(postgresql postgresql-14)))
@end lisp
A complete operating system declaration for use with a Guix System
container would look something like this:
@lisp
(use-modules (gnu))
(use-package-modules databases)
(use-service-modules databases)
(operating-system
(host-name "container")
(timezone "Europe/Berlin")
(file-systems (cons (file-system
(device (file-system-label "does-not-matter"))
(mount-point "/")
(type "ext4"))
%base-file-systems))
(bootloader (bootloader-configuration
(bootloader grub-bootloader)
(targets '("/dev/sdX"))))
(services
(cons* (service postgresql-service-type
(postgresql-configuration
(postgresql postgresql-14)
(config-file
(postgresql-config-file
(log-destination "stderr")
(hba-file
(plain-file "pg_hba.conf"
"\
local all all trust
host all all 10.0.0.1/32 trust"))
(extra-config
'(("listen_addresses" "*")
("log_directory" "/var/log/postgresql")))))))
(service postgresql-role-service-type
(postgresql-role-configuration
(roles
(list (postgresql-role
(name "test")
(create-database? #t))))))
%base-services)))
@end lisp
With @code{postgresql-role-service-type} we define a role ``test'' and
create a matching database, so that we can test right away without any
further manual setup. The @code{postgresql-config-file} settings allow
a client from IP address 10.0.0.1 to connect without requiring
authentication---a bad idea in production systems, but convenient for
this example.
Let's build a script that will launch an instance of this Guix System as
a container. Write the @code{operating-system} declaration above to a
file @file{os.scm} and then use @command{guix system container} to build
the launcher. (@pxref{Invoking guix system,,, guix, GNU Guix Reference
Manual}).
@example
$ guix system container os.scm
The following derivations will be built:
/gnu/store/@dots{}-run-container.drv
@dots{}
building /gnu/store/@dots{}-run-container.drv...
/gnu/store/@dots{}-run-container
@end example
Now that we have a launcher script we can run it to spawn the new system
with a running PostgreSQL service. Note that due to some as yet
unresolved limitations we need to run the launcher as the root user, for
example with @command{sudo}.
@example
$ sudo /gnu/store/@dots{}-run-container
system container is running as PID 5983
@dots{}
@end example
Background the process with @key{Ctrl-z} followed by @command{bg}. Note
the process ID in the output; we will need it to connect to the
container later. You know what? Let's try attaching to the container
right now. We will use @command{nsenter}, a tool provided by the
@code{util-linux} package:
@example
$ guix shell util-linux
$ sudo nsenter -a -t 5983
root@@container /# pgrep -a postgres
49 /gnu/store/@dots{}-postgresql-14.4/bin/postgres -D /var/lib/postgresql/data --config-file=/gnu/store/@dots{}-postgresql.conf -p 5432
51 postgres: checkpointer
52 postgres: background writer
53 postgres: walwriter
54 postgres: autovacuum launcher
55 postgres: stats collector
56 postgres: logical replication launcher
root@@container /# exit
@end example
The PostgreSQL service is running in the container!
@node Container Networking
@subsection Container Networking
@cindex container networking
What good is a Guix System running a PostgreSQL database service as a
container when we can only talk to it with processes originating in the
container? It would be much better if we could talk to the database
over the network.
The easiest way to do this is to create a pair of connected virtual
Ethernet devices (known as @code{veth}). We move one of the devices
(@code{ceth-test}) into the @code{net} namespace of the container and
leave the other end (@code{veth-test}) of the connection on the host
system.
@example
pid=5983
ns="guix-test"
host="veth-test"
client="ceth-test"
# Attach the new net namespace "guix-test" to the container PID.
sudo ip netns attach $ns $pid
# Create the pair of devices
sudo ip link add $host type veth peer name $client
# Move the client device into the container's net namespace
sudo ip link set $client netns $ns
@end example
Then we configure the host side:
@example
sudo ip link set $host up
sudo ip addr add 10.0.0.1/24 dev $host
@end example
@dots{}and then we configure the client side:
@example
sudo ip netns exec $ns ip link set lo up
sudo ip netns exec $ns ip link set $client up
sudo ip netns exec $ns ip addr add 10.0.0.2/24 dev $client
@end example
At this point the host can reach the container at IP address 10.0.0.2,
and the container can reach the host at IP 10.0.0.1. This is all we
need to talk to the database server inside the container from the host
system on the outside.
@example
$ psql -h 10.0.0.2 -U test
psql (14.4)
Type "help" for help.
test=> CREATE TABLE hello (who TEXT NOT NULL);
CREATE TABLE
test=> INSERT INTO hello (who) VALUES ('world');
INSERT 0 1
test=> SELECT * FROM hello;
who
-------
world
(1 row)
@end example
Now that we're done with this little demonstration let's clean up:
@example
sudo kill $pid
sudo ip netns del $ns
sudo ip link del $host
@end example
@c *********************************************************************
@node Advanced package management
@chapter Advanced package management
Guix is a functional package manager that offers many features beyond
what more traditional package managers can do. To the uninitiated,
those features might not have obvious use cases at first. The purpose
of this chapter is to demonstrate some advanced package management
concepts.
@pxref{Package Management,,, guix, GNU Guix Reference Manual} for a complete
reference.
@menu
* Guix Profiles in Practice:: Strategies for multiple profiles and manifests.
@end menu
@node Guix Profiles in Practice
@section Guix Profiles in Practice
Guix provides a very useful feature that may be quite foreign to newcomers:
@emph{profiles}. They are a way to group package installations together and all users
on the same system are free to use as many profiles as they want.
Whether you're a developer or not, you may find that multiple profiles bring you
great power and flexibility. While they shift the paradigm somewhat compared to
@emph{traditional package managers}, they are very convenient to use once you've
understood how to set them up.
If you are familiar with Python's @samp{virtualenv}, you can think of a profile as a
kind of universal @samp{virtualenv} that can hold any kind of software whatsoever, not
just Python software. Furthermore, profiles are self-sufficient: they capture
all the runtime dependencies which guarantees that all programs within a profile
will always work at any point in time.
Multiple profiles have many benefits:
@itemize
@item
Clean semantic separation of the various packages a user needs for different contexts.
@item
Multiple profiles can be made available into the environment either on login
or within a dedicated shell.
@item
Profiles can be loaded on demand. For instance, the user can use multiple
shells, each of them running different profiles.
@item
Isolation: Programs from one profile will not use programs from the other, and
the user can even install different versions of the same programs to the two
profiles without conflict.
@item
Deduplication: Profiles share dependencies that happens to be the exact same.
This makes multiple profiles storage-efficient.
@item
Reproducible: when used with declarative manifests, a profile can be fully
specified by the Guix commit that was active when it was set up. This means
that the exact same profile can be
@uref{https://guix.gnu.org/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/,
set up anywhere and anytime}, with just the commit information. See the
section on @ref{Reproducible profiles}.
@item
Easier upgrades and maintenance: Multiple profiles make it easy to keep
package listings at hand and make upgrades completely frictionless.
@end itemize
Concretely, here follows some typical profiles:
@itemize
@item
The dependencies of a project you are working on.
@item
Your favourite programming language libraries.
@item
Laptop-specific programs (like @samp{powertop}) that you don't need on a desktop.
@item
@TeX{}live (this one can be really useful when you need to install just one
package for this one document you've just received over email).
@item
Games.
@end itemize
Let's dive in the set up!
@node Basic setup with manifests
@subsection Basic setup with manifests
A Guix profile can be set up @i{via} a @dfn{manifest}. A manifest is a
snippet of Scheme code that specifies the set of packages you want to
have in your profile; it looks like this:
@lisp
(specifications->manifest
'("package-1"
;; Version 1.3 of package-2.
"package-2@@1.3"
;; The "lib" output of package-3.
"package-3:lib"
; ...
"package-N"))
@end lisp
@xref{Writing Manifests,,, guix, GNU Guix Reference Manual}, for
more information about the syntax.
We can create a manifest specification per profile and install them this way:
@example
GUIX_EXTRA_PROFILES=$HOME/.guix-extra-profiles
mkdir -p "$GUIX_EXTRA_PROFILES"/my-project # if it does not exist yet
guix package --manifest=/path/to/guix-my-project-manifest.scm --profile="$GUIX_EXTRA_PROFILES"/my-project/my-project
@end example
Here we set an arbitrary variable @samp{GUIX_EXTRA_PROFILES} to point to the directory
where we will store our profiles in the rest of this article.
Placing all your profiles in a single directory, with each profile getting its
own sub-directory, is somewhat cleaner. This way, each sub-directory will
contain all the symlinks for precisely one profile. Besides, ``looping over
profiles'' becomes obvious from any programming language (e.g.@: a shell script) by
simply looping over the sub-directories of @samp{$GUIX_EXTRA_PROFILES}.
Note that it's also possible to loop over the output of
@example
guix package --list-profiles
@end example
although you'll probably have to filter out @file{~/.config/guix/current}.
To enable all profiles on login, add this to your @file{~/.bash_profile} (or similar):
@example
for i in $GUIX_EXTRA_PROFILES/*; do
profile=$i/$(basename "$i")
if [ -f "$profile"/etc/profile ]; then
GUIX_PROFILE="$profile"
. "$GUIX_PROFILE"/etc/profile
fi
unset profile
done
@end example
Note to Guix System users: the above reflects how your default profile
@file{~/.guix-profile} is activated from @file{/etc/profile}, that latter being loaded by
@file{~/.bashrc} by default.
You can obviously choose to only enable a subset of them:
@example
for i in "$GUIX_EXTRA_PROFILES"/my-project-1 "$GUIX_EXTRA_PROFILES"/my-project-2; do
profile=$i/$(basename "$i")
if [ -f "$profile"/etc/profile ]; then
GUIX_PROFILE="$profile"
. "$GUIX_PROFILE"/etc/profile
fi
unset profile
done
@end example
When a profile is off, it's straightforward to enable it for an individual shell
without "polluting" the rest of the user session:
@example
GUIX_PROFILE="path/to/my-project" ; . "$GUIX_PROFILE"/etc/profile
@end example
The key to enabling a profile is to @emph{source} its @samp{etc/profile} file. This file
contains shell code that exports the right environment variables necessary to
activate the software contained in the profile. It is built automatically by
Guix and meant to be sourced.
It contains the same variables you would get if you ran:
@example
guix package --search-paths=prefix --profile=$my_profile"
@end example
Once again, see (@pxref{Invoking guix package,,, guix, GNU Guix Reference Manual})
for the command line options.
To upgrade a profile, simply install the manifest again:
@example
guix package -m /path/to/guix-my-project-manifest.scm -p "$GUIX_EXTRA_PROFILES"/my-project/my-project
@end example
To upgrade all profiles, it's easy enough to loop over them. For instance,
assuming your manifest specifications are stored in
@file{~/.guix-manifests/guix-$profile-manifest.scm}, with @samp{$profile} being the name
of the profile (e.g.@: "project1"), you could do the following in Bourne shell:
@example
for profile in "$GUIX_EXTRA_PROFILES"/*; do
guix package --profile="$profile" --manifest="$HOME/.guix-manifests/guix-$profile-manifest.scm"
done
@end example
Each profile has its own generations:
@example
guix package -p "$GUIX_EXTRA_PROFILES"/my-project/my-project --list-generations
@end example
You can roll-back to any generation of a given profile:
@example
guix package -p "$GUIX_EXTRA_PROFILES"/my-project/my-project --switch-generations=17
@end example
Finally, if you want to switch to a profile without inheriting from the
current environment, you can activate it from an empty shell:
@example
env -i $(which bash) --login --noprofile --norc
. my-project/etc/profile
@end example
@node Required packages
@subsection Required packages
Activating a profile essentially boils down to exporting a bunch of
environmental variables. This is the role of the @samp{etc/profile} within the
profile.
@emph{Note: Only the environmental variables of the packages that consume them will
be set.}
For instance, @samp{MANPATH} won't be set if there is no consumer application for man
pages within the profile. So if you need to transparently access man pages once
the profile is loaded, you've got two options:
@itemize
@item
Either export the variable manually, e.g.
@example
export MANPATH=/path/to/profile$@{MANPATH:+:@}$MANPATH
@end example
@item
Or include @samp{man-db} to the profile manifest.
@end itemize
The same is true for @samp{INFOPATH} (you can install @samp{info-reader}),
@samp{PKG_CONFIG_PATH} (install @samp{pkg-config}), etc.
@node Default profile
@subsection Default profile
What about the default profile that Guix keeps in @file{~/.guix-profile}?
You can assign it the role you want. Typically you would install the manifest
of the packages you want to use all the time.
Alternatively, you could keep it ``manifest-less'' for throw-away packages
that you would just use for a couple of days.
This way makes it convenient to run
@example
guix install package-foo
guix upgrade package-bar
@end example
without having to specify the path to a profile.
@node The benefits of manifests
@subsection The benefits of manifests
Manifests are a convenient way to keep your package lists around and, say,
to synchronize them across multiple machines using a version control system.
A common complaint about manifests is that they can be slow to install when they
contain large number of packages. This is especially cumbersome when you just
want get an upgrade for one package within a big manifest.
This is one more reason to use multiple profiles, which happen to be just
perfect to break down manifests into multiple sets of semantically connected
packages. Using multiple, small profiles provides more flexibility and
usability.
Manifests come with multiple benefits. In particular, they ease maintenance:
@itemize
@item
When a profile is set up from a manifest, the manifest itself is
self-sufficient to keep a ``package listing'' around and reinstall the profile
later or on a different system. For ad-hoc profiles, we would need to
generate a manifest specification manually and maintain the package versions
for the packages that don't use the default version.
@item
@code{guix package --upgrade} always tries to update the packages that have
propagated inputs, even if there is nothing to do. Guix manifests remove this
problem.
@item
When partially upgrading a profile, conflicts may arise (due to diverging
dependencies between the updated and the non-updated packages) and they can be
annoying to resolve manually. Manifests remove this problem altogether since
all packages are always upgraded at once.
@item
As mentioned above, manifests allow for reproducible profiles, while the
imperative @code{guix install}, @code{guix upgrade}, etc. do not, since they produce
different profiles every time even when they hold the same packages. See
@uref{https://issues.guix.gnu.org/issue/33285, the related discussion on the matter}.
@item
Manifest specifications are usable by other @samp{guix} commands. For example, you
can run @code{guix weather -m manifest.scm} to see how many substitutes are
available, which can help you decide whether you want to try upgrading today
or wait a while. Another example: you can run @code{guix pack -m manifest.scm} to
create a pack containing all the packages in the manifest (and their
transitive references).
@item
Finally, manifests have a Scheme representation, the @samp{<manifest>} record type.
They can be manipulated in Scheme and passed to the various Guix @uref{https://en.wikipedia.org/wiki/Api, APIs}.
@end itemize
It's important to understand that while manifests can be used to declare
profiles, they are not strictly equivalent: profiles have the side effect that
they ``pin'' packages in the store, which prevents them from being
garbage-collected (@pxref{Invoking guix gc,,, guix, GNU Guix Reference Manual})
and ensures that they will still be available at any point in
the future.
Let's take an example:
@enumerate
@item
We have an environment for hacking on a project for which there isn't a Guix
package yet. We build the environment using a manifest, and then run @code{guix
environment -m manifest.scm}. So far so good.
@item
Many weeks pass and we have run a couple of @code{guix pull} in the mean time.
Maybe a dependency from our manifest has been updated; or we may have run
@code{guix gc} and some packages needed by our manifest have been
garbage-collected.
@item
Eventually, we set to work on that project again, so we run @code{guix shell
-m manifest.scm}. But now we have to wait for Guix to build and install
stuff!
@end enumerate
Ideally, we could spare the rebuild time. And indeed we can, all we need is to
install the manifest to a profile and use @code{GUIX_PROFILE=/the/profile;
. "$GUIX_PROFILE"/etc/profile} as explained above: this guarantees that our
hacking environment will be available at all times.
@emph{Security warning:} While keeping old profiles around can be convenient, keep in
mind that outdated packages may not have received the latest security fixes.
@node Reproducible profiles
@subsection Reproducible profiles
To reproduce a profile bit-for-bit, we need two pieces of information:
@itemize
@item
a manifest,
@item
a Guix channel specification.
@end itemize
Indeed, manifests alone might not be enough: different Guix versions (or
different channels) can produce different outputs for a given manifest.
You can output the Guix channel specification with @samp{guix describe
--format=channels}.
Save this to a file, say @samp{channel-specs.scm}.
On another computer, you can use the channel specification file and the manifest
to reproduce the exact same profile:
@example
GUIX_EXTRA_PROFILES=$HOME/.guix-extra-profiles
GUIX_EXTRA=$HOME/.guix-extra
mkdir -p "$GUIX_EXTRA"/my-project
guix pull --channels=channel-specs.scm --profile="$GUIX_EXTRA/my-project/guix"
mkdir -p "$GUIX_EXTRA_PROFILES/my-project"
"$GUIX_EXTRA"/my-project/guix/bin/guix package --manifest=/path/to/guix-my-project-manifest.scm --profile="$GUIX_EXTRA_PROFILES"/my-project/my-project
@end example
It's safe to delete the Guix channel profile you've just installed with the
channel specification, the project profile does not depend on it.
@c *********************************************************************
@node Environment management
@chapter Environment management
Guix provides multiple tools to manage environment. This chapter
demonstrate such utilities.
@menu
* Guix environment via direnv:: Setup Guix environment with direnv
@end menu
@node Guix environment via direnv
@section Guix environment via direnv
Guix provides a @samp{direnv} package, which could extend shell after
directory change. This tool could be used to prepare a pure Guix
environment.
The following example provides a shell function for @file{~/.direnvrc}
file, which could be used from Guix Git repository in
@file{~/src/guix/.envrc} file to setup a build environment similar to
described in @pxref{Building from Git,,, guix, GNU Guix Reference
Manual}.
Create a @file{~/.direnvrc} with a Bash code:
@example
# Thanks <https://github.com/direnv/direnv/issues/73#issuecomment-152284914>
export_function()
@{
local name=$1
local alias_dir=$PWD/.direnv/aliases
mkdir -p "$alias_dir"
PATH_add "$alias_dir"
local target="$alias_dir/$name"
if declare -f "$name" >/dev/null; then
echo "#!$SHELL" > "$target"
declare -f "$name" >> "$target" 2>/dev/null
# Notice that we add shell variables to the function trigger.
echo "$name \$*" >> "$target"
chmod +x "$target"
fi
@}
use_guix()
@{
# Set GitHub token.
export GUIX_GITHUB_TOKEN="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
# Unset 'GUIX_PACKAGE_PATH'.
export GUIX_PACKAGE_PATH=""
# Recreate a garbage collector root.
gcroots="$HOME/.config/guix/gcroots"
mkdir -p "$gcroots"
gcroot="$gcroots/guix"
if [ -L "$gcroot" ]
then
rm -v "$gcroot"
fi
# Miscellaneous packages.
PACKAGES_MAINTENANCE=(
direnv
git
git:send-email
git-cal
gnupg
guile-colorized
guile-readline
less
ncurses
openssh
xdot
)
# Environment packages.
PACKAGES=(help2man guile-sqlite3 guile-gcrypt)
# Thanks <https://lists.gnu.org/archive/html/guix-devel/2016-09/msg00859.html>
eval "$(guix environment --search-paths --root="$gcroot" --pure guix --ad-hoc $@{PACKAGES[@@]@} $@{PACKAGES_MAINTENANCE[@@]@} "$@@")"
# Predefine configure flags.
configure()
@{
./configure --localstatedir=/var --prefix=
@}
export_function configure
# Run make and optionally build something.
build()
@{
make -j 2
if [ $# -gt 0 ]
then
./pre-inst-env guix build "$@@"
fi
@}
export_function build
# Predefine push Git command.
push()
@{
git push --set-upstream origin
@}
export_function push
clear # Clean up the screen.
git-cal --author='Your Name' # Show contributions calendar.
# Show commands help.
echo "
build build a package or just a project if no argument provided
configure run ./configure with predefined parameters
push push to upstream Git repository
"
@}
@end example
Every project containing @file{.envrc} with a string @code{use guix}
will have predefined environment variables and procedures.
Run @command{direnv allow} to setup the environment for the first time.
@c *********************************************************************
@node Installing Guix on a Cluster
@chapter Installing Guix on a Cluster
@cindex cluster installation
@cindex high-performance computing, HPC
@cindex HPC, high-performance computing
Guix is appealing to scientists and @acronym{HPC, high-performance
computing} practitioners: it makes it easy to deploy potentially complex
software stacks, and it lets you do so in a reproducible fashion---you
can redeploy the exact same software on different machines and at
different points in time.
In this chapter we look at how a cluster sysadmin can install Guix for
system-wide use, such that it can be used on all the cluster nodes, and
discuss the various tradeoffs@footnote{This chapter is adapted from a
@uref{https://hpc.guix.info/blog/2017/11/installing-guix-on-a-cluster/,
blog post published on the Guix-HPC web site in 2017}.}.
@quotation Note
Here we assume that the cluster is running a GNU/Linux distro other than
Guix System and that we are going to install Guix on top of it.
@end quotation
@menu
* Setting Up a Head Node:: The node that runs the daemon.
* Setting Up Compute Nodes:: Client nodes.
* Cluster Network Access:: Dealing with network access restrictions.
* Cluster Disk Usage:: Disk usage considerations.
* Cluster Security Considerations:: Keeping the cluster secure.
@end menu
@node Setting Up a Head Node
@section Setting Up a Head Node
The recommended approach is to set up one @emph{head node} running
@command{guix-daemon} and exporting @file{/gnu/store} over NFS to
compute nodes.
Remember that @command{guix-daemon} is responsible for spawning build
processes and downloads on behalf of clients (@pxref{Invoking
guix-daemon,,, guix, GNU Guix Reference Manual}), and more generally
accessing @file{/gnu/store}, which contains all the package binaries
built by all the users (@pxref{The Store,,, guix, GNU Guix Reference
Manual}). ``Client'' here refers to all the Guix commands that users
see, such as @code{guix install}. On a cluster, these commands may be
running on the compute nodes and we'll want them to talk to the head
node's @code{guix-daemon} instance.
To begin with, the head node can be installed following the usual binary
installation instructions (@pxref{Binary Installation,,, guix, GNU Guix
Reference Manual}). Thanks to the installation script, this should be
quick. Once installation is complete, we need to make some adjustments.
Since we want @code{guix-daemon} to be reachable not just from the head
node but also from the compute nodes, we need to arrange so that it
listens for connections over TCP/IP. To do that, we'll edit the systemd
startup file for @command{guix-daemon},
@file{/etc/systemd/system/guix-daemon.service}, and add a
@code{--listen} argument to the @code{ExecStart} line so that it looks
something like this:
@example
ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon --build-users-group=guixbuild --listen=/var/guix/daemon-socket/socket --listen=0.0.0.0
@end example
For these changes to take effect, the service needs to be restarted:
@example
systemctl daemon-reload
systemctl restart guix-daemon
@end example
@quotation Note
The @code{--listen=0.0.0.0} bit means that @code{guix-daemon} will
process @emph{all} incoming TCP connections on port 44146
(@pxref{Invoking guix-daemon,,, guix, GNU Guix Reference Manual}). This
is usually fine in a cluster setup where the head node is reachable
exclusively from the cluster's local area network---you don't want that
to be exposed to the Internet!
@end quotation
The next step is to define our NFS exports in
@uref{https://linux.die.net/man/5/exports,@file{/etc/exports}} by adding
something along these lines:
@example
/gnu/store *(ro)
/var/guix *(rw, async)
/var/log/guix *(ro)
@end example
The @file{/gnu/store} directory can be exported read-only since only
@command{guix-daemon} on the master node will ever modify it.
@file{/var/guix} contains @emph{user profiles} as managed by @code{guix
package}; thus, to allow users to install packages with @code{guix
package}, this must be read-write.
Users can create as many profiles as they like in addition to the
default profile, @file{~/.guix-profile}. For instance, @code{guix
package -p ~/dev/python-dev -i python} installs Python in a profile
reachable from the @code{~/dev/python-dev} symlink. To make sure that
this profile is protected from garbage collection---i.e., that Python
will not be removed from @file{/gnu/store} while this profile exists---,
@emph{home directories should be mounted on the head node} as well so
that @code{guix-daemon} knows about these non-standard profiles and
avoids collecting software they refer to.
It may be a good idea to periodically remove unused bits from
@file{/gnu/store} by running @command{guix gc} (@pxref{Invoking guix
gc,,, guix, GNU Guix Reference Manual}). This can be done by adding a
crontab entry on the head node:
@example
root@@master# crontab -e
@end example
@noindent
... with something like this:
@example
# Every day at 5AM, run the garbage collector to make sure
# at least 10 GB are free on /gnu/store.
0 5 * * 1 /usr/local/bin/guix gc -F10G
@end example
We're done with the head node! Let's look at compute nodes now.
@node Setting Up Compute Nodes
@section Setting Up Compute Nodes
First of all, we need compute nodes to mount those NFS directories that
the head node exports. This can be done by adding the following lines
to @uref{https://linux.die.net/man/5/fstab,@file{/etc/fstab}}:
@example
@var{head-node}:/gnu/store /gnu/store nfs defaults,_netdev,vers=3 0 0
@var{head-node}:/var/guix /var/guix nfs defaults,_netdev,vers=3 0 0
@var{head-node}:/var/log/guix /var/log/guix nfs defaults,_netdev,vers=3 0 0
@end example
@noindent
... where @var{head-node} is the name or IP address of your head node.
From there on, assuming the mount points exist, you should be able to
mount each of these on the compute nodes.
Next, we need to provide a default @command{guix} command that users can
run when they first connect to the cluster (eventually they will invoke
@command{guix pull}, which will provide them with their ``own''
@command{guix} command). Similar to what the binary installation script
did on the head node, we'll store that in @file{/usr/local/bin}:
@example
mkdir -p /usr/local/bin
ln -s /var/guix/profiles/per-user/root/current-guix/bin/guix \
/usr/local/bin/guix
@end example
We then need to tell @code{guix} to talk to the daemon running on our
master node, by adding these lines to @code{/etc/profile}:
@example
GUIX_DAEMON_SOCKET="guix://@var{head-node}"
export GUIX_DAEMON_SOCKET
@end example
To avoid warnings and make sure @code{guix} uses the right locale, we
need to tell it to use locale data provided by Guix (@pxref{Application
Setup,,, guix, GNU Guix Reference Manual}):
@example
GUIX_LOCPATH=/var/guix/profiles/per-user/root/guix-profile/lib/locale
export GUIX_LOCPATH
# Here we must use a valid locale name. Try "ls $GUIX_LOCPATH/*"
# to see what names can be used.
LC_ALL=fr_FR.utf8
export LC_ALL
@end example
For convenience, @code{guix package} automatically generates
@file{~/.guix-profile/etc/profile}, which defines all the environment
variables necessary to use the packages---@code{PATH},
@code{C_INCLUDE_PATH}, @code{PYTHONPATH}, etc. Thus it's a good idea to
source it from @code{/etc/profile}:
@example
GUIX_PROFILE="$HOME/.guix-profile"
if [ -f "$GUIX_PROFILE/etc/profile" ]; then
. "$GUIX_PROFILE/etc/profile"
fi
@end example
Last but not least, Guix provides command-line completion notably for
Bash and zsh. In @code{/etc/bashrc}, consider adding this line:
@verbatim
. /var/guix/profiles/per-user/root/current-guix/etc/bash_completion.d/guix
@end verbatim
Voilà!
You can check that everything's in place by logging in on a compute node
and running:
@example
guix install hello
@end example
The daemon on the head node should download pre-built binaries on your
behalf and unpack them in @file{/gnu/store}, and @command{guix install}
should create @file{~/.guix-profile} containing the
@file{~/.guix-profile/bin/hello} command.
@node Cluster Network Access
@section Network Access
Guix requires network access to download source code and pre-built
binaries. The good news is that only the head node needs that since
compute nodes simply delegate to it.
It is customary for cluster nodes to have access at best to a
@emph{white list} of hosts. Our head node needs at least
@code{ci.guix.gnu.org} in this white list since this is where it gets
pre-built binaries from by default, for all the packages that are in
Guix proper.
Incidentally, @code{ci.guix.gnu.org} also serves as a
@emph{content-addressed mirror} of the source code of those packages.
Consequently, it is sufficient to have @emph{only}
@code{ci.guix.gnu.org} in that white list.
Software packages maintained in a separate repository such as one of the
various @uref{https://hpc.guix.info/channels, HPC channels} are of
course unavailable from @code{ci.guix.gnu.org}. For these packages, you
may want to extend the white list such that source and pre-built
binaries (assuming this-party servers provide binaries for these
packages) can be downloaded. As a last resort, users can always
download source on their workstation and add it to the cluster's
@file{/gnu/store}, like this:
@verbatim
GUIX_DAEMON_SOCKET=ssh://compute-node.example.org \
guix download http://starpu.gforge.inria.fr/files/starpu-1.2.3/starpu-1.2.3.tar.gz
@end verbatim
The above command downloads @code{starpu-1.2.3.tar.gz} @emph{and} sends
it to the cluster's @code{guix-daemon} instance over SSH.
Air-gapped clusters require more work. At the moment, our suggestion
would be to download all the necessary source code on a workstation
running Guix. For instance, using the @option{--sources} option of
@command{guix build} (@pxref{Invoking guix build,,, guix, GNU Guix
Reference Manual}), the example below downloads all the source code the
@code{openmpi} package depends on:
@example
$ guix build --sources=transitive openmpi
@dots{}
/gnu/store/xc17sm60fb8nxadc4qy0c7rqph499z8s-openmpi-1.10.7.tar.bz2
/gnu/store/s67jx92lpipy2nfj5cz818xv430n4b7w-gcc-5.4.0.tar.xz
/gnu/store/npw9qh8a46lrxiwh9xwk0wpi3jlzmjnh-gmp-6.0.0a.tar.xz
/gnu/store/hcz0f4wkdbsvsdky3c0vdvcawhdkyldb-mpfr-3.1.5.tar.xz
/gnu/store/y9akh452n3p4w2v631nj0injx7y0d68x-mpc-1.0.3.tar.gz
/gnu/store/6g5c35q8avfnzs3v14dzl54cmrvddjm2-glibc-2.25.tar.xz
/gnu/store/p9k48dk3dvvk7gads7fk30xc2pxsd66z-hwloc-1.11.8.tar.bz2
/gnu/store/cry9lqidwfrfmgl0x389cs3syr15p13q-gcc-5.4.0.tar.xz
/gnu/store/7ak0v3rzpqm2c5q1mp3v7cj0rxz0qakf-libfabric-1.4.1.tar.bz2
/gnu/store/vh8syjrsilnbfcf582qhmvpg1v3rampf-rdma-core-14.tar.gz
…
@end example
(In case you're wondering, that's more than 320@ MiB of
@emph{compressed} source code.)
We can then make a big archive containing all of this (@pxref{Invoking
guix archive,,, guix, GNU Guix Reference Manual}):
@verbatim
$ guix archive --export \
`guix build --sources=transitive openmpi` \
> openmpi-source-code.nar
@end verbatim
@dots{} and we can eventually transfer that archive to the cluster on
removable storage and unpack it there:
@verbatim
$ guix archive --import < openmpi-source-code.nar
@end verbatim
This process has to be repeated every time new source code needs to be
brought to the cluster.
As we write this, the research institutes involved in Guix-HPC do not
have air-gapped clusters though. If you have experience with such
setups, we would like to hear feedback and suggestions.
@node Cluster Disk Usage
@section Disk Usage
@cindex disk usage, on a cluster
A common concern of sysadmins' is whether this is all going to eat a lot
of disk space. If anything, if something is going to exhaust disk
space, it's going to be scientific data sets rather than compiled
software---that's our experience with almost ten years of Guix usage on
HPC clusters. Nevertheless, it's worth taking a look at how Guix
contributes to disk usage.
First, having several versions or variants of a given package in
@file{/gnu/store} does not necessarily cost much, because
@command{guix-daemon} implements deduplication of identical files, and
package variants are likely to have a number of common files.
As mentioned above, we recommend having a cron job to run @code{guix gc}
periodically, which removes @emph{unused} software from
@file{/gnu/store}. However, there's always a possibility that users will
keep lots of software in their profiles, or lots of old generations of
their profiles, which is ``live'' and cannot be deleted from the
viewpoint of @command{guix gc}.
The solution to this is for users to regularly remove old generations of
their profile. For instance, the following command removes generations
that are more than two-month old:
@example
guix package --delete-generations=2m
@end example
Likewise, it's a good idea to invite users to regularly upgrade their
profile, which can reduce the number of variants of a given piece of
software stored in @file{/gnu/store}:
@example
guix pull
guix upgrade
@end example
As a last resort, it is always possible for sysadmins to do some of this
on behalf of their users. Nevertheless, one of the strengths of Guix is
the freedom and control users get on their software environment, so we
strongly recommend leaving users in control.
@node Cluster Security Considerations
@section Security Considerations
@cindex security, on a cluster
On an HPC cluster, Guix is typically used to manage scientific software.
Security-critical software such as the operating system kernel and
system services such as @code{sshd} and the batch scheduler remain under
control of sysadmins.
The Guix project has a good track record delivering security updates in
a timely fashion (@pxref{Security Updates,,, guix, GNU Guix Reference
Manual}). To get security updates, users have to run @code{guix pull &&
guix upgrade}.
Because Guix uniquely identifies software variants, it is easy to see if
a vulnerable piece of software is in use. For instance, to check whether
the glibc@ 2.25 variant without the mitigation patch against
``@uref{https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt,Stack
Clash}'', one can check whether user profiles refer to it at all:
@example
guix gc --referrers /gnu/store/…-glibc-2.25
@end example
This will report whether profiles exist that refer to this specific
glibc variant.
@c *********************************************************************
@node Acknowledgments
@chapter Acknowledgments
Guix is based on the @uref{https://nixos.org/nix/, Nix package manager},
which was designed and
implemented by Eelco Dolstra, with contributions from other people (see
the @file{nix/AUTHORS} file in Guix.) Nix pioneered functional package
management, and promoted unprecedented features, such as transactional
package upgrades and rollbacks, per-user profiles, and referentially
transparent build processes. Without this work, Guix would not exist.
The Nix-based software distributions, Nixpkgs and NixOS, have also been
an inspiration for Guix.
GNU@tie{}Guix itself is a collective work with contributions from a
number of people. See the @file{AUTHORS} file in Guix for more
information on these fine people. The @file{THANKS} file lists people
who have helped by reporting bugs, taking care of the infrastructure,
providing artwork and themes, making suggestions, and more---thank you!
This document includes adapted sections from articles that have
previously been published on the Guix blog at
@uref{https://guix.gnu.org/blog} and on the Guix-HPC blog at
@uref{https://hpc.guix.info/blog}.
@c *********************************************************************
@node GNU Free Documentation License
@appendix GNU Free Documentation License
@cindex license, GNU Free Documentation License
@include fdl-1.3.texi
@c *********************************************************************
@node Concept Index
@unnumbered Concept Index
@printindex cp
@bye
@c Local Variables:
@c ispell-local-dictionary: "american";
@c End:
Reference in New Issue View Git Blame Copy Permalink