Having substitute URLs explicitly listed in the service startup file
makes it clearer what should be modified to permanently change the list
of substitute URLs.
* config-daemon.ac: Rename ‘guix_substitute_urls’ to
‘GUIX_SUBSTITUTE_URLS’ and substitute it.
* nix/local.mk (etc/guix-%.service, etc/init.d/guix-daemon)
(etc/guix-%.conf): Substitute it.
* etc/guix-daemon.conf.in, etc/guix-daemon.service.in,
etc/init.d/guix-daemon.in: Add an explicit ‘--substitute-urls’ option.
Change-Id: Ie491b7fab5c42e54dca582801c03805a85de2bf9
Fixes <https://issues.guix.gnu.org/70877>.
* nix/libutil/util.cc (copyFile): Loop over ‘copy_file_range’ instead of
throwing upon short write.
Reported-by: Ricardo Wurmus <rekado@elephly.net>
Change-Id: Id7b8a65ea59006c2d91bc23732309a68665b9ca0
This is a followup to 8f4ffb3fae.
Commit 8f4ffb3fae fell short in two
ways: (1) it didn’t have any effet for fixed-output derivations
performed in a chroot, which is the case for all of them except those
using “builtin:download” and “builtin:git-download”, and (2) it did not
preserve ownership when copying, leading to “suspicious ownership or
permission […] rejecting this build output” errors.
* nix/libstore/build.cc (DerivationGoal::buildDone): Account for
‘chrootRootDir’ when copying ‘drv.outputs’.
* nix/libutil/util.cc (copyFileRecursively): Add ‘fchown’ and ‘fchownat’
calls to preserve file ownership; this is necessary for chrooted
fixed-output derivation builds.
* nix/libutil/util.hh: Update comment.
Change-Id: Ib59f040e98fed59d1af81d724b874b592cbef156
This fixes a security issue (CVE-2024-27297) whereby a fixed-output
derivation build process could open a writable file descriptor to its
output, send it to some outside process for instance over an abstract
AF_UNIX socket, which would then allow said process to modify the file
in the store after it has been marked as “valid”.
Vulnerability discovered by puck <https://github.com/puckipedia>.
Nix security advisory:
https://github.com/NixOS/nix/security/advisories/GHSA-2ffj-w4mj-pg37
Nix fix:
244f3eee0b
* nix/libutil/util.cc (readDirectory): Add variants that take a DIR* and
a file descriptor. Rewrite the ‘Path’ variant accordingly.
(copyFile, copyFileRecursively): New functions.
* nix/libutil/util.hh (copyFileRecursively): New declaration.
* nix/libstore/build.cc (DerivationGoal::buildDone): When ‘fixedOutput’
is true, call ‘copyFileRecursively’ followed by ‘rename’ on each output.
Change-Id: I7952d41093eed26e123e38c14a4c1424be1ce1c4
Reported-by: Picnoir <picnoir@alternativebit.fr>, Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
Change-Id: Idb5f2757f35af86b032a9851cecb19b70227bd88
Having a timeout seems generally preferable as it makes sure build slots
are not kept busy for no good reason (few package builds, if any, are
expected to exceed these values).
* nix/libstore/globals.cc (Settings::Settings): Change ‘maxSilentTime’
and ‘buildTimeout’.
* gnu/services/base.scm (<guix-configuration>)[max-silent-time]
[timeout]: Change default values.
* doc/guix.texi (Invoking guix-daemon, Base Services): Adjust
accordingly.
Change-Id: I25c50893f3f7fcca451b8f093d9d4d1a15fa58d8
* nix/libstore/build.cc (SubstitutionGoal::finished): Don’t show what
the empty status looks like.
Change-Id: Ie898432aeb047aff3d59024de6ed6d18f68903c4
This reverts commit 69f6edc1a8.
The intention is good, but nodist_systemdservice_DATA are meant to be
disposable artefacts generated from corresponding ‘.in’ files.
etc/guix-gc.timer doesn't fit that description, breaking builds:
$ make clean && make
…
make[2]: *** No rule to make target 'etc/guix-gc.timer', needed by 'all-am'. Stop.
The new builder makes it possible to break cycles that occurs when the
fixed-output derivation for the source of a dependency of ‘git’ would
itself depend on ‘git’.
* guix/scripts/perform-download.scm (perform-git-download): New
procedure.
(perform-download): Move fixed-output derivation check to…
(guix-perform-download): … here. Invoke ‘perform-download’ or
‘perform-git-download’ depending on what ‘derivation-builder’ returns.
* nix/libstore/builtins.cc (builtins): Add “git-download”.
* tests/derivations.scm ("built-in-builders"): Update.
("'git-download' built-in builder")
("'git-download' built-in builder, invalid hash")
("'git-download' built-in builder, invalid commit")
("'git-download' built-in builder, not found"): New tests.
The sqlite.hh file uses fixed-width integer types from stdint.h. As
such, it needs to include <cstdint>. Without this include, the file
doesn't compile successfully with GCC13.
See: https://gcc.gnu.org/gcc-13/porting_to.html#header-dep-changes
* nix/libstore/sqlite.hh: include <cstdint>
Signed-off-by: Sören Tempel <soeren@soeren-tempel.net>
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
* nix/libutil/hash.cc (parseHash): Show the hash algorithm name and
expected size in the error message.
* tests/derivations.scm ("fixed-output derivation, invalid hash size"):
New test.
Fixes <https://issues.guix.gnu.org/25018>.
* nix/libstore/gc.cc (readTempRoots): Add a check to guard against removing
the temporary roots of a living process.
Signed-off-by: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Drop the confusing ‘invalid’ jargon and display a hint like we do
for ‘--fallback’.
* nix/libstore/build.cc (DerivationGoal::outputsSubstituted): Rewrite error message.
DerivationGoal::startBuilder() is waiting for an empty line as a check that
the environment set up is fine.
Fixes <https://issues.guix.gnu.org/55324>.
* nix/libstore/build.cc (DerivationGoal::runChild): Remove 'debug'
statement corresponding to bind mounts.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Fixes <https://issues.guix.gnu.org/46212>.
Reported by Christopher Baines <mail@cbaines.net>.
Previously, the nar size returned by 'guix substitute' would be read as
an 'int'; thus, values above 2^31 - 1 would be read and then stored as
negative integers in the database.
Regression introduced in 9dfa20a22a.
* nix/libstore/build.cc (SubstitutionGoal::finished): Use templatized
'string2Int' instead of 'std::atoi' to get an 'unsigned long long',
which is the type of 'hash.second'.
* tests/store.scm ("substitute and large size"): New test.
Fixes <https://issues.guix.gnu.org/51983>.
Reported by Tobias Geerinckx-Rice <me@tobias.gr>.
* nix/libstore/local-store.cc (LocalStore::querySubstitutablePathInfos):
Expect 'unsigned long long' for 'downloadSize' and 'narSize'.
* tests/store.scm ("substitute query and large size"): New test.
'remove' calls 'unlink' first and falls back to 'rmdir' upon EISDIR.
This change gets rid of the 'unlink' call for every directory being
removed.
* nix/libutil/util.cc (_deletePath): Call 'unlink' or 'rmdir' depending
on 'st.st_mode', rather than call 'remove'.
Files smaller than 8 KiB typically represent ~70% of the entries in
/gnu/store/.links but only contribute to ~4% of the space savings
afforded by deduplication.
Not considering these files for deduplication speeds up file insertion
in the store and, more importantly, leaves 'removeUnusedLinks' with
fewer entries to traverse, thereby speeding it up proportionally.
Partly fixes <https://issues.guix.gnu.org/24937>.
* config-daemon.ac: Remove symlink hard link check and CAN_LINK_SYMLINK
definition.
* guix/store/deduplication.scm (%deduplication-minimum-size): New
variable.
(deduplicate)[loop]: Do not recurse when FILE's size is below
%DEDUPLICATION-MINIMUM-SIZE.
(dump-port): New procedure.
(dump-file/deduplicate)[hash]: Turn into...
[dump-and-compute-hash]: ... this thunk.
Call 'deduplicate' only when SIZE is greater than
%DEDUPLICATION-MINIMUM-SIZE; otherwise call 'dump-port'.
* nix/libstore/gc.cc (LocalStore::removeUnusedLinks): Drop files where
st.st_size < deduplicationMinSize.
* nix/libstore/local-store.hh (deduplicationMinSize): New declaration.
* nix/libstore/optimise-store.cc (deduplicationMinSize): New variable.
(LocalStore::optimisePath_): Return when PATH is a symlink or smaller
than 'deduplicationMinSize'.
* tests/derivations.scm ("identical files are deduplicated"): Produce
files bigger than %DEDUPLICATION-MINIMUM-SIZE.
* tests/nar.scm ("restore-file-set with directories (signed, valid)"):
Likewise.
* tests/store-deduplication.scm ("deduplicate, below %deduplication-minimum-size"):
New test.
("deduplicate", "deduplicate, ENOSPC"): Produce files bigger than
%DEDUPLICATION-MINIMUM-SIZE.
* tests/store.scm ("substitute, deduplication"): Likewise.
Without this invoking ‘make clean’ would remove ‘guix-gc.timer’, and ‘make’
would fail with.
make[2]: *** No rule to make target 'etc/guix-gc.timer', needed by 'all-am'. Stop.
* nix/local.mk (nodist_systemdservice_DATA): Remove ‘guix-gc.timer’.
This avoids the situation where error messages would unintentionally go
to stderr and be wrongfully interpreted as a reply by the daemon.
Fixes <https://bugs.gnu.org/46362>.
This is a followup to ee3226e9d5.
* guix/scripts/substitute.scm (display-narinfo-data): Add 'port'
parameter and honor it.
(process-query): Likewise.
(process-substitution): Likewise.
(%error-to-file-descriptor-4?, with-redirected-error-port): Remove.
(%reply-file-descriptor): New variable.
(guix-substitute): Remove use of 'with-redirected-error-port'. Define
'reply-port' and pass it to 'process-query' and 'process-substitution'.
* nix/libstore/build.cc (SubstitutionGoal::handleChildOutput): Swap
'builderOut' and 'fromAgent'.
* nix/libstore/local-store.cc (LocalStore::getLineFromSubstituter):
Likewise.
* tests/substitute.scm <top level>: Set '%reply-file-descriptor'
rather than '%error-to-file-descriptor-4?'.
Fixes <https://bugs.gnu.org/47229>.
Reported by Nathan Nye of WhiteBeam Security.
* nix/libstore/build.cc (DerivationGoal::startBuilder): When 'useChroot'
is true, add "/top" to 'tmpDir'.
(DerivationGoal::deleteTmpDir): Adjust accordingly. When
'settings.keepFailed' is true, chown in two steps: first the "/top"
sub-directory, and then rename "/top" to its parent.
Previously, we'd get:
$ guix-daemon --discover
error: basic_string::_M_construct null not valid
* nix/nix-daemon/guix-daemon.cc (parse_opt): Change second argument to
'settings.set' to properly handle case where ARG is NULL.
This removes the main source of latency between subsequent downloads.
* nix/libstore/build.cc (SubstitutionGoal::tryToRun): Add a
"deduplicate" key to ENV.
(SubstitutionGoal::finished): Remove call to 'optimisePath'.
* guix/scripts/substitute.scm (process-substitution)[destination-in-store?]
[dump-file/deduplicate*]: New variables.
Pass #:dump-file to 'restore-file'.
* guix/scripts/substitute.scm (guix-substitute)[deduplicate?]: New
variable.
Pass #:deduplicate? to 'process-substitution'.
* guix/serialization.scm (dump-file): Export and augment 'dump-file'.
'guix substitute' now takes care of it via 'restore-file'.
* nix/libstore/build.cc (SubstitutionGoal::finished): Remove call to
'canonicalisePathMetaData'.
This way, the hash of the store item can be computed as it is restored,
thereby avoiding an additional file tree traversal ('hashPath' call)
later on in the daemon. Consequently, it should reduce latency between
subsequent substitute downloads.
This is a followup to 5ff521452b.
* guix/scripts/substitute.scm (narinfo-hash-algorithm+value): New
procedure.
(process-substitution): Wrap INPUT into a hash input port, 'hashed', and
read from it. Compare the actual and expected hashes, and print a
"hash-mismatch" status line when they differ. When they match, print
not just "success" but also the nar hash and size.
* nix/libstore/build.cc (class SubstitutionGoal)[expectedHashStr]:
Remove.
(SubstitutionGoal::finished): Tokenize 'status'. Parse it and handle
"success" and "hash-mismatch" accordingly. Call 'hashPath' only when
the returned hash is not SHA256.
(SubstitutionGoal::handleChildOutput): Remove 'expectedHashStr'
handling.
* tests/substitute.scm ("substitute, invalid hash"): Rename to...
("substitute, invalid narinfo hash"): ... this.
("substitute, invalid hash"): New test.
It was already impossible in practice for 'expectedHashStr' to be empty
if 'status' == "success".
* nix/libstore/build.cc (SubstitutionGoal::finished): Throw 'SubstError'
when 'expectedHashStr' is empty.
That way, when fetching a series of substitutes from the same server(s),
the connection is reused instead of being closed/opened for each
substitutes, which saves on network round trips and TLS handshakes.
* guix/http-client.scm (http-fetch): Add #:keep-alive? and honor it.
* guix/progress.scm (progress-report-port): Add #:close? parameter and
honor it.
* guix/scripts/substitute.scm (at-most): Return the tail as a second
value.
(fetch): Add #:port and #:keep-alive? and honor them.
(%max-cached-connections): New variable.
(open-connection-for-uri/cached, call-with-cached-connection): New
procedures.
(with-cached-connection): New macro.
(process-substitution): Wrap 'fetch' call in 'with-cached-connection'.
Pass #:close? to 'progress-report-port'.
This avoids spawning one substitute process per substitution.
* nix/libstore/build.cc (class Worker)[substituter]: New field.
[outPipe, logPipe, pid]: Remove.
(class SubstitutionGoal)[expectedHashStr, status, substituter]: New fields.
(SubstitutionGoal::timedOut): Adjust to check 'substituter'.
(SubstitutionGoal::tryToRun): Remove references to 'outPipe' and
'logPipe'. Run "guix substitute --substitute" as an 'Agent'. Send the
request with 'writeLine'.
(SubstitutionGoal::finished): Likewise.
(SubstitutionGoal::handleChildOutput): Change to fill in
'expectedHashStr' and 'status'.
(SubstitutionGoal::handleEOF): Call 'wakeUp' unconditionally.
(SubstitutionGoal::~SubstitutionGoal): Adjust to check 'substituter'.
* guix/scripts/substitute.scm (process-substitution): Write "success\n"
to stdout upon success.
(%error-to-file-descriptor-4?): New variable.
(guix-substitute): Set 'current-error-port' to file descriptor 4
unless (%error-to-file-descriptor-4?) is false.
Remove "--substitute" arguments. Loop reading line from stdin.
* tests/substitute.scm <top level>: Call '%error-to-file-descriptor-4?'.
(request-substitution): New procedure.
("substitute, no signature")
("substitute, invalid hash")
("substitute, unauthorized key")
("substitute, authorized key")
("substitute, unauthorized narinfo comes first")
("substitute, unsigned narinfo comes first")
("substitute, first narinfo is unsigned and has wrong hash")
("substitute, first narinfo is unsigned and has wrong refs")
("substitute, two invalid narinfos")
("substitute, narinfo with several URLs"): Adjust to new "guix
substitute --substitute" calling convention.
* nix/libstore/local-store.hh (class LocalStore)[substituter]: New
method.
[runningSubstituter]: Turn into a shared_ptr.
* nix/libstore/local-store.cc (LocalStore::querySubstitutablePaths):
Call 'substituter' instead of using inline code.
(LocalStore::querySubstitutablePathInfos): Likewise.
(LocalStore::substituter): New method.
* nix/libstore/local-store.hh (RunningSubstituter): Remove.
(LocalStore)[runningSubstituter]: Change to unique_ptr<Agent>.
[setSubstituterEnv, didSetSubstituterEnv]: Remove.
[getLineFromSubstituter, getIntLineFromSubstituter]: Take an 'Agent'.
* nix/libstore/local-store.cc (LocalStore::~LocalStore): Remove
reference to 'runningSubstituter'.
(LocalStore::setSubstituterEnv, LocalStore::startSubstituter): Remove.
(LocalStore::getLineFromSubstituter): Adjust to 'run' being an 'Agent'.
(LocalStore::querySubstitutablePaths): Spawn substituter agent if
needed. Adjust to 'Agent' interface.
(LocalStore::querySubstitutablePathInfos): Likewise.
* nix/libstore/build.cc (SubstitutionGoal::tryToRun): Remove call to
'setSubstituterEnv' and add 'setenv' call for "_NIX_OPTIONS" instead.
(SubstitutionGoal::finished): Remove 'readLine' call for 'dummy'.
* guix/scripts/substitute.scm (%allow-unauthenticated-substitutes?):
Remove second argument to 'make-parameter'.
(process-query): Call 'warn-about-missing-authentication'
when (%allow-unauthenticated-substitutes?) is #t.
(guix-substitute): Wrap body in 'parameterize'. Set 'guix-warning-port'
too. No longer exit when 'substitute-urls' returns the empty list. No
longer print newline initially.
* tests/substitute.scm (test-quit): Parameterize 'current-error-port' to
account for the port changes in 'guix-substitute'.
