Previously, test cases could fail if some process was listening
at a hard-coded port. This patch eliminates most of these potential
failures, by automatically assigning an unbound port. This should
allow for building multiple guix trees in parallel outside a build
container, though this is currently untested.
The test "home-page: Connection refused" in tests/lint.scm still
hardcodes port 9999, however.
* guix/tests/http.scm
(http-server-can-listen?): remove now unused procedure.
(%http-server-port): default to port 0, meaning the OS
will automatically choose a port.
(open-http-server-socket): remove the false statement claiming
this procedure is exported and also return the allocated port
number.
(%local-url): raise an error if the port is obviously unbound.
(call-with-http-server): set %http-server-port to the allocated
port while the thunk is called.
* tests/derivations.scm: adjust test cases to use automatically
assign a port. As there is no risk of a port conflict now,
do not make any tests conditional upon 'http-server-can-listen?'
anymore.
* tests/elpa.scm: likewise.
* tests/lint.scm: likewise, and add a TODO comment about a port
that is still hard-coded.
* tests/texlive.scm: likewise.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
This reverts commit b1248016e0, which was
a mistake: the two '%derivation-cache' are two different things. It
broke tests that use 'call-with-external-store'.
This means there's a module for working with substitutes, rather than all the
code sitting in the script. The need for this can be seen with the weather and
challenge scripts, that now don't have to use code from the substitute script,
but can instead use the substitute module.
The separation here between the actual functionality of the substitute script
and the underlying functionality used both there and elsewhere should make
maintenance easier moving forward.
This commit just moves code, none of the code should have been changed
significantly.
* guix/scripts/substitute.scm (%narinfo-cache-directory, %narinfo-ttl,
%narinfo-negative-ttl, %narinfo-transient-error-ttl, %unreachable-hosts): Move
variables to guix/substitutes.scm.
(narinfo-cache-file, cached-narinfo, cache-narinfo!, narinfo-request,
read-to-eof, call-with-connection-error-handling, fetch-narinfos,
lookup-narinfos, lookup-narinfos/diverse): Move procedures to
guix/substitutes.scm.
* guix/substitutes.scm: New file.
* Makefile.am: Add it.
* guix/narinfo.scm: Remove redundant module.
* guix/scripts/challenge.scm: Change (guix scripts substitute) to (guix
substitutes).
* guix/scripts/weather.scm: Change (guix scripts substitute) to (guix
substitutes).
Previously, transformations applied from a manifest (rather than via
"guix install") would be lost. This change fixes that and simplifies
things.
Reported by zimoun at
<https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00153.html>.
* guix/profiles.scm (default-properties): New procedure.
(package->manifest-entry): Use it for #:properties.
* guix/scripts/pack.scm (guix-pack)[with-transformations]: Remove.
Remove caller.
* guix/scripts/package.scm (transaction-upgrade-entry): Remove calls to
'manifest-entry-with-transformations'.
* tests/guix-package.sh: Add test.
* tests/transformations.scm ("options->transformation + package->manifest-entry"):
New test.
Fixes a regression introduced in
c6d6aee6659acb293eb33f498fdac3b47a19a48, where #:reference-graphs would
end up referring to native inputs.
This would notably break the compilation of systems using a childhurd,
because they would attempt to build the 'hurd' package natively.
* guix/gexp.scm (lower-reference-graphs)[tuple->gexp-input]: Honor TARGET.
* tests/gexp.scm ("gexp->derivation #:references-graphs cross-compilation"):
New test.
This fixes a bug whereby the Guix importer considers files like .f.* to be
Fortran files.
The expression "\\.f(90|95)?" would match a lot of files containing ".f"
although they are not Fortran files. Instead we should only consider files
with this *suffix*.
* guix/import/cran.scm (directory-needs-fortran?): Only check for suffixes.
Co-authored-by: Ricardo Wurmus <rekado@elephly.net>
This reduces allocations and bit twiddling in the loop.
* guix/serialization.scm (write-literal-strings): New macro.
(write-file-tree): Use it in lieu of 'write-string' calls where applicable.
* guix/build/syscalls.scm (<mount>): New record type.
(option-string->mount-flags, mount-flags)
(octal-decode, mounts): New procedures.
(mount-points): Rewrite in terms of 'mount'.
* tests/syscalls.scm ("mounts"): New test.
Fixes <https://bugs.gnu.org/46362>.
Reported by Zhu Zihao <all_but_last@163.com>.
Previously, backtraces due to uncaught exceptions would always to go
file descriptor 2; the daemon would read it and error out with something
like:
error: got unexpected path `Backtrace:' from substituter
This patch fixes that by ensuring backtraces are properly displayed on
file descriptor 4.
* guix/scripts/substitute.scm (with-redirected-error-port): New macro.
(guix-substitute): Use 'with-redirected-error-port' instead of 'parameterize'.
This prevents generated launchers and desktop files from inadvertently
crashing if the directory name contains a space.
* gnu/build/renpy-build-system.scm (install, install-desktop-file): Use ~s
to format data directory.
This avoids a closure allocation when 'lookup-cached-object' is called.
* guix/store.scm (lookup-cached-object): Avoid optional/keyword
arguments and inline.
(%mcached): Adjust accordingly.
This reduces the total amount of memory allocated by 8% when running
"guix build qemu -d --no-grafts".
* guix/gexp.scm (fold/tree): New procedure.
(gexp-inputs)[interesting?]: New procedure.
[add-reference-inputs]: Change (lst ...) clause to (? pair? lst), and
use 'fold/tree' to recurse into it.
(gexp-inputs)[add-reference-output]: Likewise, and remove
plain (lst ...) clause.
Call 'fold'.
(gexp->sexp)[reference->sexp]: In the list case, avoid boxing and
recursive call when the object has a plain non-aggregate type.
This avoids double traversal of references and extra bookkeeping,
thereby further reducing memory allocations.
* guix/gexp.scm (lower-gexp): Include only one call to 'lower-inputs'.
(gexp-inputs): Remove #:native? parameter.
[set-gexp-input-native?]: New procedure.
[add-reference-inputs]: Use it.
(gexp-native-inputs): Remove.
* tests/gexp.scm (gexp-native-inputs): Remove.
(gexp-input->tuple): Include 'gexp-input-native?'.
("let-system")
("let-system, nested")
("ungexp + ungexp-native")
("ungexp + ungexp-native, nested")
("ungexp + ungexp-native, nested, special mixture")
("input list")
("input list + ungexp-native")
("input list splicing")
("input list splicing + ungexp-native-splicing")
("gexp list splicing + ungexp-splicing"): Adjust accordingly.
This slightly reduces memory allocation.
* guix/gexp.scm (lower-inputs): Expect a list of <gexp-input> rather
than a list of tuples.
(lower-reference-graphs)[tuple->gexp-input]: New procedure.
Use it.
(gexp-inputs): Return a list of <gexp-input> rather than a list of
tuples.
* tests/gexp.scm (gexp-input->tuple): New procedure.
("one input package")
("one input package, dotted list")
("one input origin")
("one local file")
("one local file, symlink")
("one plain file")
("two input packages, one derivation, one file")
("file-append")
("file-append, output")
("file-append, nested")
("let-system")
("let-system, nested")
("ungexp + ungexp-native")
("ungexp + ungexp-native, nested")
("ungexp + ungexp-native, nested, special mixture")
("input list")
("input list + ungexp-native")
("input list splicing")
("input list splicing + ungexp-native-splicing")
("gexp list splicing + ungexp-splicing"): Adjust accordingly.
Previously, 'guix install foo -r bar' would crash with a backtrace
because NAME would be #\r (a character instead of a string).
* guix/scripts.scm (parse-command-line)[parse-options-from]: Call
'option-hint' only when NAME is a string.
This is part of trying to reduce the interdependency of code within the
substitute module.
This commit addresses some of the error handling that was performed through
open-connection-for-uri/maybe. The new approach is to use
call-with-connection-error-handling, and wrap calls to http-multiple-get and
http-fetch with that procedure, which takes care of handling connection
errors.
I think this is even slightly more rigerous than the previous setup, because
this approach handles connection errors that occur when http-multiple-get
reconnects to a host.
* guix/scripts/substitute.scm (open-connection-for-uri/maybe): Transform in to
call-with-connection-error-handling.
(fetch-narinfos): Use call-with-connection-error-handling.
(process-query): Replace open-connection-for-uri/maybe with
open-connection-for-uri/cached.
(open-connection-for-uri/cached): Set a default timeout, matching the
behaviour in open-connection-for-uri/maybe.
(process-substitution): Use call-with-connection-error-handling.
At least by default. Instead, make the open-connection procedure a parameter,
and make the default guix:open-connection-for-uri. Do so similarly for
lookup-narinfos and lookup-narinfos/diverse which work towards calling
fetch-narinfos.
This means this code can be moved to a different module, without having
use/move the connection caching code.
* guix/scripts/substitute.scm (fetch-narinfos): Add #:open-connection
argument, and call http-multiple-get with it.
(lookup-narinfos) Add #:open-connection argument, and call fetch-narinfos with
it.
(lookup-narinfos/diverse): Add #:open-connection argument, and call
lookup-narinfos with it.
(process-query): Call lookup-narinfos/diverse with #:open-connection
open-connection-for-uri/maybe.
As it's only called in one place, and this should make the code easier to
read.
* guix/scripts/substitute.scm (fetch): Move procedure inside…
(process-substitution): …here.
It's just called in one place, with hardcoded argument values, so just inline
them.
* guix/scripts/substitute.scm (fetch): Remove arguments that don't vary, copy
the values from the call site in process-substitution.
(process-substitution): Remove unnecessary argument values from fetch call.
Failures now should be handled where they occur, and if there's a problem
that's symptomatic of an issue with the connection, the port should be closed.
* guix/scripts/substitute.scm (call-with-cached-connection): Remove procedure.
(with-cached-connection): Remove syntax rule.
Just pass open-connection-for-uri/maybe to http-fetch, this removes the need
for with-cached-connection and passing the port in.
* guix/scripts/substitute.scm (fetch): Don't take a port as an argument, and
pass open-connection-for-uri/maybe to http-fetch.
(process-substitution): Don't call fetch with with-cached-connection.
So that an alternative procedure can be passed in, perhaps to perform
connection caching.
* guix/http-client.scm (http-fetch): Add an #:open-connection keyword
argument.
Instead, just pass open-connection-for-uri/maybe to http-multiple-get. This
code should be functionaly similar to the previous code. The eventual aim of
this is to make the connection caching not mandatory in fetch-narinfos.
* guix/scripts/substitute.scm (fetch-narinfos): Remove use of
call-with-cached-connection.
As this is used by http-fetch and http-multiple-get when they call the
specified open connection procedure.
* guix/scripts/substitute.scm (open-connection-for-uri/maybe): Support
#:verify-certificate?.
Making sure to close the port if it looks to be unusable. This closing of the
port will allow for caching connections, without caching broken connections,
as the cache can avoid handing out closed ports.
* guix/http-client.scm (http-multiple-get): Try to catch exceptions that
happen if the port is unusable, this is a adaptation of code within the (guix
scripts substitute) module.
