That way, when fetching a series of substitutes from the same server(s),
the connection is reused instead of being closed/opened for each
substitutes, which saves on network round trips and TLS handshakes.
* guix/http-client.scm (http-fetch): Add #:keep-alive? and honor it.
* guix/progress.scm (progress-report-port): Add #:close? parameter and
honor it.
* guix/scripts/substitute.scm (at-most): Return the tail as a second
value.
(fetch): Add #:port and #:keep-alive? and honor them.
(%max-cached-connections): New variable.
(open-connection-for-uri/cached, call-with-cached-connection): New
procedures.
(with-cached-connection): New macro.
(process-substitution): Wrap 'fetch' call in 'with-cached-connection'.
Pass #:close? to 'progress-report-port'.
This avoids spawning one substitute process per substitution.
* nix/libstore/build.cc (class Worker)[substituter]: New field.
[outPipe, logPipe, pid]: Remove.
(class SubstitutionGoal)[expectedHashStr, status, substituter]: New fields.
(SubstitutionGoal::timedOut): Adjust to check 'substituter'.
(SubstitutionGoal::tryToRun): Remove references to 'outPipe' and
'logPipe'. Run "guix substitute --substitute" as an 'Agent'. Send the
request with 'writeLine'.
(SubstitutionGoal::finished): Likewise.
(SubstitutionGoal::handleChildOutput): Change to fill in
'expectedHashStr' and 'status'.
(SubstitutionGoal::handleEOF): Call 'wakeUp' unconditionally.
(SubstitutionGoal::~SubstitutionGoal): Adjust to check 'substituter'.
* guix/scripts/substitute.scm (process-substitution): Write "success\n"
to stdout upon success.
(%error-to-file-descriptor-4?): New variable.
(guix-substitute): Set 'current-error-port' to file descriptor 4
unless (%error-to-file-descriptor-4?) is false.
Remove "--substitute" arguments. Loop reading line from stdin.
* tests/substitute.scm <top level>: Call '%error-to-file-descriptor-4?'.
(request-substitution): New procedure.
("substitute, no signature")
("substitute, invalid hash")
("substitute, unauthorized key")
("substitute, authorized key")
("substitute, unauthorized narinfo comes first")
("substitute, unsigned narinfo comes first")
("substitute, first narinfo is unsigned and has wrong hash")
("substitute, first narinfo is unsigned and has wrong refs")
("substitute, two invalid narinfos")
("substitute, narinfo with several URLs"): Adjust to new "guix
substitute --substitute" calling convention.
* nix/libstore/local-store.hh (RunningSubstituter): Remove.
(LocalStore)[runningSubstituter]: Change to unique_ptr<Agent>.
[setSubstituterEnv, didSetSubstituterEnv]: Remove.
[getLineFromSubstituter, getIntLineFromSubstituter]: Take an 'Agent'.
* nix/libstore/local-store.cc (LocalStore::~LocalStore): Remove
reference to 'runningSubstituter'.
(LocalStore::setSubstituterEnv, LocalStore::startSubstituter): Remove.
(LocalStore::getLineFromSubstituter): Adjust to 'run' being an 'Agent'.
(LocalStore::querySubstitutablePaths): Spawn substituter agent if
needed. Adjust to 'Agent' interface.
(LocalStore::querySubstitutablePathInfos): Likewise.
* nix/libstore/build.cc (SubstitutionGoal::tryToRun): Remove call to
'setSubstituterEnv' and add 'setenv' call for "_NIX_OPTIONS" instead.
(SubstitutionGoal::finished): Remove 'readLine' call for 'dummy'.
* guix/scripts/substitute.scm (%allow-unauthenticated-substitutes?):
Remove second argument to 'make-parameter'.
(process-query): Call 'warn-about-missing-authentication'
when (%allow-unauthenticated-substitutes?) is #t.
(guix-substitute): Wrap body in 'parameterize'. Set 'guix-warning-port'
too. No longer exit when 'substitute-urls' returns the empty list. No
longer print newline initially.
* tests/substitute.scm (test-quit): Parameterize 'current-error-port' to
account for the port changes in 'guix-substitute'.
* guix/scripts/substitute.scm (fetch-narinfos)
[handle-narinfo-response]: Add code 202 to the condition. This will
allow in the future the use of 404 for nar requests which cannot be
built in the present, leaving for the 202 to indicate that a request has
triggered a nar generation that might be completed later, as specified
in <https://www.ietf.org/rfc/rfc2616.txt>.
Fixes <https://bugs.gnu.org/43039>.
Reported by Adam Griffiths <adam.lw.griffiths@gmail.com>.
Previously, a client could lead 'guix substitute' to run in a non-UTF-8
locale, such as the "C" locale. Consequently, 'guix substitute' would
now fail to decode UTF-8 file names such as those that appear in the
'nss-certs' package.
* guix/scripts/substitute.scm (guix-substitute): Set LC_MESSAGES, not LC_ALL.
* Makefile.am (MODULES): Remove guix/zlib.scm and guix/lzlib.scm,
(SCM_TESTS): remove tests/zlib.scm, tests/lzlib.scm.
* build-aux/build-self.scm (make-config.scm): Remove unused %libz variable.
* configure.ac: Remove LIBZ and LIBLZ variables and check instead for
Guile-zlib and Guile-lzlib.
* doc/guix.texi ("Requirements"): Remove zlib requirement and add Guile-zlib
and Guile-lzlib instead.
* gnu/packages/package-management.scm (guix)[native-inputs]: Add "guile-zlib"
and "guile-lzlib",
[inputs]: remove "zlib" and "lzlib",
[propagated-inputs]: ditto,
[arguments]: add "guile-zlib" and "guile-lzlib" to Guile load path.
* guix/config.scm.in (%libz, %liblz): Remove them.
* guix/lzlib.scm: Remove it.
* guix/man-db.scm: Use (zlib) instead of (guix zlib).
* guix/profiles.scm (manual-database): Do not stub (guix config) in imported
modules list, instead add "guile-zlib" to the extension list.
* guix/scripts/publish.scm: Use (zlib) instead of (guix zlib) and (lzlib)
instead of (guix lzlib),
(string->compression-type, effective-compression): do not check for zlib and
lzlib availability.
* guix/scripts/substitute.scm (%compression-methods): Do not check for lzlib
availability.
* guix/self.scm (specification->package): Add "guile-zlib" and "guile-lzlib"
and remove "zlib" and "lzlib",
(compiled-guix): remove "zlib" and "lzlib" arguments and add guile-zlib and
guile-lzlib to the dependencies, also do not pass "zlib" and "lzlib" to
"make-config.scm" procedure,
(make-config.scm): remove "zlib" and "lzlib" arguments as well as %libz and
%liblz variables.
* guix/utils.scm (lzip-port): Use (lzlib) instead of (guix lzlib) and do not
check for lzlib availability.
* guix/zlib.scm: Remove it.
* m4/guix.m4 (GUIX_LIBZ_LIBDIR, GUIX_LIBLZ_FILE_NAME): Remove them.
* tests/lzlib.scm: Use (zlib) instead of (guix zlib) and (lzlib)
instead of (guix lzlib), and do not check for zlib and lzlib availability.
* tests/publish.scm: Ditto.
* tests/substitute.scm: Do not check for lzlib availability.
* tests/utils.scm: Ditto.
* tests/zlib.scm: Remove it.
Fixes <https://bugs.gnu.org/39090>.
Reported by Gábor Boskovits <boskovits@gmail.com>.
Fixes a regression introduced in 9e3f9ac3c00906f5bc647ea8398e4ed5a370614e.
* guix/scripts/substitute.scm (http-multiple-get): In the "Connection: close"
case, pass (drop requests (+ 1 processed)) to 'loop' as the remaining
REQUESTS value.
Previously, we would pass a list containing duplicates, and thus the
final result would also contain duplicates. When sent to the daemon,
that list would lead to a daemon error:
got unexpected path `/gnu/store/…' from substituter
Fixes <https://bugs.gnu.org/20145>.
* guix/build/download.scm (%tls-ports, register-tls-record-port): Remove.
(tls-wrap): Remove call to 'register-tls-record-port'. Return a custom
binary input/output port instead. This is a backport of what Guile
2.2's (web client) module has been doing.
(close-connection): Define as an alias for 'close-port'.
* guix/http-client.scm (http-fetch): Remove #:keep-alive? parameter,
which was ignored and unused.
Pass #:keep-alive? #f to 'http-get'.
* guix/lint.scm (probe-uri): Use 'close-port' instead of 'close-connection'.
* guix/scripts/substitute.scm (http-multiple-get): Likewise.
Previously, in the unlikely case 'http-multiple-get' was passed more
than 1,000 requests, it could have dropped all those above 1,000.
* guix/scripts/substitute.scm (http-multiple-get): Define 'batch'. Use
that for the 'write-request' loop. Add 'processed' parameter to 'loop'
and use that to compute the remaining requests and call 'connect' in the
recursion base case.
* guix/scripts/substitute.scm (warn-about-missing-authentication): New
procedure.
(%allow-unauthenticated-substitutes?): Turn into a public parameter and
use 'warn-about-missing-authentication'.
(valid-narinfo?): Adjust accordingly.
* tests/substitute.scm (call-with-narinfo): Likewise.
This avoids one GET request every time 'fetch-narinfos' is called.
The file itself was essentially useless.
* guix/scripts/substitute.scm (<cache-info>, download-cache-info):
Remove.
(%unreachable-hosts): New variable.
(open-connection-for-uri/maybe): New procedure.
(fetch-narinfos)[handle-narinfo-response]: Check whether NARINFO has its
'path' under (%store-prefix) and ignore it otherwise. Move
'update-progress!' call before 'if'.
[do-fetch]: Remove 'port' parameter. Use
'open-connection-for-uri/maybe'.
Remove call to 'download-cache-info'.
When a server publishes several URLs with different compression methods,
'guix substitute' can now choose the best one among the compression
methods that it supports.
* guix/scripts/substitute.scm (<narinfo>)[uri]: Replace with...
[uris]: ... this.
[compression]: Replace with...
[compressions]: ... this.
[file-size]: Replace with...
[file-sizes]: ... this.
[file-hash]: Replace with...
[file-hashes]: ... this.
(narinfo-maker): Adjust accordingly. Ensure 'file-sizes' and
'file-hashes' have the right length.
(assert-valid-signature, valid-narinfo?): Use the first element of
'narinfo-uris' in error messages.
(read-narinfo): Expect "URL", "Compression", "FileSize", and "FileHash"
to occur multiple times.
(display-narinfo-data): Call 'select-uri' to determine the file size.
(%compression-methods): New variable.
(supported-compression?, compresses-better?, select-uri): New
procedures.
(process-substitution): Call 'select-uri' to select the URI and
compression.
* guix/scripts/weather.scm (report-server-coverage): Account for all the
values returned by 'narinfo-file-sizes'.
* tests/substitute.scm ("substitute, narinfo with several URLs"): New
test.
Fixes <https://bugs.gnu.org/33733>.
Fixes a bug whereby 'guix substitute' would accept narinfos whose
signature does not cover the StorePath/NarHash/References tuple.
* guix/scripts/substitute.scm (narinfo-sha256)[%mandatory-fields]: New
variable.
Compute SIGNED-FIELDS; return #f unless each of the %MANDATORY-FIELDS
is among SIGNED-FIELDS.
* tests/substitute.scm ("query narinfo with signature over nothing")
("query narinfo with signature over irrelevant bits"): New tests.
* guix/scripts/container.scm (guix-container):
* guix/scripts/import.scm (guix-import):
* guix/scripts/substitute.scm (guix-substitute): Add -V as the short option
for --version to match show-help's description.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Fixes <https://bugs.gnu.org/28731>.
Reported by Jan Nieuwenhuizen <janneke@gnu.org>.
* guix/scripts/substitute.scm (at-most): New procedure.
(http-multiple-get): Use it to send at most 1000 requests at once.
Among other things, this removes (guix utils), (guix ui), (guix config),
etc. from the closure of (guix build download), as was the case since
798648515b77507c242752457b4dc17c155bad6e.
* guix/utils.scm (<progress-reporter>, call-with-progress-reporter):
Move to...
* guix/progress.scm: ... here. New file.
* Makefile.am (MODULES): Add it.
* guix/build/download.scm (current-terminal-columns)
(nearest-exact-integer, duration->seconds, seconds->string)
(byte-count->string, progress-bar, string-pad-middle)
(rate-limited, progress-reporter/file, dump-port*)
(time-monotonic): Move to progress.scm.
* guix/scripts/download.scm: Adjust accordingly.
* guix/scripts/substitute.scm: Likewise.
Fixes <https://bugs.gnu.org/28756>.
* guix/utils.scm (filtered-port): Close the 'input' port for the current
process, and close it upon exit in the child process.
* guix/scripts/substitute.scm (progress-substitution): Display "\n\n" after
the reporter has finished.
* guix/utils.scm (<progress-reporter>): New record type.
(call-with-progress-reporter): New procedure.
* guix/build/download.scm (dump-port*, rate-limited, progress-reporter/file):
New procedures.
(ftp-fetch, http-fetch): Use 'dump-port*'.
(progress-proc): Remove procedure.
* guix/scripts/substitute.scm (progress-report-port): Rewrite in terms of
<progress-reporter>.
(process-substitution): Adjust accordingly.
This allows substitutes to be downloaded from unauthorized servers, as
long as they advertise the same hash and references as one of the
authorized servers.
* guix/scripts/substitute.scm (assert-valid-narinfo): Remove.
(valid-narinfo?): Add #:verbose?. Handle each case of
'signature-case'.
(equivalent-narinfo?): New procedure.
(lookup-narinfos/diverse): Add 'authorized?' parameter and honor it.
[select-hit]: New procedure.
(lookup-narinfo): Add 'authorized?' parameter and pass it.
(process-query): Adjust callers accordingly.
(process-substitution): Remove call to 'assert-valid-narinfo'. Check
whether 'lookup-narinfo' returns true and call 'leave' if not.
* tests/substitute.scm (%main-substitute-directory)
(%alternate-substitute-directory): New variables.
(call-with-narinfo): Make 'narinfo-directory' a parameter. Call
'mkdir-p' to create it. Change unwind handler to check whether
CACHE-DIRECTORY exists before deleting it.
(with-narinfo*): New macro.
("substitute, no signature")
("substitute, invalid hash")
("substitute, unauthorized key"): Change expected error message to "no
valid substitute".
("substitute, unauthorized narinfo comes first")
("substitute, unsigned narinfo comes first")
("substitute, first narinfo is unsigned and has wrong hash")
("substitute, first narinfo is unsigned and has wrong refs")
("substitute, unsigned narinfo comes first")
("substitute, two invalid narinfos"): New tests.
* doc/guix.texi (Substitutes): Explain the new behavior.
* guix/scripts/substitute.scm (%cache-urls): Rename to...
(%default-substitute-urls): ... this.
(substitute-urls): New variable.
(guix-substitute): Use it instead of %CACHE-URLS.
* tests/substitute.scm: Likewise.
Previously this operation was linear in the number of requests and
involved costly calls to 'string-contains'.
* guix/scripts/substitute.scm (fetch-narinfos)[hash-part->path]: New
procedure.
[handle-narinfo-response]: Use it for caching when CODE is not 200.
Prior to this change, half of the cached narinfos would expire
immediately since they contained the number of nanoseconds instead of
the number of seconds as their date.
* guix/scripts/substitute.scm (time-monotonic) <guile-2.2>: Define, as a
workaround.
This allows 'guix publish' threads as well as 'guix substitute' and
'guix offload' processes to be properly labeled in 'top', 'pstree', etc.
* guix/workers.scm (worker-thunk): Add #:thread-name parameter and honor it.
(make-pool): Likewise.
* guix/scripts/publish.scm (http-write): Add calls to 'set-thread-name'
in bodies of 'call-with-new-thread'.
(guix-publish): Call 'set-thread-name'. Pass #:thread-name to 'make-pool'.
* guix/scripts/offload.scm (guix-offload): Call 'set-thread-name'.
* guix/scripts/substitute.scm (guix-substitute): Likewise.
