Fixes <https://bugs.gnu.org/39090>.
Reported by Gábor Boskovits <boskovits@gmail.com>.
Fixes a regression introduced in 9e3f9ac3c0.
* guix/scripts/substitute.scm (http-multiple-get): In the "Connection: close"
case, pass (drop requests (+ 1 processed)) to 'loop' as the remaining
REQUESTS value.
Previously, we would pass a list containing duplicates, and thus the
final result would also contain duplicates. When sent to the daemon,
that list would lead to a daemon error:
got unexpected path `/gnu/store/…' from substituter
Fixes <https://bugs.gnu.org/20145>.
* guix/build/download.scm (%tls-ports, register-tls-record-port): Remove.
(tls-wrap): Remove call to 'register-tls-record-port'. Return a custom
binary input/output port instead. This is a backport of what Guile
2.2's (web client) module has been doing.
(close-connection): Define as an alias for 'close-port'.
* guix/http-client.scm (http-fetch): Remove #:keep-alive? parameter,
which was ignored and unused.
Pass #:keep-alive? #f to 'http-get'.
* guix/lint.scm (probe-uri): Use 'close-port' instead of 'close-connection'.
* guix/scripts/substitute.scm (http-multiple-get): Likewise.
Previously, in the unlikely case 'http-multiple-get' was passed more
than 1,000 requests, it could have dropped all those above 1,000.
* guix/scripts/substitute.scm (http-multiple-get): Define 'batch'. Use
that for the 'write-request' loop. Add 'processed' parameter to 'loop'
and use that to compute the remaining requests and call 'connect' in the
recursion base case.
* guix/scripts/substitute.scm (warn-about-missing-authentication): New
procedure.
(%allow-unauthenticated-substitutes?): Turn into a public parameter and
use 'warn-about-missing-authentication'.
(valid-narinfo?): Adjust accordingly.
* tests/substitute.scm (call-with-narinfo): Likewise.
This avoids one GET request every time 'fetch-narinfos' is called.
The file itself was essentially useless.
* guix/scripts/substitute.scm (<cache-info>, download-cache-info):
Remove.
(%unreachable-hosts): New variable.
(open-connection-for-uri/maybe): New procedure.
(fetch-narinfos)[handle-narinfo-response]: Check whether NARINFO has its
'path' under (%store-prefix) and ignore it otherwise. Move
'update-progress!' call before 'if'.
[do-fetch]: Remove 'port' parameter. Use
'open-connection-for-uri/maybe'.
Remove call to 'download-cache-info'.
When a server publishes several URLs with different compression methods,
'guix substitute' can now choose the best one among the compression
methods that it supports.
* guix/scripts/substitute.scm (<narinfo>)[uri]: Replace with...
[uris]: ... this.
[compression]: Replace with...
[compressions]: ... this.
[file-size]: Replace with...
[file-sizes]: ... this.
[file-hash]: Replace with...
[file-hashes]: ... this.
(narinfo-maker): Adjust accordingly. Ensure 'file-sizes' and
'file-hashes' have the right length.
(assert-valid-signature, valid-narinfo?): Use the first element of
'narinfo-uris' in error messages.
(read-narinfo): Expect "URL", "Compression", "FileSize", and "FileHash"
to occur multiple times.
(display-narinfo-data): Call 'select-uri' to determine the file size.
(%compression-methods): New variable.
(supported-compression?, compresses-better?, select-uri): New
procedures.
(process-substitution): Call 'select-uri' to select the URI and
compression.
* guix/scripts/weather.scm (report-server-coverage): Account for all the
values returned by 'narinfo-file-sizes'.
* tests/substitute.scm ("substitute, narinfo with several URLs"): New
test.
Fixes <https://bugs.gnu.org/33733>.
Fixes a bug whereby 'guix substitute' would accept narinfos whose
signature does not cover the StorePath/NarHash/References tuple.
* guix/scripts/substitute.scm (narinfo-sha256)[%mandatory-fields]: New
variable.
Compute SIGNED-FIELDS; return #f unless each of the %MANDATORY-FIELDS
is among SIGNED-FIELDS.
* tests/substitute.scm ("query narinfo with signature over nothing")
("query narinfo with signature over irrelevant bits"): New tests.
* guix/scripts/container.scm (guix-container):
* guix/scripts/import.scm (guix-import):
* guix/scripts/substitute.scm (guix-substitute): Add -V as the short option
for --version to match show-help's description.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Fixes <https://bugs.gnu.org/28731>.
Reported by Jan Nieuwenhuizen <janneke@gnu.org>.
* guix/scripts/substitute.scm (at-most): New procedure.
(http-multiple-get): Use it to send at most 1000 requests at once.
Among other things, this removes (guix utils), (guix ui), (guix config),
etc. from the closure of (guix build download), as was the case since
798648515b.
* guix/utils.scm (<progress-reporter>, call-with-progress-reporter):
Move to...
* guix/progress.scm: ... here. New file.
* Makefile.am (MODULES): Add it.
* guix/build/download.scm (current-terminal-columns)
(nearest-exact-integer, duration->seconds, seconds->string)
(byte-count->string, progress-bar, string-pad-middle)
(rate-limited, progress-reporter/file, dump-port*)
(time-monotonic): Move to progress.scm.
* guix/scripts/download.scm: Adjust accordingly.
* guix/scripts/substitute.scm: Likewise.
Fixes <https://bugs.gnu.org/28756>.
* guix/utils.scm (filtered-port): Close the 'input' port for the current
process, and close it upon exit in the child process.
* guix/scripts/substitute.scm (progress-substitution): Display "\n\n" after
the reporter has finished.
* guix/utils.scm (<progress-reporter>): New record type.
(call-with-progress-reporter): New procedure.
* guix/build/download.scm (dump-port*, rate-limited, progress-reporter/file):
New procedures.
(ftp-fetch, http-fetch): Use 'dump-port*'.
(progress-proc): Remove procedure.
* guix/scripts/substitute.scm (progress-report-port): Rewrite in terms of
<progress-reporter>.
(process-substitution): Adjust accordingly.
This allows substitutes to be downloaded from unauthorized servers, as
long as they advertise the same hash and references as one of the
authorized servers.
* guix/scripts/substitute.scm (assert-valid-narinfo): Remove.
(valid-narinfo?): Add #:verbose?. Handle each case of
'signature-case'.
(equivalent-narinfo?): New procedure.
(lookup-narinfos/diverse): Add 'authorized?' parameter and honor it.
[select-hit]: New procedure.
(lookup-narinfo): Add 'authorized?' parameter and pass it.
(process-query): Adjust callers accordingly.
(process-substitution): Remove call to 'assert-valid-narinfo'. Check
whether 'lookup-narinfo' returns true and call 'leave' if not.
* tests/substitute.scm (%main-substitute-directory)
(%alternate-substitute-directory): New variables.
(call-with-narinfo): Make 'narinfo-directory' a parameter. Call
'mkdir-p' to create it. Change unwind handler to check whether
CACHE-DIRECTORY exists before deleting it.
(with-narinfo*): New macro.
("substitute, no signature")
("substitute, invalid hash")
("substitute, unauthorized key"): Change expected error message to "no
valid substitute".
("substitute, unauthorized narinfo comes first")
("substitute, unsigned narinfo comes first")
("substitute, first narinfo is unsigned and has wrong hash")
("substitute, first narinfo is unsigned and has wrong refs")
("substitute, unsigned narinfo comes first")
("substitute, two invalid narinfos"): New tests.
* doc/guix.texi (Substitutes): Explain the new behavior.
* guix/scripts/substitute.scm (%cache-urls): Rename to...
(%default-substitute-urls): ... this.
(substitute-urls): New variable.
(guix-substitute): Use it instead of %CACHE-URLS.
* tests/substitute.scm: Likewise.
Previously this operation was linear in the number of requests and
involved costly calls to 'string-contains'.
* guix/scripts/substitute.scm (fetch-narinfos)[hash-part->path]: New
procedure.
[handle-narinfo-response]: Use it for caching when CODE is not 200.
Prior to this change, half of the cached narinfos would expire
immediately since they contained the number of nanoseconds instead of
the number of seconds as their date.
* guix/scripts/substitute.scm (time-monotonic) <guile-2.2>: Define, as a
workaround.
This allows 'guix publish' threads as well as 'guix substitute' and
'guix offload' processes to be properly labeled in 'top', 'pstree', etc.
* guix/workers.scm (worker-thunk): Add #:thread-name parameter and honor it.
(make-pool): Likewise.
* guix/scripts/publish.scm (http-write): Add calls to 'set-thread-name'
in bodies of 'call-with-new-thread'.
(guix-publish): Call 'set-thread-name'. Pass #:thread-name to 'make-pool'.
* guix/scripts/offload.scm (guix-offload): Call 'set-thread-name'.
* guix/scripts/substitute.scm (guix-substitute): Likewise.
* guix/scripts/substitute.scm (cached-narinfo): When VALUE is #f, use
the TTL that is read instead of %NARINFO-NEGATIVE-TTL.
(cached-narinfo-expiration-time): Likewise.
* guix/cache.scm, tests/cache.scm: New files.
* Makefile.am (MODULES, SCM_TESTS): Add them.
* guix/scripts/substitute.scm (obsolete?): Remove.
(remove-expired-cached-narinfos): Rename to...
(cached-narinfo-expiration-time): ... this. Remove the removal part and
only keep the expiration time part.
(narinfo-cache-directories): Add optional 'directory' parameter and
honor it.
(maybe-remove-expired-cached-narinfo): Remove.
(cached-narinfo-files): New procedure.
(guix-substitute): Use 'maybe-remove-expired-cache-entries' instead of
'maybe-remove-expired-cached-narinfo'.
* guix/build/download.scm (open-connection-for-uri): Add note about
same-named binding in Guile 2.2.0.
* guix/http-client.scm: Use 'guix:open-connection-for-uri' for the
procedure coming from (guix build download).
* guix/scripts/lint.scm: Likewise.
* guix/scripts/substitute.scm: Likewise.
Fixes a regression introduced in
9e4e431e04 as a consequence of
bc3c41ce36.
Reported by Marius Bakke <mbakke@fastmail.com>.
* guix/scripts/substitute.scm (fetch): Pass #:verify-certificate? #f to
'open-connection-for-uri' and 'http-fetch'.
(download-cache-info): Likewise.
(http-multiple-get): Add #:verify-certificate? and honor it.
(fetch-narinfos): Pass #:verify-certificate? #f.
'guix substitute' now displays 2 lines per download instead of 4.
Suggested by Ricardo Wurmus.
* guix/scripts/substitute.scm (assert-valid-narinfo): Have #:verbose?
default to #f. Remove leading newline in message.
(process-substitution): Display the URI rather than the store file name.
Display two newlines after the substitution.
Previously, using something like
"--substitute-urls=http://example.org///" would lead to a
'cache-narinfo!' call with #f as its second argument.
It would also do the wrong thing for URLs with a non-empty initial path
component, such as "http://example.org/foo/bar".
* guix/scripts/substitute.scm (fetch-narinfos)[handle-narinfo-response]:
Add call to 'basename' for PATH.
