Commit Graph

33 Commits

Author SHA1 Message Date
Ludovic Courtès
0793833c59
database: Remove #:deduplicate? from 'register-items'.
It is now up to the caller to deduplicate store contents.

* guix/store/database.scm (register-items): Remove #:deduplicate?
parameter and call to 'deduplicate'.
(register-path): Call 'deduplicate' when #:deduplicate? is true.
* gnu/build/image.scm (register-closure): Adjust call accordingly.
* gnu/build/vm.scm (register-closure): Likewise.
* guix/nar.scm (finalize-store-file): Likewise.
* guix/scripts/pack.scm (store-database): Likewise.
2020-12-15 17:32:11 +01:00
Ludovic Courtès
dea1ee1fd7
database: Remove #:reset-timestamps? from 'register-items'.
The assumption now is that the caller took care of resetting timestamps
and permissions.

* guix/store/database.scm (register-items): Remove #:reset-timestamps?
parameter and the call to 'reset-timestamps'.
(register-path): Adjust accordingly and add call to 'reset-timestamps'.
* gnu/build/image.scm (register-closure): Remove #:reset-timestamps?
parameter to 'register-items'.
* gnu/build/vm.scm (register-closure): Likewise.
* guix/nar.scm (finalize-store-file): Adjust accordingly.
* guix/scripts/pack.scm (store-database)[build]: Likewise.
2020-12-15 17:32:10 +01:00
Ludovic Courtès
2718c29c3f
nar: Deduplicate files right as they are restored.
This avoids having to traverse and re-read the files that we have just
restored, thereby reducing I/O.

* guix/serialization.scm (dump-file): New procedure.
(restore-file): Add #:dump-file parameter and honor it.
* guix/store/deduplication.scm (tee, dump-file/deduplicate): New
procedures.
* guix/nar.scm (restore-one-item): Pass #:dump-file to 'restore-file'.
(finalize-store-file): Pass #:deduplicate? #f to 'register-items'.
* tests/nar.scm <top level>: Call 'setenv' to set "NIX_STORE".
2020-12-15 17:32:09 +01:00
Ludovic Courtès
ed7d02f7c1
serialization: 'restore-file' sets canonical timestamp and permissions.
* guix/serialization.scm (restore-file): Set the permissions and mtime
of FILE.
* guix/nar.scm (finalize-store-file): Pass #:reset-timestamps? #f to
'register-items'.
* tests/nar.scm (rm-rf): Add 'chmod' calls to ensure files are writable.
("write-file + restore-file with symlinks"): Ensure every file in OUTPUT
passes 'canonical-file?'.
* tests/guix-archive.sh: Run "chmod -R +w" before "rm -rf".
2020-12-15 17:32:09 +01:00
Ludovic Courtès
7a68d3ccad
doc: Distinguish the "nar bundle" format from "nar".
* doc/guix.texi (Invoking guix archive): Introduce the term "nar bundle"
and clarify what the output of "guix archive --export" really is.
* guix/nar.scm (restore-one-item, restore-file-set): Use the term "nar
bundle" in docstrings.
2020-09-11 17:53:58 +02:00
Ludovic Courtès
3b72618f6e
nar: Use (guix i18n).
* guix/nar.scm: Use (guix i18n) instead of (guix ui).
2020-06-18 14:48:17 +02:00
Ludovic Courtès
d7fb553801
nar: Avoid opening the database an additional time.
* guix/nar.scm (finalize-store-file): Call 'register-items' and pass it
DB.  This avoids opening the database a second time and hopefully
reduces contention on 'db.sqlite-shm'.
2020-06-18 14:48:17 +02:00
Caleb Ristvedt
b338c41c82
nar: 'with-temporary-store-file' uses a single connection
Previously the 'with-store' form was entered every time a different temporary
file was tried.  This caused there to be as many simultaneous open connections
as there were attempts, and prevented the (loop ...) call from being a tail
call.  This change fixes that.

* guix/nar.scm (with-temporary-store-file): open connection once prior to
  entering the loop.
2020-05-07 05:56:01 -05:00
Caleb Ristvedt
37edbc91e3
nar: 'finalize-store-file' follows proper store lock protocol.
* guix/nar.scm (finalize-store-file): check for deletion token when acquiring
  lock, write deletion token and delete lock file before releasing lock.
2020-05-07 05:56:01 -05:00
Ludovic Courtès
70a7a1b5dc
nar: Really lock store files.
Previously, 'lock-store-file' would immediately close the file
descriptor of the '.lock' file, and thus it would immediately release
the lock.

* guix/nar.scm (lock-store-file, unlock-store-file): Remove.
(finalize-store-file): Use 'lock-file' and 'unlock-file' instead.
2019-06-05 23:10:37 +02:00
Ludovic Courtès
1afe1985f5
nar: Access the database instead of connecting to the daemon.
* guix/store/database.scm (%default-database-file): New variable.
(path-id): Export.
* guix/nar.scm (finalize-store-file): Use 'with-database' instead of
'with-store', and use 'path-id' instead of 'valid-path?'.
2018-11-13 14:59:45 +01:00
Ludovic Courtès
ca71942445
Switch to Guile-Gcrypt.
This removes (guix hash) and (guix pk-crypto), which now live as part of
Guile-Gcrypt (version 0.1.0.)

* guix/gcrypt.scm, guix/hash.scm, guix/pk-crypto.scm,
tests/hash.scm, tests/pk-crypto.scm: Remove.
* configure.ac: Test for Guile-Gcrypt.  Remove LIBGCRYPT and
LIBGCRYPT_LIBDIR assignments.
* m4/guix.m4 (GUIX_ASSERT_LIBGCRYPT_USABLE): Remove.
* README: Add Guile-Gcrypt to the dependencies; move libgcrypt as
"required unless --disable-daemon".
* doc/guix.texi (Requirements): Likewise.
* gnu/packages/bash.scm, guix/derivations.scm, guix/docker.scm,
guix/git.scm, guix/http-client.scm, guix/import/cpan.scm,
guix/import/cran.scm, guix/import/crate.scm, guix/import/elpa.scm,
guix/import/gnu.scm, guix/import/hackage.scm,
guix/import/texlive.scm, guix/import/utils.scm, guix/nar.scm,
guix/pki.scm, guix/scripts/archive.scm,
guix/scripts/authenticate.scm, guix/scripts/download.scm,
guix/scripts/hash.scm, guix/scripts/pack.scm,
guix/scripts/publish.scm, guix/scripts/refresh.scm,
guix/scripts/substitute.scm, guix/store.scm,
guix/store/deduplication.scm, guix/tests.scm, tests/base32.scm,
tests/builders.scm, tests/challenge.scm, tests/cpan.scm,
tests/crate.scm, tests/derivations.scm, tests/gem.scm,
tests/nar.scm, tests/opam.scm, tests/pki.scm,
tests/publish.scm, tests/pypi.scm, tests/store-deduplication.scm,
tests/store.scm, tests/substitute.scm: Adjust imports.
* gnu/system/vm.scm: Likewise.
(guile-sqlite3&co): Rename to...
(gcrypt-sqlite3&co): ... this.  Add GUILE-GCRYPT.
(expression->derivation-in-linux-vm)[config]: Remove.
(iso9660-image)[config]: Remove.
(qemu-image)[config]: Remove.
(system-docker-image)[config]: Remove.
* guix/scripts/pack.scm: Adjust imports.
(guile-sqlite3&co): Rename to...
(gcrypt-sqlite3&co): ... this.  Add GUILE-GCRYPT.
(self-contained-tarball)[build]: Call 'make-config.scm' without
 #:libgcrypt argument.
(squashfs-image)[libgcrypt]: Remove.
[build]: Call 'make-config.scm' without #:libgcrypt.
(docker-image)[config, json]: Remove.
[build]: Add GUILE-GCRYPT to the extensions  Remove (guix config) from
the imported modules.
* guix/self.scm (specification->package): Remove "libgcrypt", add
"guile-gcrypt".
(compiled-guix): Remove #:libgcrypt.
[guile-gcrypt]: New variable.
[dependencies]: Add it.
[*core-modules*]: Remove #:libgcrypt from 'make-config.scm' call.
Add #:extensions.
[*config*]: Remove #:libgcrypt from 'make-config.scm' call.
(%dependency-variables): Remove %libgcrypt.
(make-config.scm): Remove #:libgcrypt.
* build-aux/build-self.scm (guile-gcrypt): New variable.
(make-config.scm): Remove #:libgcrypt.
(build-program)[fake-gcrypt-hash]: New variable.
Add (gcrypt hash) to the imported modules.  Adjust load path
assignments.
* gnu/packages/package-management.scm (guix)[propagated-inputs]: Add
GUILE-GCRYPT.
[arguments]: In 'wrap-program' phase, add GUILE-GCRYPT to the search
path.
2018-09-04 17:25:11 +02:00
Ludovic Courtès
df2f6400b1
store: Remove 'register-path'.
* guix/store.scm (register-path): Remove.
* guix/nar.scm: Use (guix store database).
* guix/scripts/system.scm: Likewise.
* tests/store-database.scm: Remove #:hide (register-path).
* tests/store.scm ("register-path"): Remove.
2018-06-14 11:17:00 +02:00
Ludovic Courtès
69daee23af
ui: Rename '_' to 'G_'.
This avoids collisions with '_' when the latter is used as a 'match'
pattern for instance.  See
<https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00464.html>.

* guix/ui.scm: Rename '_' to 'G_'.
* po/guix/Makevars (XGETTEXT_OPTIONS): Adjust accordingly.
* build-aux/compile-all.scm (warnings): Remove 'format'.
* gnu/packages.scm,
gnu/services.scm,
gnu/services/shepherd.scm,
gnu/system.scm,
gnu/system/shadow.scm,
guix/gnupg.scm,
guix/http-client.scm,
guix/import/cpan.scm,
guix/import/elpa.scm,
guix/import/pypi.scm,
guix/nar.scm,
guix/scripts.scm,
guix/scripts/archive.scm,
guix/scripts/authenticate.scm,
guix/scripts/build.scm,
guix/scripts/challenge.scm,
guix/scripts/container.scm,
guix/scripts/container/exec.scm,
guix/scripts/copy.scm,
guix/scripts/download.scm,
guix/scripts/edit.scm,
guix/scripts/environment.scm,
guix/scripts/gc.scm,
guix/scripts/graph.scm,
guix/scripts/hash.scm,
guix/scripts/import.scm,
guix/scripts/import/cpan.scm,
guix/scripts/import/cran.scm,
guix/scripts/import/crate.scm,
guix/scripts/import/elpa.scm,
guix/scripts/import/gem.scm,
guix/scripts/import/gnu.scm,
guix/scripts/import/hackage.scm,
guix/scripts/import/nix.scm,
guix/scripts/import/pypi.scm,
guix/scripts/import/stackage.scm,
guix/scripts/lint.scm,
guix/scripts/offload.scm,
guix/scripts/pack.scm,
guix/scripts/package.scm,
guix/scripts/perform-download.scm,
guix/scripts/publish.scm,
guix/scripts/pull.scm,
guix/scripts/refresh.scm,
guix/scripts/size.scm,
guix/scripts/substitute.scm,
guix/scripts/system.scm,
guix/ssh.scm,
guix/upstream.scm: Use 'G_' instead of '_'.  Most of this change was
obtained by running: "sed -i -e's/(_ "/(G_ "/g' `find -name \*.scm`".
2017-05-03 16:16:17 +02:00
Ludovic Courtès
4e0ea3eb28
utils: Move 'fcntl-flock' to (guix build syscalls).
* guix/utils.scm (%struct-flock, F_SETLKW, F_SETLK, F_xxLCK)
(fcntl-flock): Move to...
* guix/build/syscalls.scm: ... here.  New variables.
* guix/nar.scm: Adjust imports accordingly.
* tests/utils.scm ("fcntl-flock wait", "fcntl-flock non-blocking"): Move
to...
* tests/syscalls.scm: ... here.  New tests.
(temp-file): New variable.
2016-05-06 13:25:30 +02:00
Ludovic Courtès
71c1d5280c nar: Read archive signatures as Latin-1 strings.
Fixes <http://bugs.gnu.org/19610>.
Reported by Mark H Weaver <mhw@netris.org>.

* guix/nar.scm (restore-one-item): Use 'read-latin1-string' to read the
  signature.
2015-01-18 22:19:04 +01:00
Ludovic Courtès
0363991a25 Break module cycle involving (guix store) and (guix ui).
Before, there was a cycle along the lines of:

  (guix store) -> (guix nar) -> (guix ui) -> (guix store)

This caused problems, as discussed at:

  http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00109.html

This patch removes cycles in the (guix ...) modules.

* guix/nar.scm (&nar-error, &nar-read-error, dump, write-contents,
  read-contents, %archive-version-1, write-file, restore-file): Move to...
* guix/serialization.scm: ... here.
* guix/store.scm: Remove dependency on (guix nar).
* guix/scripts/hash.scm, guix/scripts/offload.scm,
  guix/scripts/substitute-binary.scm, tests/nar.scm, tests/store.scm,
  tests/substitute-binary.scm: Adjust accordingly.
2014-10-09 23:51:19 +02:00
Ludovic Courtès
c6df09941b guix-register: By default, attempt to deduplicate registered items.
* nix/guix-register/guix-register.cc (GUIX_OPT_DEDUPLICATE): New macro.
  (options): Add --no-deduplication.
  (deduplication): New variable.
  (parse_opt): Handle GUIX_OPT_DEDUPLICATE.
  (register_validity): Add 'optimize' parameter and honor it.
  (main): Move 'store' instanatiation after 'settings.nixStore'
  assignment.
* tests/guix-register.sh: Add test for deduplication.
* guix/nar.scm (finalize-store-file): Update comment above
  'register-path' call.
2014-08-25 00:16:49 +02:00
Ludovic Courtès
6ef3644e34 pk-crypto: Add pretty-printer to 'gcry-error' exceptions.
* guix/pk-crypto.scm (string->canonical-sexp, sign, generate-key): Pass
  the procedure name as the first argument to 'throw'.
  (gcrypt-error-printer): New procedure.
  <top level>: Add call to 'set-exception-printer!'.
* guix/nar.scm (restore-one-item): Add 'proc' parameter to 'catch'
  handler for 'gcry-error.
* guix/scripts/archive.scm (%options, generate-key-pair, authorize-key):
  Likewise.
* guix/scripts/substitute-binary.scm (narinfo-signature->canonical-sexp):
  Likewise.
2014-04-22 11:41:52 +02:00
Ludovic Courtès
50db7d82b3 nar: Really really protect the temporary store directory from GC.
This is a follow-up to 6071b55e10.
See <https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00167.html>
for the original report, and
<https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00198.html>
for an alternate solution that has been discussed.

* guix/nar.scm (temporary-store-file): Remove call to
  'add-permanent-root'; don't loop.
  (with-temporary-store-file): Rewrite using 'with-store' and
  'add-temp-root'.
2014-04-21 23:24:23 +02:00
Ludovic Courtès
6071b55e10 nar: Really protect the temporary store directory from GC.
Prevents garbage collection of the temporary store directory while
restoring a file set, as it could previously happen:
<https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00167.html>.

* guix/nar.scm (temporary-store-directory): Rename to...
  (temporary-store-file): ... this.  Use 'add-permanent-root' instead of
  'add-indirect-root'.
  (with-temporary-store-file): New macro.
  (restore-one-item): New procedure, with code formerly in
  'restore-file-set'.  Use 'with-temporary-store-file'.
  (restore-file-set): Use it.
2014-04-12 23:03:56 +02:00
Ludovic Courtès
9132b9bd72 nar: 'restore-file-set' registers the temporary result as a GC root.
* guix/nar.scm (temporary-store-directory): Use 'add-indirect-root', not
  'add-temp-root'.  Reported by Andreas Enge <andreas@enge.fr> at
  <https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00161.html>.
2014-04-09 23:01:02 +02:00
Mark H Weaver
d2d8779b21 Work around behavior of old 'scandir' in Guile 2.0.5.
Problem reported by John Darrington <john@darrington.wattle.id.au>.

* guix/nar.scm (write-file): Filter out "." and ".." from the result of
  'scandir'.  Previously we did this by passing a suitable predicate.
2014-04-07 04:20:30 -04:00
Ludovic Courtès
e4687a5e68 Use 'signature-case' in (guix nar) and 'substitute-binary'.
* guix/nar.scm (restore-file-set)[assert-valid-signature]: Rewrite in
  terms of 'signature-case'.
* guix/scripts/substitute-binary.scm (narinfo-signature->canonical-sexp):
  Call 'leave' instead of 'raise' when SIGNATURE is invalid.
  (&nar-signature-error, &nar-invalid-hash-error): Remove.
  (assert-valid-signature): Add 'narinfo' parameter; remove 'port'.
  Rewrite in terms of 'signature-case' and 'leave'.  Mention NARINFO's
  URI in error messages.  Adjust caller.
  (narinfo-sha256): New procedure.
  (assert-valid-narinfo): Use it.
  (valid-narinfo?): Rewrite using 'narinfo-sha256' and
  'signature-case'.
* tests/substitute-binary.scm (assert-valid-signature,
  test-error-condition): Remove.
  ("corrupt signature data", "unauthorized public key", "invalid
  signature"): Remove.
2014-03-31 23:47:02 +02:00
Nikita Karetnikov
24194b6b54 nar: Clarify that 'assert-valid-signature' accepts a string.
* guix/nar.scm (assert-valid-signature): Improve the wording.
2014-03-30 12:02:10 +02:00
Ludovic Courtès
96c7448f37 nar: Produce archives with files sorted in C collation order.
* guix/nar.scm (write-file) <directory>: Pass 'string<?' as the second
  argument to 'scandir'.
* tests/nar.scm ("write-file puts file in C locale collation order"):
  New test.
2014-02-21 23:49:53 +01:00
Ludovic Courtès
3140f2df42 guix hash: Add '--recursive'.
* guix/scripts/hash.scm (show-help): Add --recursive.
  (%options): Likewise.
  (guix-hash)[file-hash]: New procedure.  Honor --recursive.
  Use it.
* guix/nar.scm (write-file): Add missing field to the &nar-error
  condition raised upon unsupported file type; change its message to be
  more descriptive.
* tests/guix-hash.sh: Add tests with -r.
* doc/guix.texi (Invoking guix hash): Document --recursive.
2014-02-21 23:49:52 +01:00
Ludovic Courtès
a93e91ff48 nar: 'write-file' can write to non-file ports.
* guix/nar.scm (write-contents): Use 'sendfile' only when P is a file
  port.
* tests/nar.scm ("write-file supports non-file output ports"): New test.
2014-02-21 23:49:52 +01:00
Ludovic Courtès
cd4027fa47 nar: Add 'restore-file-set', for use by build hooks.
* guix/nar.scm (&nar-invalid-hash-error, &nar-signature-error): New
  condition types.
  (&nar-error): Add 'file' and 'port' fields.
  (&nar-read-error): Remove 'port' and 'file' fields.
  (lock-store-file, unlock-store-file, finalize-store-file,
  temporary-store-directory, restore-file-set): New procedures.
* tests/nar.scm (%seed): New variable.
  (random-text): New procedure.
  ("restore-file-set (signed, valid)", "restore-file-set (missing
  signature)", "restore-file-set (corrupt)"): New tests.
* po/Makevars (XGETTEXT_OPTIONS): Add '--keyword=message'.nar fixes
* po/POTFILES.in: Add guix/nar.scm.
2014-01-24 00:01:50 +01:00
Ludovic Courtès
48e488eb2c nar: Fix file descriptor leak when writing a Nar.
* guix/nar.scm (write-contents)[call-with-binary-input-file]: Always
  close PORT.
2013-09-23 00:33:50 +02:00
Ludovic Courtès
8f3114b7a4 nar: Add support for symlinks.
* guix/nar.scm (write-file): Add case for type `symlink'.
  (restore-file): Likewise.
* tests/nar.scm (random-file-size, make-file-tree, delete-file-tree,
  with-file-tree, file-tree-equal?, make-random-bytevector,
  populate-file): New procedures.
  (%test-dir): New variable.
  ("write-file + restore-file"): Use `%test-dir' and `file-tree-equal?'.
  ("write-file + restore-file with symlinks"): New test.
2013-04-12 14:42:56 +02:00
Ludovic Courtès
53c63ee937 nar: Implement restoration from Nar.
* guix/nar.scm (&nar-error, &nar-read-error): New condition types.
  (dump): New procedure.
  (write-contents)[dump]: Remove.  Use the one above instead.
  (read-contents, write-file, restore-file): New procedures.
  (%archive-version-1): New variable.
2013-04-08 23:21:38 +02:00
Ludovic Courtès
0f41c26f9b Add (guix nar) and (guix serialization).
* guix/store.scm (write-int, read-int, write-long-long, read-long-long,
  write-padding, write-string, read-string, read-latin1-string,
  write-string-list, read-string-list, write-store-path,
  read-store-path, write-store-path-list, read-store-path-list): Move to
  serialization.scm.
  (write-contents, write-file): Move to nar.scm.
* guix/nar.scm, guix/serialization.scm: New files.
* Makefile.am (MODULES): Add them.
2013-04-04 22:29:08 +02:00