futurile/guix-play
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

build-system: Add pyproject-build-system.

Browse Source 
This is an experimental build system based on python-build-system
that implements PEP 517-compliant builds.

* doc/guix.texi (Build Systems): Add pyproject-build-system section.
* doc/contributing.texi (Python Modules): Mention pyproject.toml and the
PYTHON-TOOLCHAIN package, as well as differences to python-build-system.
* guix/build-system/pyproject.scm,
guix/build/pyproject-build-system.scm,
gnu/packages/aux-files/python/sanity-check-next.py,
gnu/packages/python-commencement.scm: New files.
* Makefile.am (MODULES): Register the new build systems.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add python-commencement.scm.
* gnu/packages/python.scm (python-sans-pip, python-sans-pip-wrapper): New
variables.

Co-authored-by: Marius Bakke <marius@gnu.org>
This commit is contained in:
Lars-Dominik Braun 2022-04-23 11:36:55 +02:00 committed by Marius Bakke
parent b4e2effb30
commit 400a7a4c80
No known key found for this signature in database
GPG Key ID: A2A06DF2A33A54FA
7 changed files with 732 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Makefile.am
View File

@ -166,6 +166,7 @@ MODULES = \
guix/build-system/maven.scm \
guix/build-system/node.scm \
guix/build-system/perl.scm \
guix/build-system/pyproject.scm \
guix/build-system/python.scm \
guix/build-system/renpy.scm \
guix/build-system/ocaml.scm \
@ -222,6 +223,7 @@ MODULES = \
guix/build/minetest-build-system.scm \
guix/build/node-build-system.scm \
guix/build/perl-build-system.scm \
guix/build/pyproject-build-system.scm \
guix/build/python-build-system.scm \
guix/build/ocaml-build-system.scm \
guix/build/qt-build-system.scm \

36
doc/contributing.texi
View File

@ -786,12 +786,29 @@ for instance, the module python-dateutil is packaged under the names
starts with @code{py} (e.g.@: @code{pytz}), we keep it and prefix it as
described above.
@quotation Note
Currently there are two different build systems for Python packages in Guix:
@var{python-build-system} and @var{pyproject-build-system}. For the
longest time, Python packages were built from an informally specified
@file{setup.py} file. That worked amazingly well, considering Python's
success, but was difficult to build tooling around. As a result, a host
of alternative build systems emerged and the community eventually settled on a
@url{https://peps.python.org/pep-0517/, formal standard} for specifying build
requirements. @var{pyproject-build-system} is Guix's implementation of this
standard. It is considered ``experimental'' in that it does not yet support
all the various PEP-517 @emph{build backends}, but you are encouraged to try
it for new Python packages and report any problems. It will eventually be
deprecated and merged into @var{python-build-system}.
@end quotation
@subsubsection Specifying Dependencies
@cindex inputs, for Python packages
Dependency information for Python packages is usually available in the
package source tree, with varying degrees of accuracy: in the
@file{setup.py} file, in @file{requirements.txt}, or in @file{tox.ini}.
@file{pyproject.toml} file, the @file{setup.py} file, in
@file{requirements.txt}, or in @file{tox.ini} (the latter mostly for
test dependencies).
Your mission, when writing a recipe for a Python package, is to map
these dependencies to the appropriate type of ``input'' (@pxref{package
@ -802,10 +819,12 @@ following check list to determine which dependency goes where.
@itemize
@item
We currently package Python 2 with @code{setuptools} and @code{pip}
installed like Python 3.4 has per default. Thus you don't need to
specify either of these as an input. @command{guix lint} will warn you
if you do.
We currently package Python with @code{setuptools} and @code{pip}
installed per default. This is about to change, and users are encouraged
to use @code{python-toolchain} if they want a build environment for Python.
@command{guix lint} will warn if @code{setuptools} or @code{pip} are
added as native-inputs because they are generally not necessary.
@item
Python dependencies required at run time go into
@ -814,9 +833,10 @@ Python dependencies required at run time go into
@file{requirements.txt} file.
@item
Python packages required only at build time---e.g., those listed with
the @code{setup_requires} keyword in @file{setup.py}---or only for
testing---e.g., those in @code{tests_require}---go into
Python packages required only at build time---e.g., those listed under
@code{build-system.requires} in @file{pyproject.toml} or with the
@code{setup_requires} keyword in @file{setup.py}---or dependencies only
for testing---e.g., those in @code{tests_require} or @file{tox.ini}---go into
@code{native-inputs}. The rationale is that (1) they do not need to be
propagated because they are not needed at run time, and (2) in a
cross-compilation context, it's the ``native'' input that we'd want.

29
doc/guix.texi
View File

@ -9313,7 +9313,36 @@ instead of the default @code{"out"} output. This is useful for packages that
include a Python package as only a part of the software, and thus want to
combine the phases of @code{python-build-system} with another build system.
Python bindings are a common usecase.
@end defvr
@defvr {Scheme Variable} pyproject-build-system
This is a variable exported by @code{guix build-system pyproject}. It
is based on @var{python-build-system}, and adds support for
@file{pyproject.toml} and @url{https://peps.python.org/pep-0517/, PEP 517}.
It also supports a variety of build backends and test frameworks.
The API is slightly different from @var{python-build-system}:
@itemize
@item
@code{#:use-setuptools?} and @code{#:test-target} is removed.
@item
@code{#:build-backend} is added. It defaults to @code{#false} and will try
to guess the appropriate backend based on @file{pyproject.toml}.
@item
@code{#:test-backend} is added. It defaults to @code{#false} and will guess
an appropriate test backend based on what is available in package inputs.
@item
@code{#:test-flags} is added. The default is @code{#false}, and varies based
on the detected @code{#:test-backend}.
@end itemize
It is considered ``experimental'' in that the implementation details are
not set in stone yet, however users are encouraged to try it for new
Python projects (even those using @file{setup.py}). The API is subject to
change, but any breaking changes in the Guix channel will be dealt with.
Eventually this build system will be deprecated and merged back into
@var{python-build-system}, probably some time in 2024.
@end defvr
@defvr {Scheme Variable} perl-build-system

99
gnu/packages/aux-files/python/sanity-check-next.py Normal file
View File

@ -0,0 +1,99 @@
# -*- coding: utf-8 -*-
# GNU Guix --- Functional package management for GNU
# Copyright © 2021, 2022 Lars-Dominik Braun <lars@6xq.net>
#
# This file is part of GNU Guix.
#
# GNU Guix is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or (at
# your option) any later version.
#
# GNU Guix is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
# This version adds a small change to accommodate missing python-setuptools.
# TODO: Merge with sanity-check.py in the next core-updates cycle.
from __future__ import print_function # Python 2 support.
import importlib
import sys
import traceback
try:
import pkg_resources
except ImportError:
print('Warning: Skipping, because python-setuptools are not available.')
sys.exit(0)
try:
from importlib.machinery import PathFinder
except ImportError:
PathFinder = None
ret = 0
# Only check site-packages installed by this package, but not dependencies
# (which pkg_resources.working_set would include). Path supplied via argv.
ws = pkg_resources.find_distributions(sys.argv[1])
for dist in ws:
print('validating', repr(dist.project_name), dist.location)
try:
print('...checking requirements: ', end='')
req = str(dist.as_requirement())
# dist.activate() is not enough to actually check requirements, we
# have to .require() it.
pkg_resources.require(req)
print('OK')
except Exception as e:
print('ERROR:', req, repr(e))
ret = 1
continue
# Try to load top level modules. This should not have any side-effects.
try:
metalines = dist.get_metadata_lines('top_level.txt')
except (KeyError, EnvironmentError):
# distutils (i.e. #:use-setuptools? #f) will not install any metadata.
# This file is also missing for packages built using a PEP 517 builder
# such as poetry.
print('WARNING: cannot determine top-level modules')
continue
for name in metalines:
# Only available on Python 3.
if PathFinder and PathFinder.find_spec(name) is None:
# Ignore unavailable modules, often C modules, which were not
# installed at the top-level. Cannot use ModuleNotFoundError,
# because it is raised by failed imports too.
continue
try:
print('...trying to load module', name, end=': ')
importlib.import_module(name)
print('OK')
except Exception:
print('ERROR:')
traceback.print_exc(file=sys.stdout)
ret = 1
# Try to load entry points of console scripts too, making sure they
# work. They should be removed if they don't. Other groups may not be
# safe, as they can depend on optional packages.
for group, v in dist.get_entry_map().items():
if group not in {'console_scripts', 'gui_scripts'}:
continue
for name, ep in v.items():
try:
print('...trying to load endpoint', group, name, end=': ')
ep.load()
print('OK')
except Exception:
print('ERROR:')
traceback.print_exc(file=sys.stdout)
ret = 1
sys.exit(ret)

35
gnu/packages/python.scm
View File

@ -26,7 +26,7 @@
;;; Copyright © 2016, 2017 Nikita <nikita@n0.is>
;;; Copyright © 2016 Dylan Jeffers <sapientech@sapientech@openmailbox.org>
;;; Copyright © 2016 David Craven <david@craven.ch>
;;; Copyright © 2016, 2017, 2018, 2019, 2020, 2021 Marius Bakke <marius@gnu.org>
;;; Copyright © 2016-2022 Marius Bakke <marius@gnu.org>
;;; Copyright © 2016, 2017 Stefan Reichör <stefan@xsteve.at>
;;; Copyright © 2016, 2017 Alex Vong <alexvong1995@gmail.com>
;;; Copyright © 2016, 2017, 2018 Arun Isaac <arunisaac@systemreboot.net>
@ -60,6 +60,7 @@
;;; Copyright © 2020, 2021 Greg Hogan <code@greghogan.com>
;;; Copyright © 2022 Philip McGrath <philip@philipmcgrath.com>
;;; Copyright © 2022 jgart <jgart@dismail.de>
;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
;;;
;;; This file is part of GNU Guix.
;;;
@ -87,6 +88,7 @@
#:use-module (gnu packages hurd)
#:use-module (gnu packages libffi)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages python-build)
#:use-module (gnu packages readline)
#:use-module (gnu packages sqlite)
#:use-module (gnu packages tcl)
@ -674,6 +676,37 @@ and the unversioned commands available.")))
(define-public python-wrapper (wrap-python3 python))
(define-public python-minimal-wrapper (wrap-python3 python-minimal))
;; The Python used in pyproject-build-system.
(define-public python-sans-pip
(hidden-package
(package/inherit python
(arguments
(substitute-keyword-arguments (package-arguments python)
((#:configure-flags flags #~())
#~(append '("--with-ensurepip=no")
(delete "--with-ensurepip=install" #$flags))))))))
(define-public python-sans-pip-wrapper
(wrap-python3 python-sans-pip))
(define-public python-toolchain
(let ((base (package/inherit python-sans-pip-wrapper)))
(package
(inherit base)
(properties '())
(name "python-toolchain")
(propagated-inputs
(modify-inputs (package-propagated-inputs base)
(append python-pip
python-pypa-build
python-setuptools
python-wheel)))
(synopsis "Python toolchain")
(description
"Python toolchain including Python itself, setuptools and pip.
Use this package if you need a minimal Python toolchain instead of just
the interpreter."))))
(define-public micropython
(package
(name "micropython")

147
guix/build-system/pyproject.scm Normal file
View File

@ -0,0 +1,147 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
;;; Copyright © 2022 Marius Bakke <marius@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
(define-module (guix build-system pyproject)
#:use-module ((gnu packages) #:select (search-auxiliary-file))
#:use-module (guix gexp)
#:use-module (guix store)
#:use-module (guix utils)
#:use-module (guix memoization)
#:use-module (guix gexp)
#:use-module (guix monads)
#:use-module (guix packages)
#:use-module (guix derivations)
#:use-module (guix search-paths)
#:use-module (guix build-system)
#:use-module (guix build-system gnu)
#:use-module (guix build-system python)
#:use-module (ice-9 match)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-26)
#:export (%pyproject-build-system-modules
default-python
pyproject-build
pyproject-build-system))
;; Commentary:
;;
;; Standard build procedure for Python packages using 'pyproject.toml'.
;; This is implemented as an extension of 'python-build-system'.
;;
;; Code:
(define %pyproject-build-system-modules
;; Build-side modules imported by default.
`((guix build pyproject-build-system)
(guix build json)
,@%python-build-system-modules))
(define (default-python)
"Return the default Python package."
;; Lazily resolve the binding to avoid a circular dependency.
(let ((python (resolve-interface '(gnu packages python))))
(module-ref python 'python-toolchain)))
(define sanity-check.py
;; TODO: Merge with sanity-check.py in the next rebuild cycle.
(search-auxiliary-file "python/sanity-check-next.py"))
(define* (lower name
#:key source inputs native-inputs outputs system target
(python (default-python))
#:allow-other-keys
#:rest arguments)
"Return a bag for NAME."
(define private-keywords
'(#:target #:python #:inputs #:native-inputs))
(and (not target) ;XXX: no cross-compilation
(bag
(name name)
(system system)
(host-inputs `(,@(if source
`(("source" ,source))
'())
,@inputs
;; Keep the standard inputs of 'gnu-build-system'.
,@(standard-packages)))
(build-inputs `(("python" ,python)
("sanity-check.py" ,(local-file sanity-check.py))
,@native-inputs))
(outputs (append outputs '(wheel)))
(build pyproject-build)
(arguments (strip-keyword-arguments private-keywords arguments)))))
(define* (pyproject-build name inputs
#:key source
(tests? #t)
(configure-flags ''())
(build-backend #f)
(test-backend #f)
(test-flags #f)
(phases '%standard-phases)
(outputs '("out" "wheel"))
(search-paths '())
(system (%current-system))
(guile #f)
(imported-modules %pyproject-build-system-modules)
(modules '((guix build pyproject-build-system)
(guix build utils))))
"Build SOURCE using PYTHON, and with INPUTS."
(define build
(with-imported-modules imported-modules
#~(begin
(use-modules #$@(sexp->gexp modules))
#$(with-build-variables inputs outputs
#~(pyproject-build
#:name #$name
#:source #+source
#:configure-flags #$configure-flags
#:system #$system
#:build-backend #$build-backend
#:test-backend #$test-backend
#:test-flags #$test-flags
#:tests? #$tests?
#:phases #$(if (pair? phases)
(sexp->gexp phases)
phases)
#:outputs %outputs
#:search-paths '#$(sexp->gexp
(map search-path-specification->sexp
search-paths))
#:inputs %build-inputs)))))
(mlet %store-monad ((guile (package->derivation (or guile (default-guile))
system #:graft? #f)))
(gexp->derivation name build
#:system system
#:graft? #f ;consistent with 'gnu-build'
#:target #f
#:guile-for-build guile)))
(define pyproject-build-system
(build-system
(name 'pyproject)
(description "The PEP517-compliant Python build system")
(lower lower)))
;;; pyproject.scm ends here

393
guix/build/pyproject-build-system.scm Normal file
View File

@ -0,0 +1,393 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
;;; Copyright © 2022 Marius Bakke <marius@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
(define-module (guix build pyproject-build-system)
#:use-module ((guix build python-build-system) #:prefix python:)
#:use-module (guix build utils)
#:use-module (guix build json)
#:use-module (ice-9 match)
#:use-module (ice-9 ftw)
#:use-module (ice-9 format)
#:use-module (ice-9 rdelim)
#:use-module (ice-9 regex)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-26)
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-35)
#:export (%standard-phases
add-installed-pythonpath
site-packages
python-version
pyproject-build))
;;; Commentary:
;;;
;;; PEP 517-compatible build system for Python packages.
;;;
;;; PEP 517 mandates the use of a TOML file called pyproject.toml at the
;;; project root, describing build and runtime dependencies, as well as the
;;; build system, which can be different from setuptools. This module uses
;;; that file to extract the build system used and call its wheel-building
;;; entry point build_wheel (see 'build). setuptools’ wheel builder is
;;; used as a fallback if either no pyproject.toml exists or it does not
;;; declare a build-system. It supports config_settings through the
;;; standard #:configure-flags argument.
;;;
;;; This wheel, which is just a ZIP file with a file structure defined
;;; by PEP 427 (https://www.python.org/dev/peps/pep-0427/), is then unpacked
;;; and its contents are moved to the appropriate locations in 'install.
;;;
;;; Then entry points, as defined by the PyPa Entry Point Specification
;;; (https://packaging.python.org/specifications/entry-points/) are read
;;; from a file called entry_points.txt in the package’s site-packages
;;; subdirectory and scripts are written to bin/. These are not part of a
;;; wheel and expected to be created by the installing utility.
;;; TODO: Add support for PEP-621 entry points.
;;;
;;; Caveats:
;;; - There is no support for in-tree build backends.
;;;
;;; Code:
;;;
;; Re-export these variables from python-build-system as many packages
;; rely on these.
(define python-version python:python-version)
(define site-packages python:site-packages)
(define add-installed-pythonpath python:add-installed-pythonpath)
;; Base error type.
(define-condition-type &python-build-error &error python-build-error?)
;; Raised when 'check cannot find a valid test system in the inputs.
(define-condition-type &test-system-not-found &python-build-error
test-system-not-found?)
;; Raised when multiple wheels are created by 'build.
(define-condition-type &cannot-extract-multiple-wheels &python-build-error
cannot-extract-multiple-wheels?)
;; Raised, when no wheel has been built by the build system.
(define-condition-type &no-wheels-built &python-build-error no-wheels-built?)
(define* (build #:key outputs build-backend configure-flags #:allow-other-keys)
"Build a given Python package."
(define (pyproject.toml->build-backend file)
"Look up the build backend in a pyproject.toml file."
(call-with-input-file file
(lambda (in)
(let loop
((line (read-line in
'concat)))
(if (eof-object? line) #f
(let ((m (string-match "build-backend = [\"'](.+)[\"']" line)))
(if m
(match:substring m 1)
(loop (read-line in
'concat)))))))))
(let* ((wheel-output (assoc-ref outputs "wheel"))
(wheel-dir (if wheel-output wheel-output "dist"))
;; There is no easy way to get data from Guile into Python via
;; s-expressions, but we have JSON serialization already, which Python
;; also supports out-of-the-box.
(config-settings (call-with-output-string (cut write-json
configure-flags <>)))
;; python-setuptools’ default backend supports setup.py *and*
;; pyproject.toml. Allow overriding this automatic detection via
;; build-backend.
(auto-build-backend (if (file-exists? "pyproject.toml")
(pyproject.toml->build-backend
"pyproject.toml") #f))
;; Use build system detection here and not in importer, because a) we
;; have alot of legacy packages and b) the importer cannot update arbitrary
;; fields in case a package switches its build system.
(use-build-backend (or build-backend auto-build-backend
"setuptools.build_meta")))
(format #t
"Using '~a' to build wheels, auto-detected '~a', override '~a'.~%"
use-build-backend auto-build-backend build-backend)
(mkdir-p wheel-dir)
;; Call the PEP 517 build function, which drops a .whl into wheel-dir.
(invoke "python"
"-c"
"import sys, importlib, json\nconfig_settings = json.loads (sys.argv[3])\nbuilder = importlib.import_module(sys.argv[1])\nbuilder.build_wheel(sys.argv[2], config_settings=config_settings)"
use-build-backend
wheel-dir
config-settings)))
(define* (check #:key inputs
outputs
tests?
test-backend
test-flags
#:allow-other-keys)
"Run the test suite of a given Python package."
(if tests?
;; Unfortunately with PEP 517 there is no common method to specify test
;; systems. Guess test system based on inputs instead.
(let* ((pytest (which "pytest"))
(nosetests (which "nosetests"))
(nose2 (which "nose2"))
(have-setup-py (file-exists? "setup.py"))
(use-test-backend
(or test-backend
;; Prefer pytest
(if pytest 'pytest #f)
(if nosetests 'nose #f)
(if nose2 'nose2 #f)
;; But fall back to setup.py, which should work for most
;; packages. XXX: would be nice not to depend on setup.py here?
;; fails more often than not to find any tests at all. Maybe
;; we can run `python -m unittest`?
(if have-setup-py 'setup.py #f))))
(format #t "Using ~a~%" use-test-backend)
(match use-test-backend
('pytest
(apply invoke (cons pytest (or test-flags '("-vv")))))
('nose
(apply invoke (cons nosetests (or test-flags '("-v")))))
('nose2
(apply invoke (cons nose2 (or test-flags '("-v" "--pretty-assert")))))
('setup.py
(apply invoke (append '("python" "setup.py")
(or test-flags '("test" "-v")))))
;; The developer should explicitly disable tests in this case.
(else (raise (condition (&test-system-not-found))))))
(format #t "test suite not run~%")))
(define* (install #:key inputs outputs #:allow-other-keys)
"Install a wheel file according to PEP 427"
;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl
(let ((site-dir (site-packages inputs outputs))
(python (assoc-ref inputs "python"))
(out (assoc-ref outputs "out")))
(define (extract file)
"Extract wheel (ZIP file) into site-packages directory"
;; Use Python’s zipfile to avoid extra dependency
(invoke "python"
"-m"
"zipfile"
"-e"
file
site-dir))
(define python-hashbang
(string-append "#!" python "/bin/python"))
(define* (merge-directories source destination
#:optional (post-move #f))
"Move all files in SOURCE into DESTINATION, merging the two directories."
(format #t "Merging directory ~a into ~a~%" source destination)
(for-each (lambda (file)
(format #t
"~a/~a -> ~a/~a~%"
source
file
destination
file)
(mkdir-p destination)
(rename-file (string-append source "/" file)
(string-append destination "/" file))
(when post-move
(post-move file)))
(scandir source
(negate (cut member <>
'("." "..")))))
(rmdir source))
(define (expand-data-directory directory)
"Move files from all .data subdirectories to their respective\ndestinations."
;; Python’s distutils.command.install defines this mapping from source to
;; destination mapping.
(let ((source (string-append directory "/scripts"))
(destination (string-append out "/bin")))
(when (file-exists? source)
(merge-directories source destination
(lambda (f)
(let ((dest-path (string-append destination "/"
f)))
(chmod dest-path #o755)
(substitute* dest-path
(("#!python")
python-hashbang)))))))
;; Data can be contained in arbitrary directory structures. Most
;; commonly it is used for share/.
(let ((source (string-append directory "/data"))
(destination out))
(when (file-exists? source)
(merge-directories source destination)))
(let* ((distribution (car (string-split (basename directory) #\-)))
(source (string-append directory "/headers"))
(destination (string-append out "/include/python"
(python-version python) "/"
distribution)))
(when (file-exists? source)
(merge-directories source destination))))
(define (list-directories base predicate)
;; Cannot use find-files here, because it’s recursive.
(scandir base
(lambda (name)
(let ((stat (lstat (string-append base "/" name))))
(and (not (member name
'("." "..")))
(eq? (stat:type stat)
'directory)
(predicate name stat))))))
(let* ((wheel-output (assoc-ref outputs "wheel"))
(wheel-dir (if wheel-output wheel-output "dist"))
(wheels (map (cut string-append wheel-dir "/" <>)
(scandir wheel-dir
(cut string-suffix? ".whl" <>)))))
(cond
((> (length wheels) 1)
;This code does not support multiple wheels
;; yet, because their outputs would have to be
;; merged properly.
(raise (condition (&cannot-extract-multiple-wheels))))
((= (length wheels) 0)
(raise (condition (&no-wheels-built)))))
(for-each extract wheels))
(let ((datadirs (map (cut string-append site-dir "/" <>)
(list-directories site-dir
(file-name-predicate "\\.data$")))))
(for-each (lambda (directory)
(expand-data-directory directory)
(rmdir directory)) datadirs))))
(define* (compile-bytecode #:key inputs outputs #:allow-other-keys)
"Compile installed byte-code in site-packages."
(let* ((site-dir (site-packages inputs outputs))
(python (assoc-ref inputs "python"))
(major-minor (map string->number
(take (string-split (python-version python) #\.) 2)))
(<3.7? (match major-minor
((major minor)
(or (< major 3)
(and (= major 3)
(< minor 7)))))))
(if <3.7?
;; These versions don’t have the hash invalidation modes and do
;; not produce reproducible bytecode files.
(format #t "Skipping bytecode compilation for Python version ~a < 3.7~%"
(python-version python))
(invoke "python" "-m" "compileall"
"--invalidation-mode=unchecked-hash" site-dir))))
(define* (create-entrypoints #:key inputs outputs (configure-flags '()) #:allow-other-keys)
"Implement Entry Points Specification
(https://packaging.python.org/specifications/entry-points/) by PyPa,
which creates runnable scripts in bin/ from entry point specification
file entry_points.txt. This is necessary, because wheels do not contain
these binaries and installers are expected to create them."
(define (entry-points.txt->entry-points file)
"Specialized parser for Python configfile-like files, in particular
entry_points.txt. Returns a list of console_script and gui_scripts
entry points."
(call-with-input-file file
(lambda (in)
(let loop ((line (read-line in))
(inside #f)
(result '()))
(if (eof-object? line)
result
(let* ((group-match (string-match "^\\[(.+)\\]$" line))
(group-name (if group-match
(match:substring group-match 1)
#f))
(next-inside (if (not group-name)
inside
(or (string=? group-name
"console_scripts")
(string=? group-name "gui_scripts"))))
(item-match (string-match
"^([^ =]+)\\s*=\\s*([^:]+):(.+)$" line)))
(if (and inside item-match)
(loop (read-line in)
next-inside
(cons (list (match:substring item-match 1)
(match:substring item-match 2)
(match:substring item-match 3))
result))
(loop (read-line in) next-inside result))))))))
(define (create-script path name module function)
"Create a Python script from an entry point’s NAME, MODULE and
FUNCTION and return write it to PATH/NAME."
(let ((interpreter (which "python"))
(file-path (string-append path "/" name)))
(format #t "Creating entry point for '~a.~a' at '~a'.~%"
module function file-path)
(call-with-output-file file-path
(lambda (port)
;; Technically the script could also include search-paths,
;; but having a generic 'wrap phases also handles manually
;; written entry point scripts.
(format port "#!~a
# Auto-generated entry point script.
import sys
import ~a as mod
sys.exit (mod.~a ())~%" interpreter module function)))
(chmod file-path #o755)))
(let* ((site-dir (site-packages inputs outputs))
(out (assoc-ref outputs "out"))
(bin-dir (string-append out "/bin"))
(entry-point-files (find-files site-dir "^entry_points.txt$")))
(mkdir-p bin-dir)
(for-each (lambda (f)
(for-each (lambda (ep)
(apply create-script
(cons bin-dir ep)))
(entry-points.txt->entry-points f)))
entry-point-files)))
(define* (set-SOURCE-DATE-EPOCH* #:rest _)
"Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools
that incorporate timestamps as a way to tell them to use a fixed timestamp.
See https://reproducible-builds.org/specs/source-date-epoch/."
;; Use a post-1980 timestamp because the Zip format used in wheels do
;; not support timestamps before 1980.
(setenv "SOURCE_DATE_EPOCH" "315619200"))
(define %standard-phases
;; The build phase only builds C extensions and copies the Python sources,
;; while the install phase copies then byte-compiles the sources to the
;; prefix directory. The check phase is moved after the installation phase
;; to ease testing the built package.
(modify-phases python:%standard-phases
(replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH*)
(replace 'build build)
(replace 'install install)
(delete 'check)
;; Must be before tests, so they can use installed packages’ entry points.
(add-before 'wrap 'create-entrypoints create-entrypoints)
(add-after 'wrap 'check check)
(add-before 'check 'compile-bytecode compile-bytecode)))
(define* (pyproject-build #:key inputs (phases %standard-phases)
#:allow-other-keys #:rest args)
"Build the given Python package, applying all of PHASES in order."
(apply python:python-build #:inputs inputs #:phases phases args))
;;; pyproject-build-system.scm ends here