238 lines
11 KiB
Plaintext
238 lines
11 KiB
Plaintext
$OpenBSD: README.internals,v 1.3 2011/11/19 14:35:56 espie Exp $
|
|
Copyright (C) 2011 Marc Espie <espie@openbsd.org>
|
|
|
|
Permission to use, copy, modify, and distribute this software for any
|
|
purpose with or without fee is hereby granted, provided that the above
|
|
copyright notice and this permission notice appear in all copies.
|
|
|
|
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
|
|
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
|
|
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
|
|
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
|
|
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
|
|
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
|
|
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
|
|
|
|
|
This file is *not* user documentation of the *mk files. The full user
|
|
documentation is available as manpages such as bsd.port.mk(5)
|
|
|
|
There are comments in bsd.port.mk but this file is already too long,
|
|
so I finally decided to put the design notes in a separate file.
|
|
|
|
Most of this is not for the faint of heart.
|
|
|
|
"Imagination could conceive almost anything in connexion with this place."
|
|
(H.P. Lovecraft, At the Mountains of Madness)
|
|
|
|
Some design principles
|
|
----------------------
|
|
- all variables and targets prefixed with _ are for internal use.
|
|
It may be that some other tools reuse them, but that's generally following
|
|
the same guidelines: no user serviceable parts inside.
|
|
|
|
- most variables will always be defined, even with an empty value. This avoids
|
|
stupid .if defined(TESTS).
|
|
|
|
- bsd.port.mk has a strict separation between variable definitions (at top)
|
|
and target definition (right after the
|
|
###
|
|
### end of variable setup. Only targets now
|
|
###
|
|
). This is because of make's "lazy" way to evaluate things: constructs in
|
|
shell-lines get evaluated very very late. But all targets (such as
|
|
${_WRKDIR_COOKIE}) get evaluated when they're encountered. There was some
|
|
significant effort in the early days when I took bsd.port.mk over to achieve
|
|
that separation. Be very careful with Makefile.inc and modules, since those
|
|
get included at a very specific location in the Makefile. They should mostly
|
|
define variables. Makefile.inc is included very early so that it can override
|
|
about anything it should be able to (but then, nothing is already defined
|
|
except for read-only variables / global user-tweakable defaults). Modules
|
|
is included a little bit later...
|
|
|
|
- a lot of the code is done through shell fragments. All of these are
|
|
internal and prefixed with _. This is done for speed and compactness (forking
|
|
an external shell would be very slow, and a total nightmare with respect
|
|
to parameter passing). These often communicate through internal shell
|
|
variables with well-defined names.
|
|
|
|
- the infrastructure tries very hard to have real files (as far as possible)
|
|
attached to most targets: "make package" points to the actual pkgfile(s)
|
|
being generated. There's a whole section of the makefile devoted to
|
|
targets generated from the distfile devoted to fetch and fetch-all
|
|
(see the section around _EVERYTHING for variable definition, and the
|
|
# Separate target for each file fetch-all will retrieve
|
|
for the actual targets). Apart from that, once a port starts building,
|
|
targets "without" an actual file will generate a cookie under WRKDIR or
|
|
some subdir of it (see all the _COOKIE* definitions). Of particular interest
|
|
are dependencies, where the cookie file is generated depending on the
|
|
name of the dependency: that's the infamous
|
|
.for _i in ${_DEPLIST}
|
|
${WRKDIR}/.dep-${_i:C,>=,ge-,g:C,<=,le-,g:C,<,lt-,g:C,>,gt-,g:C,\*,ANY,g:C,[|:/=],-,g}: ${_WRKDIR_COOKIE}
|
|
loop. The reason for that complexity is that this avoids rechecking
|
|
dependencies while working on a port, since only the new stuff will need
|
|
evaluation...
|
|
In general, make works really bad when you write
|
|
phony-target1: phony-target2
|
|
@do_some_shit
|
|
as phony targets are always out-of-date, and thus you will always do_some_shit.
|
|
|
|
Introspection
|
|
-------------
|
|
There's a lot of stuff in the infrastructure which is there only for
|
|
introspection purposes: show, show-all, print-plist-*, dump-vars,
|
|
*-dir-depends, print-package-signature (ill-named, since it actually only
|
|
asks the ports tree for information).
|
|
They allow external tools to interrogate the ports tree and get most
|
|
information from it. dpb and sqlports rely very heavily on it.
|
|
|
|
Significant work has been done to achieve better MI. The MULTI_PACKAGES
|
|
code now assume you will define MULTI_PACKAGES *independently of the
|
|
architecture, annotate subpackage with ONLY_FOR_ARCHS-sub if appropriate,
|
|
the include bsd.port.arch.mk (which was split off the main file specifically
|
|
for that purpose), and rely on BUILD_PACKAGES for building.
|
|
|
|
Shell tricks and constructs
|
|
---------------------------
|
|
- IFS is your friend: echo "a:b:c"| while IFS=: read a b c
|
|
will split things along any character without requiring external commands
|
|
such as cut.
|
|
|
|
- don't run test for checking variable values. Rather:
|
|
case "$$subdir" in *,*) ... ;; *) ... ;; esac
|
|
is entirely internal
|
|
|
|
- () forks a subshell. If you need to syntactally keep a list of commands
|
|
together, "{ cmd1; cmd2; }" is the way to do it.
|
|
|
|
- use trap to clean up at the end of a command. That's mostly used for
|
|
caching stuff (depfile and cache), but also for the locking mechanism.
|
|
|
|
- exec is often used for tail recursion: it replaces the shell with the
|
|
executed command. BUT beware of pending traps, as you will lose them.
|
|
|
|
- all of make commands are executed under -e. Thus, it's deadly to fail
|
|
without a test around it. In the end, you might end up with:
|
|
if ( eval $$toset exec ${MAKE} subupdate ); then
|
|
this forks a subshell to avoid the unpleasantness of -e, then execs the
|
|
resulting command to avoid piling up an extra process.
|
|
|
|
Make vs. Shell
|
|
--------------
|
|
There are at least *4* kind of variables in those files:
|
|
- internal shell variables
|
|
- environment variables
|
|
- make variables set on the command line
|
|
- make variables set within a makefile
|
|
|
|
Note that make variables set on the command line ARE set in stone: it's very
|
|
difficult to change them from within the Makefile, and they will override
|
|
mostly anything you can do. They also appear in the environment.
|
|
|
|
Thus, a lot of make thingies, such as FLAVOR and SUBPACKAGE must be set in
|
|
the environment because make will run into subdirs and requires them to
|
|
be changeable.
|
|
|
|
Also, note that stuff set within Makefiles is not exported to the environment,
|
|
you have to be explicit and set them when you run a command.
|
|
|
|
When things get ambiguous, This document uses $$var for a shell variable,
|
|
${VAR} for a variable set in make, and VAR for an environment variables.
|
|
|
|
pkgpath.mk
|
|
----------
|
|
Named that way because it mostly deals with pkgpath parsing, but in reality,
|
|
it's mostly a lift-up from the common parts between bsd.port.mk and
|
|
bsd.port.subdir.mk (no reason for a name change so late in the game)
|
|
|
|
fragments and common shell variables
|
|
------------------------------------
|
|
_pflavor_fragment:
|
|
takes $$subdir as a pkgpath parameter.
|
|
will parse it into a $$dir and $$toset variables, perform a cd $$dir
|
|
so that eval $$toset ${MAKE} gets you into that
|
|
dependency (this will set PKGPATH, FLAVOR, SUBPACKAGE accordingly)
|
|
|
|
depending on $$_fullpath (true/false), will consider the absence
|
|
of flavor to be an empty flavor (e.g. $$toset contains FLAVOR='' )
|
|
or to be the default flavor (e.g. unset FLAVOR)
|
|
|
|
takes ${PORTSDIR_PATH} into account
|
|
|
|
succeeds if the dependency is correct and the directory is found.
|
|
Otherwise will emit an error message that ends with $$extra_msg.
|
|
|
|
_flavor_fragment:
|
|
calls ${_pflavor_fragment} after imposing $$_fullpath=false. e.g., for
|
|
actual dependencies.
|
|
|
|
_depfile_fragment:
|
|
sets _DEPENDS_FILE to a temporary file in the environment if it's
|
|
not already set. Set a trap to remove it on end.
|
|
_DEPENDS_FILE is used by the *dir-depends and show-run-depends targets
|
|
to not go thru the same directory twice. Beware of setting it
|
|
"too globally", since it prevents directory walking from happening
|
|
twice.
|
|
|
|
_cache_fragment:
|
|
sets _DEPENDS_CACHE to a temporary directory in the environment
|
|
if it's not already set. Set a trap to remove it on end.
|
|
_DEPENDS_CACHE is used to store library lists in _libs2cache.
|
|
pkg_create(1) also knows about it and use it to store plists.
|
|
the ${SUDO} part in the trap is because some of the caching happens
|
|
during ${SUDO} pkg_create.
|
|
XXX also sets PKGPATH, as it reduces drastically the number of
|
|
PKGPATH=... needed in bsd.port.mk
|
|
|
|
Locking
|
|
-------
|
|
Done thru _DO_LOCK and _SIMPLE_LOCK mostly. Most user-visible targets
|
|
redirect through
|
|
${_DO_LOCK}; make _internal-$@
|
|
as soon as locking is used (which is the default these days).
|
|
The locking mechanism carefully maintains the _LOCK_HELD environment variable
|
|
so that dependencies can relock themselves (yeah, it looks strange, but this
|
|
happens !).
|
|
|
|
_SIMPLE_LOCK only exists to provide separate locking targets for fetch: it
|
|
assumes $$lock has been set to an appropriate lock file name.
|
|
|
|
|
|
Internal targets are:
|
|
_internal-clean _internal-package-only _internal-run-depends
|
|
_internal-runlib-depends _internal-runwantlib-depends _internal-package
|
|
_internal-build-depends _internal-buildlib-depends
|
|
_internal-buildwantlib-depends _internal-regress-depends
|
|
_internal-prepare _internal-depends _internal-fetch-all
|
|
_internal-fetch _internal-all _internal-build _internal-checksum
|
|
_internal-configure _internal-deinstall _internal-extract _internal-fake
|
|
_internal-patch _internal-plist _internal-regress _internal-subpackage
|
|
_internal-subupdate _internal-uninstall _internal-update
|
|
_internal-update-or-install _internal-update-or-install-all
|
|
_internal-update-plist _internal-distpatch
|
|
(yeah, that's a lot).
|
|
|
|
So, if you type "make", it will do
|
|
make build -> lock && make _internal-all
|
|
|
|
then make _internal-all will normally run thru
|
|
_internal-fetch _internal-extract _internal-patch _internal-build
|
|
without ever releasing the lock, thus preventing anything else to sneak
|
|
in and break the build.
|
|
|
|
Misc
|
|
----
|
|
- Modules inclusion is done through a separate modules.port.mk to handle
|
|
recursion: modules may trigger the inclusion of other modules, thus
|
|
modules.port.mk will re-include itself until the whole list is done.
|
|
|
|
- MULTI_PACKAGES and SUBPACKAGE always get set to a non-empty value:
|
|
if MULTI_PACKAGES is not set, it ends up as MULTI_PACKAGES=- and
|
|
SUBPACKAGE=-. This does make loops over all subpackages simpler, e.g.,
|
|
.for _s in ${MULTI_PACKAGES} since make doesn't do anything if the list
|
|
is empty...
|
|
As a result, most subpackage-dependent variables end up being used as their
|
|
"true" VAR- form in the case of a port without multi-packages.
|
|
Exception: FULLPKGNAME needs some specific code for the !multi-packages case.
|
|
|