Personal patches
778518d745
ordering, which had become too limited. We now build packages ordered by those that are part of the longest dependency chains first. This has the effect of building the deepest parts of the tree first and levelling out the tree height, hopefully avoiding the situation we currently face where there appear bottlenecks late in the build where the cluster becomes mostly idle while waiting for a few long dependency chains to finish building before the cluster can become fully loaded again. The algorithm is that we sort the list of remaining packages according to height (longest dependency chain), then add leaf packages from each in order until we have filled a queue of length between 100 and 200, to amortise the cost of this queue rebalancing while not losing the height averaging property. Jobs are dispatched from this queue into worker threads as machine slots become available. Unlike the make-based solution that required a fixed -j concurrency value and could not respond to addition/removal of build resources, we now can dynamically add new machines as they become available to the queue. The other advantage of using python is that we have more customisability and visibility into the build status, e.g. we periodically report the number of remaining packages, as well as the list of deepest packages that we are working on. TODO: * Implement mtime checking for parent package staleness, so that parents are rebuilt if the dependencies are touched more recently. Currently packages will not be rebuild if they exist, whether or not they are "stale" wrt their dependencies. * Offload the machine selection into an external queue manager. Currently the queue manager used here doesn't interoperate with the old one (getmachine/releasemachine) because it's not possible to use the lockf()-based mutual exclusion within a multithreaded client. Doing that will also allow for a more flexible job placement algorithm as well as finer queue customization. |
||
|---|---|---|
| accessibility | ||
| arabic | ||
| archivers | ||
| astro | ||
| audio | ||
| benchmarks | ||
| biology | ||
| cad | ||
| chinese | ||
| comms | ||
| converters | ||
| databases | ||
| deskutils | ||
| devel | ||
| dns | ||
| editors | ||
| emulators | ||
| finance | ||
| french | ||
| ftp | ||
| games | ||
| german | ||
| graphics | ||
| hebrew | ||
| hungarian | ||
| irc | ||
| japanese | ||
| java | ||
| korean | ||
| lang | ||
| math | ||
| mbone | ||
| misc | ||
| Mk | ||
| multimedia | ||
| net | ||
| net-im | ||
| net-mgmt | ||
| net-p2p | ||
| news | ||
| palm | ||
| polish | ||
| ports-mgmt | ||
| portuguese | ||
| russian | ||
| science | ||
| security | ||
| shells | ||
| sysutils | ||
| Templates | ||
| textproc | ||
| Tools | ||
| ukrainian | ||
| vietnamese | ||
| www | ||
| x11 | ||
| x11-clocks | ||
| x11-drivers | ||
| x11-fm | ||
| x11-fonts | ||
| x11-servers | ||
| x11-themes | ||
| x11-toolkits | ||
| x11-wm | ||
| .cvsignore | ||
| CHANGES | ||
| COPYRIGHT | ||
| GIDs | ||
| KNOBS | ||
| LEGAL | ||
| Makefile | ||
| MOVED | ||
| README | ||
| UIDs | ||
| UPDATING | ||
This is the FreeBSD Ports Collection. For an easy to use WEB-based interface to it, please see: http://www.FreeBSD.org/ports For general information on the Ports Collection, please see the FreeBSD Handbook ports section which is available from: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/ports.html for the latest official version or: The ports(7) manual page (man ports). These will explain how to use ports and packages. If you would like to search for a port, you can do so easily by saying (in /usr/ports): make search name="<name>" or: make search key="<keyword>" which will generate a list of all ports matching <name> or <keyword>. make search also supports wildcards, such as: make search name="gtk*" For information about contributing to FreeBSD ports, please see the Porter's Handbook, available at: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/porters-handbook/ NOTE: This tree will GROW significantly in size during normal usage! The distribution tar files can and do accumulate in /usr/ports/distfiles, and the individual ports will also use up lots of space in their work subdirectories unless you remember to "make clean" after you're done building a given port. /usr/ports/distfiles can also be periodically cleaned without ill-effect.