108 Commits

Author SHA1 Message Date
espie
2cf36ad340 make HISTORY_ONLY work again... I managed to break it twice :) 2014-04-28 12:51:41 +00:00
espie
a384b0d3a8 my tracer used lexical $_ 2014-03-17 10:49:19 +00:00
espie
bf709a9acc tweak waiting-for-lock mechanism: explicitly keep track of order,
and have each job relinquishing lock wake up the next one.

To cater for external dpbs, also try to wake up next job if we don't own the
lock.
2014-03-15 09:51:27 +00:00
espie
c8571e2c0f restore BUILD_ONCE optimization for full ports builds 2014-03-09 19:47:35 +00:00
espie
dad88b1382 restore tracing, unintentional removal 2013-10-18 07:48:09 +00:00
espie
b2de3fa518 simplify. The limiter does a good enough job of throttling the display,
so we can try to report every time, independently of the clock job.
2013-10-18 07:45:41 +00:00
espie
8877767529 - use separated Heuristics::Size object
- choose size/nosize in Config.
- use affinity or affinitystub
- move locker creation to State, so we can init cores right away.
- as core init has run, we can call DPB::Core->reap (event loop) while
processing size and build stats, so that the startup script may start
slightly earlier.
- setup more options in Config for Reporter to use.
- scaffolding in SubEngine/Build to be able to use affinity info right away
2013-10-13 18:31:50 +00:00
espie
f4bfcdb01d trying to figure out the spinning... 2013-10-11 10:26:44 +00:00
espie
722d74cc49 this should work around the spinning loop in fetch.
I still don't understand how it happens, since the queue
ought to be empty ?
2013-10-09 06:21:38 +00:00
espie
28006064d8 fix -F mode... subengine not yet ready to be split,
and we DON'T want an extra build job.
2013-10-06 15:49:22 +00:00
espie
0f33e01c84 special-case single core/single job: create a clone so that we start
building stuff in parallel with LISTING, and fold back LISTING at end.
2013-10-06 14:03:07 +00:00
espie
b46cee4c56 update my copyright to 2013 2013-10-06 13:33:24 +00:00
espie
4a5c4d7329 clean-up code a bit, move state into its own file 2013-10-02 09:00:34 +00:00
espie
e804dca486 use a "shell" to run getvars, so that we can run it again LATER. 2013-09-18 15:30:40 +00:00
espie
63daf24a0e reorganize code, so the periodic sleeper task no longer knows
its sleeping interval, but rather asks it from the reporter.

Give limiter customer access to timing information. Eventually,
the reporter could possibly shrink/enlarge the display timeout
automatically.
2013-09-03 09:44:05 +00:00
jasper
8e89fb0681 allow dpb to install packages with a non-standard localbase
ok espie@
2013-07-04 17:45:34 +00:00
espie
b50533eff0 disentangle the config/init code from the rest.
- move the meat of handle_options from dpb into config->parse_command_line
(this means a backcall to still inherit from OpenBSD::State).
- move parse_config_files from core into config.
- move the prop handling into proper HostProperties (part of config
obviously)
- create a Core::Init file that contains all the former DPB::Host::Factory
and associated jobs.

there's still a wee little bit of cleanup to do, but this should be
easier to maintain, as all option handling is now in one place, and
startup and host confi is now easier to figure out.

- add a -DMIRROR=0/1  setup that controls whether SUPDISTFILES will be
fetched (defaults to 1 for -F and 0 for -f).

- actually allow for several host files to be parsed, as the name implies
2013-06-21 09:05:18 +00:00
espie
8bc6278431 -DSYSLOG will syslog most activity from jobs starting/ending.
THIS IS A VERY BUSY OPTION. It's intended with a syslog.conf that
will send the log messages elsewhere, to try to pinpoint hang/panic
locations in a given port build, as system crash failures tend to leave
log files in a lagging state.

idea discussed with kurt@
2013-06-04 03:20:36 +00:00
espie
e71445259f tweak 'dump-vars' run to include TEST values if run with -t.
THIS REQUIRES MOST RECENT bsd.port.mk, OTHERWISE THIS WILL BREAK BADLY.

The values are not used yet, but I fully intend to make it possible to
run non-regression tests in the not too distant future.
2013-05-21 20:44:08 +00:00
espie
d339778a36 dump all the paths with fishy behavior at end of equiv.
this is a bit too much, but this catches:
SUBDIR=devel/py-gobject3,python3 make dump-vars DPB=Yes
devel/py-gobject3,python.BUILD_DEPENDS=graphics/py3-cairo STEM->=0.10.38:devel/gettext lang/python/3.2 STEM->=0.41.1p0:textproc/intltool devel/gmake archivers/xz
devel/py-gobject3,python.IS_INTERACTIVE=No
devel/py-gobject3,python.SUBPACKAGE=-main
devel/py-gobject3,python.FLAVOR=python3
devel/py-gobject3,python.BUILD_PACKAGES= -main -common
devel/py-gobject3,python.FULLPKGNAME=py3-gobject3-3.8.1p0
devel/py-gobject3,python.RUN_DEPENDS=STEM->=0.10.38:devel/gettext lang/python/3.2 devel/py-gobject3,-common graphics/py3-cairo
devel/py-gobject3,python.LIB_DEPENDS=STEM->=0.10.38:devel/gettext converters/libiconv devel/gobject-introspection
devel/py-gobject3,-common.BUILD_DEPENDS=graphics/py3-cairo STEM->=0.10.38:devel/gettext lang/python/3.2 STEM->=0.41.1p0:textproc/intltool devel/gmake archivers/xz
devel/py-gobject3,-common.IS_INTERACTIVE=No
devel/py-gobject3,-common.SUBPACKAGE=-main
devel/py-gobject3,-common.FLAVOR=python3
devel/py-gobject3,-common.BUILD_PACKAGES= -main -common
devel/py-gobject3,-common.FULLPKGNAME=py-gobject3-common-3.8.1
devel/py-gobject3,-common.RUN_DEPENDS=STEM->=0.10.38:devel/gettext lang/python/3.2

(note the "wrong" flavor, which means that
devel/py-gobject3,python3,-main ends up without an associated fullpkgname)
2013-04-22 10:06:12 +00:00
espie
99c73a5b6c fix die handler, as it should display the message as well...
grumble

make it possible to temporarily disable those handlers.
2013-04-13 10:02:11 +00:00
espie
79fa504e3a rework INFO/DIE/WARN handler to display some info in a non-opaque way. 2013-03-03 11:11:08 +00:00
espie
47036c8aec tweak the carp stuff to not recurse into itself, so that compile errors
show up correctly again.

split start_cores into a read_config/start_cores part, so that read_config
can happen right after handle_options, and so that we can put more options
in the hosts file (to be done)
2013-03-03 00:51:57 +00:00
espie
eef793741e some of args seen in trace might be undef'd. avoid triggering the WARN
handler while handling SIG{INFO}.
Also, have parse_size_info be verbose about what's going on.
2013-02-04 09:43:57 +00:00
espie
44ac9e61b7 move control of the "building in mfs" part under PortsBuilder/Heuristics.
Store previous pkgname.
do show-size only sporadically if fullpkgname didn't change
2013-02-02 13:35:17 +00:00
espie
12f4db34e8 allow for size log to contain pkgpath(pkgname) instead of just pkgpath.
make -S useful again: allow it to "prime" the log from an existing source,
but don't write back to it.
2013-02-02 10:36:31 +00:00
espie
573c499cf5 add a FETCH_TIMEOUT, similar to STUCK_TIMEOUT, but slightly simpler,
as there is no speed factor there...
2013-01-28 12:23:45 +00:00
espie
11cd4c0c77 don't bother dumping distfiles, history is quite enough 2013-01-28 10:50:46 +00:00
espie
c259d653cc tweak memory handling to make it less special: treat -M as an override_prop
as we should.
Use core methods to access the memory threshold.

UI simplification: -M can take a suffix, default is K, but you can just say
-M 520M or -M 2G now.
2013-01-28 10:14:17 +00:00
espie
cd3023d079 add debugging message that does not depend on Devel::StackTrace, may be
useful to find out where dpb gets stuck
2013-01-28 09:56:00 +00:00
espie
2f7c8bf00b bump default junk back up to 150, makes more sense with new accounting 2013-01-27 23:15:57 +00:00
espie
0a438fe138 prompted by jasper: reintroduce -S, but as a rw option, to override
the rolling journal location for size log.

make it possible to share dpb -s output from a given architecture with
similar architectures
2013-01-21 14:50:30 +00:00
espie
9ad81d2390 simplify wrkdir size handling, turn that into a rolling log. 2013-01-21 12:03:32 +00:00
espie
647327b6ba don't go thru patch/configure/build for very small ports. 2013-01-10 12:27:21 +00:00
espie
d8408ea93f fix a long-standing stupid bug. Now I know why weight changes didn't matter,
I wasn't passing them through, sigh.

Add new -X option that sets xpaths, to be used...
2013-01-07 10:57:31 +00:00
espie
8a23d3ca7e ask the ports tree to clean up stale locks, finally... 2013-01-06 11:59:40 +00:00
espie
209b83159b allow independent setup for the locks location. I strongly suggest
using a local fs.
2013-01-05 18:26:31 +00:00
espie
779bfd4ae7 fix typo 2013-01-05 15:13:48 +00:00
espie
116acde577 change to NO_HISTORY and document (since there are other NO_* thingies) 2013-01-05 13:34:28 +00:00
espie
85b554d119 make it possible to skip "UPDATING DISTFILES" if we're just running dpb
again without any change.
2013-01-05 13:32:28 +00:00
espie
9eb2e2c9df fix typo. Symptom: errors out at end of build
noticed by rpe@
2013-01-04 18:29:13 +00:00
espie
9a2e664c54 force report display to fire whenever some new ports get built. 2013-01-04 12:46:20 +00:00
espie
a1d376fbf6 take the rate-limiting concept used for the engine, and make it available
elsewhere. Also use it for the Reporter, as it makes no sense to spend
THAT much time reporting quick changes, which actually slows the build.
($factor to tweak as needed).
2013-01-04 12:06:25 +00:00
espie
c39d9434ff fix infamous "doesn't really quit when it's done" stop bug.
also move the report to the main package, and show a prominent
STOPPED in the title bar if you forget you stopped it during a
previous run ('why is my dpb not building anything ?')
2013-01-03 15:43:27 +00:00
espie
7adbbd9561 remove optimization that no longer makes sense. 2012-12-28 06:40:11 +00:00
espie
e4c671527c add affinity information to restart jobs on the right host preferentially.
- affinity info is similar to locks, but with a completely different
lifetime.
- streamline the main loop of the engine, so that it can do two passes:
first pass shuns paths with the wrong affinity. If no good path is found,
those are considered during the second pass.
- make the Core factory aware of what hosts might be running, so that
affinity info for machines removed from a config file will be ignored.

thanks to landry@ for a few tests.
2012-12-24 17:24:46 +00:00
espie
ba8f355da8 more descriptive name 2012-11-18 12:37:21 +00:00
espie
7fafb5c793 allow for -DWAIT_TIMEOUT=0 to propagate correctly
lower default for junk
2012-11-18 01:58:36 +00:00
espie
d73a931704 don't mix up wait_timeout and waiting_timeout, as noticed by naddy@.
re-enable wait_timeout on localhost temporarily (should be done in another
way, most probably by checking whether repo is on nfs, we can steal code
from VStat.pm)

pass umask through ssh. This took us long enough to figure out, and it's
considerably simpler than tweaking every login class once again.
2012-11-06 08:26:29 +00:00
espie
04618136d2 fix default property handling 2012-10-13 09:00:02 +00:00