Commit Graph

3270 Commits

Author SHA1 Message Date
espie
85b554d119 make it possible to skip "UPDATING DISTFILES" if we're just running dpb
again without any change.
2013-01-05 13:32:28 +00:00
espie
492a2898b1 fix thinko.
as noticed by
Mikolaj Kucharski.
2013-01-05 07:01:44 +00:00
espie
df5a288315 make sure we zap any lingering socket before we restart our master. 2013-01-04 21:19:18 +00:00
espie
a9b8de6833 switch to IO::File and explicit flush for the engine log.
should be slightly faster
2013-01-04 19:34:10 +00:00
espie
d606cc382f another half-baked change 2013-01-04 18:49:39 +00:00
espie
9eb2e2c9df fix typo. Symptom: errors out at end of build
noticed by rpe@
2013-01-04 18:29:13 +00:00
espie
5b5a52a451 synch manpage with reality 2013-01-04 12:56:26 +00:00
espie
7c00e154b6 ... and don't forget to propagate it thru 2013-01-04 12:49:06 +00:00
espie
9a2e664c54 force report display to fire whenever some new ports get built. 2013-01-04 12:46:20 +00:00
espie
9612257de1 tweak limiter output some more. offset display is not really needed,
but knowing about force computes might.
2013-01-04 12:45:44 +00:00
espie
66ec642cc7 remove unneeded scaffold 2013-01-04 12:34:29 +00:00
espie
9d015892e8 document new exhaustive rate-limiting 2013-01-04 12:16:51 +00:00
espie
a1d376fbf6 take the rate-limiting concept used for the engine, and make it available
elsewhere. Also use it for the Reporter, as it makes no sense to spend
THAT much time reporting quick changes, which actually slows the build.
($factor to tweak as needed).
2013-01-04 12:06:25 +00:00
espie
3db19612eb forgot that one commit for nojunk. Oops 2013-01-04 12:03:06 +00:00
espie
5e21b7b72c explain rationale and working of unjunk 2013-01-03 22:52:01 +00:00
espie
d17cfb86a3 tweak the Engine some more: don't bother adjusting distfiles in a loop
while dependencies are missing

Following landry's remark, also take packages to build out of the race
if some RUN_DEPENDS are going to be ignored. Only do it right before we
put the package in the queue, so that the test is run exactly once per
package instead of during every scan.

I was also worried about multi-packages, but this only takes one fullpkgpath
out of the loop, and "make package" is going to fail half-way through anyways.
2013-01-03 15:47:45 +00:00
espie
de678efd4a add support for unjunk property, that will prevent junking from actually
happening while some 'unjunk' ports are building.

this is a work-around for a known problem with cmake and qt4 include
dependency handlers...

Also, cache fullpkgpath while building a job, as this contributes for
a large part to the speed (not) of the display when building lots of
ports.
2013-01-03 15:45:18 +00:00
espie
c39d9434ff fix infamous "doesn't really quit when it's done" stop bug.
also move the report to the main package, and show a prominent
STOPPED in the title bar if you forget you stopped it during a
previous run ('why is my dpb not building anything ?')
2013-01-03 15:43:27 +00:00
espie
59ee0568e8 stats are comparatively cheap, run them even when the engine doesn't run.
Yields more accurate stat output, where we can see a staircase effect on
the queue when the engine is run not that often, and Built actually go
up until it goes back down again.
2013-01-02 11:13:52 +00:00
espie
f86dbb772a allow overriding the start-up script from the command line. 2012-12-31 09:50:55 +00:00
espie
092496c30a following landry's suggestion, add module hooks for post-install
I'm not too happy about stuff that will remove .la automatically though,
seems to me this shouldn't be generated in the first place.

ETOOMUCHMAGIC ?
2012-12-31 09:14:07 +00:00
espie
61c2611d5c document extra log files... 2012-12-31 08:29:31 +00:00
espie
dcf231188f forgot to GC this along with the imake weirdness.
seen by landry@
2012-12-30 20:40:40 +00:00
espie
d985cc05e2 mismatches are more annoying than extra cpu:
if we end up with only mismatches and we didn't run check_buildable last
time around, try a check_buildable first.
2012-12-30 14:33:46 +00:00
espie
45b128b893 fix elusive "jobs don't show up where they should".
duh. put "live" affinity markers while we're building stuff.
We don't re-read the on-disk markers outside of restart (should we ?)
so they HAVE to be in the pkgpath struct proper.
2012-12-30 14:04:17 +00:00
espie
eb77a6071b why not ? track total distsize for each port. might correlate with other
stuff later.
2012-12-30 11:47:24 +00:00
espie
ad8ad1c28f smarter affinity mismatch handling: if we have other cores idle, ask them.
also, officialize the engine-perf log somewhat (and give it a more descriptive
name).
2012-12-29 19:15:37 +00:00
espie
fa169c69ae allow looking for a given core when trying to match affinity 2012-12-29 19:14:51 +00:00
espie
ebb2a2c90a on start-up, log affinity in a more consistent way, with pid of running dpb. 2012-12-29 19:14:28 +00:00
jasper
fb9809c424 cleanup GCC, XCONTRIB, SUNSITE and GNOME mirror lists. 2012-12-29 07:58:03 +00:00
jasper
e60a243cf3 *** empty log message *** 2012-12-29 07:42:37 +00:00
espie
c3a8df2e5f log the markers we retrieve on any restart... I suspect something foul
happening with localhost.
2012-12-28 21:09:45 +00:00
espie
7adbbd9561 remove optimization that no longer makes sense. 2012-12-28 06:40:11 +00:00
espie
d624d61917 since we have a quick path where stuff move from tobuild to install without
adjusting things, stuff that can be built may have BEXTRA which are not
completed...
2012-12-28 06:31:03 +00:00
espie
3ca05b12fe replace next_check = last_check + check_interval
tweak the algorithm slightly (since we forget the old check_interval).
In particular, never keep old times in reserve.

Makes for simpler and clearer reading
2012-12-27 16:23:17 +00:00
espie
e293bae75a remove the need to loop around adjust_tobuild, fudge things a bit:
Most things will move as a result of {install} changes.

Things that move because of EXTRA depends that are satisfied are unlikely to trigger
further changes.

So, stagger changes for "normal" tobuild -> queue first, and extra depends later.
2012-12-27 14:14:19 +00:00
espie
99ac03e94d oops 2012-12-27 12:20:36 +00:00
espie
fc5ee33b62 add timestamp in phases for Job/Port.pm.
may help synchronize events from distinct logs in case this matters.
2012-12-27 12:03:33 +00:00
espie
7614ffa97a remove 'dead' cycles: changes in ToBuild can't trigger new changes in Built. 2012-12-27 11:04:43 +00:00
espie
98849d8242 self-adaptive delay... with a bit of log scaffolding for now. 2012-12-25 20:41:41 +00:00
espie
67de0deb11 if we have a sizeable queue (>50) and we ran check_buildable less than 30s
ago, don't run it again right now.

This prevents the Engine from busy-checking the same data when it's building
lots of small packages very fast.

(50 and 30 may need some fine-tuning)
2012-12-25 16:07:02 +00:00
espie
11cb77656d make "rebuild" behavior a proper subclass of DPB::PortBuilder, instead
of having tests all over the place.

Take advantage of that to not rechecking signatures if already done.
2012-12-25 10:43:36 +00:00
espie
66b2639394 compute dependencies right away in the job: we don't actually have to
fork depends/prepare/prepare-results for ports which don't have any
depends.

This also prevents some lock contention, obviously.
2012-12-25 10:25:04 +00:00
espie
1af810c8d6 on rebuild, mark all existing files as 'uptodate'. Will prevent extra
rebuilds on @option always-update multi-packages when using -R.
2012-12-25 10:00:08 +00:00
espie
3de193b931 also take things off the later list if they were there. Prevents behavior
like the following:

14426@1356424364: B: x11/kde/graphics3
14426@1356424364: B: x11/kde/graphics3,-kpov
14426@1356424364: B: x11/kde/graphics3,-kamera
14426@1356424364: B: x11/kde/graphics3,-sane
14426@1356424364: B: x11/kde/graphics3,-main
14426@1356424364: V: x11/kde/graphics3,-kpov	<--- note put back on queue
14426@1356424364: V: x11/kde/graphics3
14426@1356424364: V: x11/kde/graphics3,-kamera
14426@1356424364: I: x11/kde/graphics3,-sane
14426@1356424364: I: x11/kde/graphics3
14426@1356424364: I: x11/kde/graphics3,-kamera
14426@1356424364: I: x11/kde/graphics3,-main
14426@1356424366: J: graphics/povray echo.home 0
14426@1356424496: B: devel/goffice
14426@1356424496: I: devel/goffice
14426@1356424496: Q: math/gnumeric
14426@1356424498: B: x11/kde/graphics3		<--- so, duh rebuilt...
14426@1356424498: B: x11/kde/graphics3,-main
14426@1356424498: B: x11/kde/graphics3,-kamera
14426@1356424498: B: x11/kde/graphics3,-sane
14426@1356424498: J: graphics/opencv,-docs echo.home 0
14426@1356424675: B: graphics/povray
14426@1356424675: I: x11/kde/graphics3,-sane
14426@1356424675: I: graphics/povray
14426@1356424675: I: x11/kde/graphics3		<-- and installable, again.
14426@1356424675: I: x11/kde/graphics3,-kpov
14426@1356424675: I: x11/kde/graphics3,-kamera
14426@1356424675: I: x11/kde/graphics3,-main
2012-12-25 09:46:24 +00:00
espie
130df9a3bc will work better with a non-reversed test... 2012-12-25 09:31:54 +00:00
espie
e4c671527c add affinity information to restart jobs on the right host preferentially.
- affinity info is similar to locks, but with a completely different
lifetime.
- streamline the main loop of the engine, so that it can do two passes:
first pass shuns paths with the wrong affinity. If no good path is found,
those are considered during the second pass.
- make the Core factory aware of what hosts might be running, so that
affinity info for machines removed from a config file will be ignored.

thanks to landry@ for a few tests.
2012-12-24 17:24:46 +00:00
espie
0ff5bc966a be slightly more aggressive when a pkgpath is done: for multi-packages,
zap all subpackages from the queue.

need to distinguish the normal case (is_done) with a simpler case
(is_done_quick) that's used when scanning the ports tree. Otherwise, we
would check multi-packages loads of time.
2012-12-24 17:22:15 +00:00
espie
c4f1bfc74e reorg the tests slightly, no functional change 2012-12-24 17:19:01 +00:00
espie
027e6592d6 testing option 2012-12-23 13:28:55 +00:00