Commit Graph

283 Commits

Author SHA1 Message Date
espie
aa851e66b1 keep a trace of "small" ports: we won't log every detail concerning
these
2013-01-10 12:05:55 +00:00
espie
1d0a9bd7bb say goodbye to prepare/show-prepare-results,
there's no need for a separate step: show-prepare-results should be
practically instantaneous, and if there's noise, prepare will exit(1)
anyways.
just need to explicitly log the output of prepare.
2013-01-10 12:04:23 +00:00
espie
2abd089617 there's no need to fork a process if the checksums are all cached 2013-01-10 12:00:38 +00:00
espie
d4a9b9d633 use new functions to simplify 2013-01-10 11:56:53 +00:00
espie
dd11c9037a deleting DIST from child is pointless, do a proper finalize 2013-01-10 10:35:36 +00:00
espie
48c91e1d7f 'memoryhog' property: avoid starting two such jobs on the same host, if
possible (mostly to be used for moz stuff which gobbles huge amounts of
memory while linking)
2013-01-10 10:30:13 +00:00
espie
e2925ad3b7 zap info DIST if we don't actually need it. 2013-01-10 10:28:29 +00:00
espie
af0beabf88 a bit of refactoring: provide simple ways to check DPB_PROPERTIES without
having a hash element spring into existence, and to list jobs running on
the same host.
2013-01-10 10:26:34 +00:00
espie
ba805a9329 small optimization, fullpkgpath is expensive, and ordering by pkgpath is
vastly enough in the current case
2013-01-10 10:25:25 +00:00
espie
d0f75703ed pass -X paths thru:
- the grabber passes them to PortBuilder
... which builds a hash of pkgnames
... and the port uninstall job excludes these from the list of ports to
junk.

Note this only works with -current pkg_delete, as I had to tell it to
ignore non-existent pkgnames in that context
2013-01-07 10:59:41 +00:00
espie
7906ac8797 if I run into lockfiles that don't contain the requisite dpb property,
just hold on until they're there.

Should happen right away, so easy to spot !
2013-01-06 14:38:14 +00:00
espie
8a23d3ca7e ask the ports tree to clean up stale locks, finally... 2013-01-06 11:59:40 +00:00
espie
a8dab3a559 perl's gc doesn't play well with filehandles.
So explicitly close what it will have problems tracking...

problem noticed by sthen@, obviously linked to my previous change to
avoid opening the same log file again and again...
2013-01-05 23:38:08 +00:00
espie
a323bb36ea better 2013-01-05 21:48:50 +00:00
espie
c1c2864b82 give every task a "name". 2013-01-05 21:48:18 +00:00
espie
fe8936deab even if we've got pre-empted, we still have to lock.
otherwise, we're going to unlock something we don't own, which might
be a race.
2013-01-05 20:08:39 +00:00
espie
a5eeeec19c store information about live dependencies in dpb proper, to avoid
synchronization issues with external fs...

the external locks are still used to synchronize with other dpbs running
2013-01-05 20:06:29 +00:00
espie
7184d32670 retrieve the lists of paths locked per-host from the old lock scanner,
to eventually be able to clean them in init.
2013-01-05 19:59:43 +00:00
espie
ba6937b81c and put serialize next to the code it protects
(again, no actual code change)
2013-01-05 18:09:30 +00:00
espie
a18e4e7af2 move install out of the way (no code change) 2013-01-05 18:08:06 +00:00
espie
b159503365 log junk lock handling 2013-01-05 17:29:14 +00:00
espie
82fbc0e7fd avoid opening filehandle again and again.
also, close STDERR after the STDOUT stuff was done, so that we get
error messages otherwise!
2013-01-05 17:22:04 +00:00
espie
ed58c56937 move the "running prepare" message up to handle_output, so that *every*
phase gets tagged.
2013-01-05 16:57:31 +00:00
espie
97912f877b thought I had done this already:
don't blindly append to history, but recreate it from scratch.
We need to remove lines corresponding to stuff that vanished at some
point but came back later!
2013-01-05 16:10:18 +00:00
espie
96eed0203f crank factors up to 150, so that dpb consumes a bit less cpu yet. 2013-01-05 13:39:36 +00:00
espie
df5a288315 make sure we zap any lingering socket before we restart our master. 2013-01-04 21:19:18 +00:00
espie
a9b8de6833 switch to IO::File and explicit flush for the engine log.
should be slightly faster
2013-01-04 19:34:10 +00:00
espie
d606cc382f another half-baked change 2013-01-04 18:49:39 +00:00
espie
7c00e154b6 ... and don't forget to propagate it thru 2013-01-04 12:49:06 +00:00
espie
9612257de1 tweak limiter output some more. offset display is not really needed,
but knowing about force computes might.
2013-01-04 12:45:44 +00:00
espie
66ec642cc7 remove unneeded scaffold 2013-01-04 12:34:29 +00:00
espie
a1d376fbf6 take the rate-limiting concept used for the engine, and make it available
elsewhere. Also use it for the Reporter, as it makes no sense to spend
THAT much time reporting quick changes, which actually slows the build.
($factor to tweak as needed).
2013-01-04 12:06:25 +00:00
espie
3db19612eb forgot that one commit for nojunk. Oops 2013-01-04 12:03:06 +00:00
espie
d17cfb86a3 tweak the Engine some more: don't bother adjusting distfiles in a loop
while dependencies are missing

Following landry's remark, also take packages to build out of the race
if some RUN_DEPENDS are going to be ignored. Only do it right before we
put the package in the queue, so that the test is run exactly once per
package instead of during every scan.

I was also worried about multi-packages, but this only takes one fullpkgpath
out of the loop, and "make package" is going to fail half-way through anyways.
2013-01-03 15:47:45 +00:00
espie
de678efd4a add support for unjunk property, that will prevent junking from actually
happening while some 'unjunk' ports are building.

this is a work-around for a known problem with cmake and qt4 include
dependency handlers...

Also, cache fullpkgpath while building a job, as this contributes for
a large part to the speed (not) of the display when building lots of
ports.
2013-01-03 15:45:18 +00:00
espie
59ee0568e8 stats are comparatively cheap, run them even when the engine doesn't run.
Yields more accurate stat output, where we can see a staircase effect on
the queue when the engine is run not that often, and Built actually go
up until it goes back down again.
2013-01-02 11:13:52 +00:00
espie
f86dbb772a allow overriding the start-up script from the command line. 2012-12-31 09:50:55 +00:00
espie
d985cc05e2 mismatches are more annoying than extra cpu:
if we end up with only mismatches and we didn't run check_buildable last
time around, try a check_buildable first.
2012-12-30 14:33:46 +00:00
espie
45b128b893 fix elusive "jobs don't show up where they should".
duh. put "live" affinity markers while we're building stuff.
We don't re-read the on-disk markers outside of restart (should we ?)
so they HAVE to be in the pkgpath struct proper.
2012-12-30 14:04:17 +00:00
espie
eb77a6071b why not ? track total distsize for each port. might correlate with other
stuff later.
2012-12-30 11:47:24 +00:00
espie
ad8ad1c28f smarter affinity mismatch handling: if we have other cores idle, ask them.
also, officialize the engine-perf log somewhat (and give it a more descriptive
name).
2012-12-29 19:15:37 +00:00
espie
fa169c69ae allow looking for a given core when trying to match affinity 2012-12-29 19:14:51 +00:00
espie
ebb2a2c90a on start-up, log affinity in a more consistent way, with pid of running dpb. 2012-12-29 19:14:28 +00:00
espie
c3a8df2e5f log the markers we retrieve on any restart... I suspect something foul
happening with localhost.
2012-12-28 21:09:45 +00:00
espie
7adbbd9561 remove optimization that no longer makes sense. 2012-12-28 06:40:11 +00:00
espie
d624d61917 since we have a quick path where stuff move from tobuild to install without
adjusting things, stuff that can be built may have BEXTRA which are not
completed...
2012-12-28 06:31:03 +00:00
espie
3ca05b12fe replace next_check = last_check + check_interval
tweak the algorithm slightly (since we forget the old check_interval).
In particular, never keep old times in reserve.

Makes for simpler and clearer reading
2012-12-27 16:23:17 +00:00
espie
e293bae75a remove the need to loop around adjust_tobuild, fudge things a bit:
Most things will move as a result of {install} changes.

Things that move because of EXTRA depends that are satisfied are unlikely to trigger
further changes.

So, stagger changes for "normal" tobuild -> queue first, and extra depends later.
2012-12-27 14:14:19 +00:00
espie
99ac03e94d oops 2012-12-27 12:20:36 +00:00
espie
fc5ee33b62 add timestamp in phases for Job/Port.pm.
may help synchronize events from distinct logs in case this matters.
2012-12-27 12:03:33 +00:00