Commit Graph

231 Commits

Author SHA1 Message Date
espie
67de0deb11 if we have a sizeable queue (>50) and we ran check_buildable less than 30s
ago, don't run it again right now.

This prevents the Engine from busy-checking the same data when it's building
lots of small packages very fast.

(50 and 30 may need some fine-tuning)
2012-12-25 16:07:02 +00:00
espie
11cb77656d make "rebuild" behavior a proper subclass of DPB::PortBuilder, instead
of having tests all over the place.

Take advantage of that to not rechecking signatures if already done.
2012-12-25 10:43:36 +00:00
espie
66b2639394 compute dependencies right away in the job: we don't actually have to
fork depends/prepare/prepare-results for ports which don't have any
depends.

This also prevents some lock contention, obviously.
2012-12-25 10:25:04 +00:00
espie
1af810c8d6 on rebuild, mark all existing files as 'uptodate'. Will prevent extra
rebuilds on @option always-update multi-packages when using -R.
2012-12-25 10:00:08 +00:00
espie
3de193b931 also take things off the later list if they were there. Prevents behavior
like the following:

14426@1356424364: B: x11/kde/graphics3
14426@1356424364: B: x11/kde/graphics3,-kpov
14426@1356424364: B: x11/kde/graphics3,-kamera
14426@1356424364: B: x11/kde/graphics3,-sane
14426@1356424364: B: x11/kde/graphics3,-main
14426@1356424364: V: x11/kde/graphics3,-kpov	<--- note put back on queue
14426@1356424364: V: x11/kde/graphics3
14426@1356424364: V: x11/kde/graphics3,-kamera
14426@1356424364: I: x11/kde/graphics3,-sane
14426@1356424364: I: x11/kde/graphics3
14426@1356424364: I: x11/kde/graphics3,-kamera
14426@1356424364: I: x11/kde/graphics3,-main
14426@1356424366: J: graphics/povray echo.home 0
14426@1356424496: B: devel/goffice
14426@1356424496: I: devel/goffice
14426@1356424496: Q: math/gnumeric
14426@1356424498: B: x11/kde/graphics3		<--- so, duh rebuilt...
14426@1356424498: B: x11/kde/graphics3,-main
14426@1356424498: B: x11/kde/graphics3,-kamera
14426@1356424498: B: x11/kde/graphics3,-sane
14426@1356424498: J: graphics/opencv,-docs echo.home 0
14426@1356424675: B: graphics/povray
14426@1356424675: I: x11/kde/graphics3,-sane
14426@1356424675: I: graphics/povray
14426@1356424675: I: x11/kde/graphics3		<-- and installable, again.
14426@1356424675: I: x11/kde/graphics3,-kpov
14426@1356424675: I: x11/kde/graphics3,-kamera
14426@1356424675: I: x11/kde/graphics3,-main
2012-12-25 09:46:24 +00:00
espie
130df9a3bc will work better with a non-reversed test... 2012-12-25 09:31:54 +00:00
espie
e4c671527c add affinity information to restart jobs on the right host preferentially.
- affinity info is similar to locks, but with a completely different
lifetime.
- streamline the main loop of the engine, so that it can do two passes:
first pass shuns paths with the wrong affinity. If no good path is found,
those are considered during the second pass.
- make the Core factory aware of what hosts might be running, so that
affinity info for machines removed from a config file will be ignored.

thanks to landry@ for a few tests.
2012-12-24 17:24:46 +00:00
espie
0ff5bc966a be slightly more aggressive when a pkgpath is done: for multi-packages,
zap all subpackages from the queue.

need to distinguish the normal case (is_done) with a simpler case
(is_done_quick) that's used when scanning the ports tree. Otherwise, we
would check multi-packages loads of time.
2012-12-24 17:22:15 +00:00
espie
c4f1bfc74e reorg the tests slightly, no functional change 2012-12-24 17:19:01 +00:00
espie
25bc62ba46 quicker restart: when adding a new_path, directly check whether it
was built already... so things will go straight to B/I instead of
getting through T/Q...
less noise in engine.log, and much more accurate information for choosing
queue items.
2012-12-21 12:26:16 +00:00
espie
440719fdce fix clean + check for packages interaction 2012-11-18 01:59:31 +00:00
espie
c2ac425fce multiplication doesn't work on undefined values... noticed by naddy@
so initialize $job->{waiting} if needed.
2012-11-09 17:49:17 +00:00
espie
d73a931704 don't mix up wait_timeout and waiting_timeout, as noticed by naddy@.
re-enable wait_timeout on localhost temporarily (should be done in another
way, most probably by checking whether repo is on nfs, we can steal code
from VStat.pm)

pass umask through ssh. This took us long enough to figure out, and it's
considerably simpler than tweaking every login class once again.
2012-11-06 08:26:29 +00:00
espie
8553ebea36 disable the waiting period if waiting_timeout is 0.
and don't do it on localhost
2012-10-13 09:06:55 +00:00
espie
3453d11c35 zap warnings 2012-10-13 08:47:32 +00:00
espie
302ada6ac0 tweaks
- waiting should know when it gets interrupted! so that it doesn't zombify
- add a time limit.
2012-10-13 08:32:58 +00:00
espie
76fe3e54e3 shrink msg 2012-10-12 22:18:10 +00:00
espie
0f91b47f42 -DALWAYS_CLEAN: clean ports even if they errored out (that's a prop, so
always_clean=1  in hosts file.

refactor task code a bit. And fix a buglet related to the NFS check
(wrong finalize order)
2012-10-12 20:24:56 +00:00
espie
959b30dd92 oops 2012-10-11 17:35:19 +00:00
espie
692f286b94 oh, and explain the why... 2012-10-11 09:03:34 +00:00
espie
1a0fb89393 actually log what's missing that we're waiting upon 2012-10-11 08:38:05 +00:00
espie
8db885a530 reorganize and comment portinfo 2012-10-11 08:05:03 +00:00
espie
9b37bcd801 change things again to have command line options override host file.
and provide the possibility of a DEFAULT entry in the host file.
2012-10-11 07:40:30 +00:00
espie
94c55bc7b4 move to using BUILD_PACKAGES, so we have the actual tidy list.
Stupid NFS work-around:
during/after cleanup, wait until the packages show up.
Each wait period of ten seconds will show up as waiting#n.
2012-10-11 07:38:38 +00:00
espie
7cdbf3d937 zap last bit of state, showing everything as host/4*2 is bad... 2012-10-08 14:16:36 +00:00
espie
cdc017bcb0 small change for people running multiple hosts:
options on the command line now define *defaults* that host files can
override (for instance -j, stuck, -p, -J).

Add -p /n to mean "take number of jobs, if >1, divide by n, round up to 2,
and use that for parallel.

Document -p.

Make junk be 'by host' (and it's a prop, so you can tweak it).

concurrent log that records how many jobs are running each time it changes.
tag parallel builds *n in the time record.
2012-10-08 12:41:03 +00:00
espie
a98e0a8036 our make does ignore further targets with the same name, but it's not
a reason to misuse that feature.

When invoking bsd.port.mk, be more specific, give us a target that's not
defined.
2012-10-06 15:38:14 +00:00
espie
63d13d7f2d oops, unmark core for swallowing before we free it. Otherwise we may
still lose jobs.
2012-09-24 20:41:57 +00:00
espie
f8d9ead7d8 new framework to deal with the new ports cluster, to be documented 2012-09-23 18:13:32 +00:00
espie
0b523b0598 according to Theo, somebody reported this to me (?)
in any case, this should "fix" krw's problem, by allowing PkgPath to
actually display the actual program for further debugging.
2012-08-22 07:49:00 +00:00
espie
64777f79b0 keep state during variable scanning (we see pkgpath in order).
Reset info for a new path systematically, instead of only creating
new infos.

Part of handling erroring paths better: if a pkgpath errors out, when
we remove the lock, the whole port will be rescanned at once, instead of
doing one subpkgpath only.
2012-08-15 09:05:05 +00:00
espie
a673636d3f better messages about caching problems, don't abort if sha info is missing
from distinfo. comment stuff.
(I had this since after the lock, plain forgot to commit it)
2012-08-15 09:02:52 +00:00
espie
5752c45e75 fix caching issues.
- during the scanning stage, we can rely on more than sizes. Specifically,
for files with cached sha values: detect problems early, zap the files, so
the new ones do fetch.

- do not allow "negative" caching: if the cached file doesn't match, just
run the checksum again to make sure (manual download would tamper with that).
This should allow builders to forget about the existence of
/usr/ports/distfiles/distinfo again.

- remove bad files so that fetch has a chance to work (todo: log some more
info, yeah landry...)

- zap code from (checksum) proper  that's no longer in-use.

okay jasper@

(gets in because fixing the mirrors for the release is important, and dpb -F
would not do the right thing without manual intervention).
2012-07-20 11:45:33 +00:00
espie
2da54d1547 delete DIST once we know 2012-07-18 10:31:18 +00:00
espie
f16903c2cc don't delete DIST, prevents checksum from doing its job 2012-07-18 10:30:56 +00:00
espie
f32dd01cdd fix distant listings with SUBDIRLIST 2012-07-12 20:27:19 +00:00
espie
07aff032b9 avoid barfing if run into a directory whose parent contains a Makefile.inc 2012-07-10 22:59:06 +00:00
espie
7591a8ffa0 missed using "clear-to-eol" actually 2012-07-10 14:28:30 +00:00
espie
171b8a9eee fuck me harder. Separate cases where fullpkgname is allowed not to
be defined, so that instead of bad stringizations, we get full time errors.

Maybe then I can debug this.
2012-07-10 09:38:37 +00:00
jasper
d207555654 - fix obivous typo, resulting in always using only one job per host. 2012-07-08 09:59:31 +00:00
espie
40c06ff3d7 prepare for new pkg_add -i/-I: we want to make sure that's NOT interactive! 2012-07-06 12:01:29 +00:00
espie
f7ea023751 remove distinction between local and distant running, always create
a shell object that can chdir, setenv, and exec commands.
(note that this executes stuff after fork, so permanent changes are cheap
and okay)

Also create it from "host" objects, which simplifies parameter passing.
2012-07-04 08:59:10 +00:00
espie
76fbad9f1b those have been moved to base/ 2012-06-19 16:47:06 +00:00
espie
ef74243178 adjust interface slightly 2012-05-16 08:22:22 +00:00
espie
a3e1fdee35 split the PkgPath component into a base class so that sqlports will be
able to reuse it
2012-05-15 10:10:13 +00:00
espie
7fdfa19c5a make sure stuff gets thru ssh correctly 2012-04-21 21:09:07 +00:00
espie
5db27d90a1 fix env name for wrapper program 2012-04-21 14:41:37 +00:00
espie
8dc35a6962 make it possible to use a specialized wrapper to collect rss stats 2012-04-21 11:30:53 +00:00
espie
8cb5363e94 whitespace 2012-04-10 17:06:15 +00:00
espie
0deaaa9fa4 avoid spewed warnings in case we don't care. 2012-04-10 16:50:33 +00:00