Commit Graph

204 Commits

Author SHA1 Message Date
espie
63d13d7f2d oops, unmark core for swallowing before we free it. Otherwise we may
still lose jobs.
2012-09-24 20:41:57 +00:00
espie
f8d9ead7d8 new framework to deal with the new ports cluster, to be documented 2012-09-23 18:13:32 +00:00
espie
0b523b0598 according to Theo, somebody reported this to me (?)
in any case, this should "fix" krw's problem, by allowing PkgPath to
actually display the actual program for further debugging.
2012-08-22 07:49:00 +00:00
espie
64777f79b0 keep state during variable scanning (we see pkgpath in order).
Reset info for a new path systematically, instead of only creating
new infos.

Part of handling erroring paths better: if a pkgpath errors out, when
we remove the lock, the whole port will be rescanned at once, instead of
doing one subpkgpath only.
2012-08-15 09:05:05 +00:00
espie
a673636d3f better messages about caching problems, don't abort if sha info is missing
from distinfo. comment stuff.
(I had this since after the lock, plain forgot to commit it)
2012-08-15 09:02:52 +00:00
espie
5752c45e75 fix caching issues.
- during the scanning stage, we can rely on more than sizes. Specifically,
for files with cached sha values: detect problems early, zap the files, so
the new ones do fetch.

- do not allow "negative" caching: if the cached file doesn't match, just
run the checksum again to make sure (manual download would tamper with that).
This should allow builders to forget about the existence of
/usr/ports/distfiles/distinfo again.

- remove bad files so that fetch has a chance to work (todo: log some more
info, yeah landry...)

- zap code from (checksum) proper  that's no longer in-use.

okay jasper@

(gets in because fixing the mirrors for the release is important, and dpb -F
would not do the right thing without manual intervention).
2012-07-20 11:45:33 +00:00
espie
2da54d1547 delete DIST once we know 2012-07-18 10:31:18 +00:00
espie
f16903c2cc don't delete DIST, prevents checksum from doing its job 2012-07-18 10:30:56 +00:00
espie
f32dd01cdd fix distant listings with SUBDIRLIST 2012-07-12 20:27:19 +00:00
espie
07aff032b9 avoid barfing if run into a directory whose parent contains a Makefile.inc 2012-07-10 22:59:06 +00:00
espie
7591a8ffa0 missed using "clear-to-eol" actually 2012-07-10 14:28:30 +00:00
espie
171b8a9eee fuck me harder. Separate cases where fullpkgname is allowed not to
be defined, so that instead of bad stringizations, we get full time errors.

Maybe then I can debug this.
2012-07-10 09:38:37 +00:00
jasper
d207555654 - fix obivous typo, resulting in always using only one job per host. 2012-07-08 09:59:31 +00:00
espie
40c06ff3d7 prepare for new pkg_add -i/-I: we want to make sure that's NOT interactive! 2012-07-06 12:01:29 +00:00
espie
f7ea023751 remove distinction between local and distant running, always create
a shell object that can chdir, setenv, and exec commands.
(note that this executes stuff after fork, so permanent changes are cheap
and okay)

Also create it from "host" objects, which simplifies parameter passing.
2012-07-04 08:59:10 +00:00
espie
76fbad9f1b those have been moved to base/ 2012-06-19 16:47:06 +00:00
espie
ef74243178 adjust interface slightly 2012-05-16 08:22:22 +00:00
espie
a3e1fdee35 split the PkgPath component into a base class so that sqlports will be
able to reuse it
2012-05-15 10:10:13 +00:00
espie
7fdfa19c5a make sure stuff gets thru ssh correctly 2012-04-21 21:09:07 +00:00
espie
5db27d90a1 fix env name for wrapper program 2012-04-21 14:41:37 +00:00
espie
8dc35a6962 make it possible to use a specialized wrapper to collect rss stats 2012-04-21 11:30:53 +00:00
espie
8cb5363e94 whitespace 2012-04-10 17:06:15 +00:00
espie
0deaaa9fa4 avoid spewed warnings in case we don't care. 2012-04-10 16:50:33 +00:00
espie
2ecf172edb elide empty L= / E= lines 2012-04-07 12:16:58 +00:00
espie
a01f052d1c on accelerated boxes, one can keep the cursor visible.
make that the default, keep vis/invis as -DNO_CURSOR.
2012-04-07 12:13:36 +00:00
espie
b2a184c1df fix landry's error.
forget was in the wrong class, oops.
2012-04-02 09:21:18 +00:00
espie
5572ec286f ignore build_stats for logging distinfo (and clean-old-distfiles(1)). 2012-03-22 16:09:37 +00:00
espie
4fe1c88e80 quickpath to fullpkgpath, called a lot during adjust. 2012-03-13 16:48:47 +00:00
espie
38db8f2149 tweak weight handling so that fixing weights and scaling weights works.
add -m option as a reverse to -x, like in pkg_add
2012-03-09 12:51:38 +00:00
espie
d1754a0b78 fix things when we don't have any statistics to start with 2012-03-06 13:41:38 +00:00
espie
5327873dc1 fix -a bug: if you specify -I ..., this shouldn't negate a -a.
"permanent build log" under %f/build-stats/%a
-J250 by default
2012-03-05 19:43:09 +00:00
espie
0a05d056a4 keep track of how long a job is stuck, max. 2012-03-02 19:33:43 +00:00
espie
7222a75d55 fix detection of existing dpbs. 2012-03-02 17:14:41 +00:00
espie
243f41a146 better: if old locks, don't warn.
if no locks to possibly remove, well, don't even run ps !
2012-02-28 14:23:27 +00:00
espie
fb2feec6cf let's do error messages differently: get die to reset_cursor and die
(reverse-override reporter), and get warn to use myprint (and need for
tabs to become spaces in that case...)
2012-02-27 15:37:36 +00:00
espie
56b1b3241a allow -I/-P/-C and command-line arguments to be lists of pkgpaths, or pkgpath
(it's easy to distinguish between a file and a directory under ports).

expand sequences for those files and hosts

when restarting dpb, kill locks that don't correspond to errors, but to a dpb
running on the same host that's no longer there.

do __WARN__ like __DIE__

option -DDONT_BUILD_ONCE
option -DDONT_CLEAN_LOCKS

document some
2012-02-27 14:51:37 +00:00
espie
e7e37de86c hack to avoid stopping midway thru, need to find a better way... ;( 2012-02-23 21:42:23 +00:00
espie
b4dfd43424 also log the fullpkgpath, no need to go look in the lock proper for that. 2012-02-21 10:37:26 +00:00
espie
1a4ddc336c record master dpb pid and hostname
(will be used for automatic clean-up of locks on startup)
2012-02-20 21:18:48 +00:00
espie
81064a2b7c make sure distdir exists before writing information in there 2012-02-20 16:38:23 +00:00
espie
ac82d6c329 cosmetic: fetches come in, so <
explain waiting time better.
2012-02-17 07:36:09 +00:00
espie
9a29fe97e4 allow weights to be scaled
add longer traceback in case we die
2012-02-17 07:35:42 +00:00
espie
c422b5ae52 when you never fetched anything after the cipher change, that directory
doesn't exist... don't error out...
2012-01-31 15:45:19 +00:00
espie
7f2b8805ae finish distfiles history handling: people who don't fetch/build the whole
tree may want to weed distfiles too, so allow for a full scan of the tree
without building/fetching anything, just to update history:

dpb -DHISTORY_ONLY

(just requires making sure the right engines are created, and a very shortened
loop at end waiting for history to be updated).
2012-01-30 15:11:04 +00:00
espie
623eb8b2c5 Better name for the task that, duh, is actually updating distfiles history. 2012-01-30 14:19:30 +00:00
espie
8ddfdef47f more flexible error path while scanning, accumulate reasons why broken.
remove a few explicit (and implicit) die from Fetch: missing/out-of-sync
distinfo no longer kill dpb, instead they're properly reported as broken
paths and things still go on (note that even a missing SUPDISTFILE checksum
*will* mark a path as broken, that's totally intentional)
2012-01-29 12:02:20 +00:00
espie
cd3220153e remove a bit of noise from the log... we checked already, no need to
log it again.
2012-01-23 10:35:38 +00:00
espie
1ec4d51eba repair dpb -f0
noticed by phessler@
2012-01-18 15:26:49 +00:00
espie
f0d702b93c also scan by_cipher for old stuff. This requires a bit more logic, as
the names of files in there are not totally trivial to figure out from the
normal filename + sha.
-> filename will be stripped off DIST_SUBDIR
-> b64 checksums interfere with filesystem semantics, e.g., /u8//ffg
will become u8/ffg
2012-01-16 18:55:35 +00:00
espie
3a6eaf5081 add support to obey PERMIT_DISTFILES* 2012-01-14 12:26:21 +00:00