Commit Graph

177 Commits

Author SHA1 Message Date
espie
4fe1c88e80 quickpath to fullpkgpath, called a lot during adjust. 2012-03-13 16:48:47 +00:00
espie
38db8f2149 tweak weight handling so that fixing weights and scaling weights works.
add -m option as a reverse to -x, like in pkg_add
2012-03-09 12:51:38 +00:00
espie
d1754a0b78 fix things when we don't have any statistics to start with 2012-03-06 13:41:38 +00:00
espie
5327873dc1 fix -a bug: if you specify -I ..., this shouldn't negate a -a.
"permanent build log" under %f/build-stats/%a
-J250 by default
2012-03-05 19:43:09 +00:00
espie
0a05d056a4 keep track of how long a job is stuck, max. 2012-03-02 19:33:43 +00:00
espie
7222a75d55 fix detection of existing dpbs. 2012-03-02 17:14:41 +00:00
espie
243f41a146 better: if old locks, don't warn.
if no locks to possibly remove, well, don't even run ps !
2012-02-28 14:23:27 +00:00
espie
fb2feec6cf let's do error messages differently: get die to reset_cursor and die
(reverse-override reporter), and get warn to use myprint (and need for
tabs to become spaces in that case...)
2012-02-27 15:37:36 +00:00
espie
56b1b3241a allow -I/-P/-C and command-line arguments to be lists of pkgpaths, or pkgpath
(it's easy to distinguish between a file and a directory under ports).

expand sequences for those files and hosts

when restarting dpb, kill locks that don't correspond to errors, but to a dpb
running on the same host that's no longer there.

do __WARN__ like __DIE__

option -DDONT_BUILD_ONCE
option -DDONT_CLEAN_LOCKS

document some
2012-02-27 14:51:37 +00:00
espie
e7e37de86c hack to avoid stopping midway thru, need to find a better way... ;( 2012-02-23 21:42:23 +00:00
espie
b4dfd43424 also log the fullpkgpath, no need to go look in the lock proper for that. 2012-02-21 10:37:26 +00:00
espie
1a4ddc336c record master dpb pid and hostname
(will be used for automatic clean-up of locks on startup)
2012-02-20 21:18:48 +00:00
espie
81064a2b7c make sure distdir exists before writing information in there 2012-02-20 16:38:23 +00:00
espie
ac82d6c329 cosmetic: fetches come in, so <
explain waiting time better.
2012-02-17 07:36:09 +00:00
espie
9a29fe97e4 allow weights to be scaled
add longer traceback in case we die
2012-02-17 07:35:42 +00:00
espie
c422b5ae52 when you never fetched anything after the cipher change, that directory
doesn't exist... don't error out...
2012-01-31 15:45:19 +00:00
espie
7f2b8805ae finish distfiles history handling: people who don't fetch/build the whole
tree may want to weed distfiles too, so allow for a full scan of the tree
without building/fetching anything, just to update history:

dpb -DHISTORY_ONLY

(just requires making sure the right engines are created, and a very shortened
loop at end waiting for history to be updated).
2012-01-30 15:11:04 +00:00
espie
623eb8b2c5 Better name for the task that, duh, is actually updating distfiles history. 2012-01-30 14:19:30 +00:00
espie
8ddfdef47f more flexible error path while scanning, accumulate reasons why broken.
remove a few explicit (and implicit) die from Fetch: missing/out-of-sync
distinfo no longer kill dpb, instead they're properly reported as broken
paths and things still go on (note that even a missing SUPDISTFILE checksum
*will* mark a path as broken, that's totally intentional)
2012-01-29 12:02:20 +00:00
espie
cd3220153e remove a bit of noise from the log... we checked already, no need to
log it again.
2012-01-23 10:35:38 +00:00
espie
1ec4d51eba repair dpb -f0
noticed by phessler@
2012-01-18 15:26:49 +00:00
espie
f0d702b93c also scan by_cipher for old stuff. This requires a bit more logic, as
the names of files in there are not totally trivial to figure out from the
normal filename + sha.
-> filename will be stripped off DIST_SUBDIR
-> b64 checksums interfere with filesystem semantics, e.g., /u8//ffg
will become u8/ffg
2012-01-16 18:55:35 +00:00
espie
3a6eaf5081 add support to obey PERMIT_DISTFILES* 2012-01-14 12:26:21 +00:00
espie
cb500b33c7 remove files from distinfo when they no longer exist, so when you clean
up files and history, it won't recreate fantom stuff...
2012-01-10 19:35:36 +00:00
espie
f8050aba6f run "expire_old" as a separate job, since it may take some time.
find files under $DISTDIR, anything we don't know about should also
end up in history.
2012-01-09 17:56:28 +00:00
espie
c0aa7effe3 lock may fail 2012-01-09 11:54:30 +00:00
espie
fa990a1dba give dpb more knowledge about distfiles.
- keep a stash indexed by checksum, so dpb can identify duplicate files.
- in a full bulk, if the scan has no errors, write to a ${DISTDIR}/history
file  the files encountered in ${DISTDIR}/distinfo that seem to no longer
be needed (with full timestamp and checksum info).

Should be enough info to know when to expire old DISTDIR entries.
2012-01-08 20:28:37 +00:00
espie
52e58e485e make fetch_manually error directly visible in the lock 2012-01-08 14:40:58 +00:00
espie
3884c2f898 buglet with links and files with a DIST_SUBDIR, as found by naddy@.
people already running dpb -F: just remove /usr/ports/distfiles/distinfo
and rerun dpb.
(or at least all entries with a subdirectory)
2012-01-05 21:49:52 +00:00
espie
d9eef11aed prevent dpb from dying if make_link dies (also does not append to global
distinfo, of course)
2012-01-01 18:34:35 +00:00
espie
9938b28564 allow the fetcher part to work as a "true" mirror maker:
- keep a cache of checked checksums as ${DISTDIR}/distinfo
- read existing cache on startup, don't need to recheck those files.
- split checksum_task: update the cache after the checksum is done
- create the by_cipher/sha256/... entries
- in fetch_only mode, do a "long" checksum on existing files not in the cache.

If you ever have doubts on the integrity of already checked files, just
rm ${DISTDIR}/distinfo.
2011-12-31 11:20:00 +00:00
espie
1f6bab1e34 filter out lib-deps with the same pkgpath_and_flavors as what we're
building, like bsd.port.mk does.
won't change anything in the build, but makes pkg_add -a less confusing,
since mp ports won't try to add itself while building.0
2011-12-11 19:55:37 +00:00
espie
d179751a0f on SIGINFO, dump T/B hashes into info.log (and mark the display for refresh,
as the shell might muck it with its own information)
2011-12-10 14:48:40 +00:00
espie
c71b9ad066 expose interface to "refresh" display after some disturbing operation. 2011-12-10 14:47:36 +00:00
espie
3081c4ac9a remove "needed" debug log now that things work fine. 2011-12-05 21:27:53 +00:00
espie
03c4aa3410 ouch, don't forget to lock at front of prepare (okay, we can avoid it
if no-depends but that's complicated)

also, two distinct critical sections: we don't need to stay locked at
the end of show-prepare-result.

Even though that's not really important, simplify logic a wee little bit
2011-12-05 21:18:55 +00:00
espie
b10178048d move uninstall behavior to uninstall, unbreak !-J
as noticed by fgs@
2011-12-05 18:29:39 +00:00
espie
abb8ddf57d turn -J into an option taking a number.
always run show-prepare-results, and always serialize
depends/prepare/show-prepare-results(/junk) on a single host.
2011-12-05 16:10:00 +00:00
espie
a38035bd67 clean-up job running a bit: add methods to Grabber and PortBuilder objects
to access more stuff they need.

Remove distantshell->make, ask state instead.
Provide state with the program we run, and the parameters we pass it.

- same make for distant and local hosts
- pass BUILD_ONCE=Yes if -a.
2011-12-04 12:05:41 +00:00
espie
3858760dd5 cut up big method into smaller parts 2011-12-04 10:57:46 +00:00
espie
69df96b71c properly forget old info when rescanning... 2011-12-04 10:39:52 +00:00
espie
1674edcbbd open needed just once. 2011-12-03 11:10:48 +00:00
espie
c1a46f71e8 faster NFS: if the missing file suddenly reappears, we don't need to rescan
the path, just put it back in the queue, and forget it ever happened.
2011-12-03 11:03:07 +00:00
espie
0037a6f50e little message tweak (to be changed anyways) 2011-12-03 09:29:41 +00:00
espie
4d52ff4e54 new -J option
- refactor redirect a bit to store multi-line results to a temporary file
- new show-prepare-results stage using that
- prepare and show-prepare-results shouldn't be NoTime, since they're
predictable.
- lock host thru depends -> junk in the junk case.
acquiring the lock is done thru polling in the child, so that it doesn't block
dpb proper.
- new "junk" stage that gathers dependencies and deletes them.
2011-12-02 22:37:36 +00:00
espie
2cfd4c615d new junk option (to remove packages as they're no longer needed) to pass
around thru JobBuilder (along with locks since we need to write stuff into
that file)
2011-12-02 22:33:46 +00:00
espie
b5c447a355 don't buffer printing to locks, to be able to get dependencies early
gather needed=* lines from locks (may want to cache that info later,
but I don't think it has any actual performance impact)
2011-12-02 22:32:07 +00:00
espie
770bcd5b5a methods required to be able to lock per-host 2011-12-02 22:29:28 +00:00
espie
18a4d6f4bf remove dangerous optimization: have paths explicitly pass thru the depends
stages, even if the package is already there.

This could break EXTRA depends where the package would already be there,
and thus its dependency tree not really solved, and thus depends could
easily miss packages...

problem reported by nigel@... and not reported by ajacoutot@ :)
2011-12-02 11:52:06 +00:00
espie
f44c50300c EXTRA should (partly) be handled like DEPENDS/BDEPENDS 2011-12-02 11:40:25 +00:00