Commit Graph

158 Commits

Author SHA1 Message Date
espie
cd3220153e remove a bit of noise from the log... we checked already, no need to
log it again.
2012-01-23 10:35:38 +00:00
espie
1ec4d51eba repair dpb -f0
noticed by phessler@
2012-01-18 15:26:49 +00:00
espie
f0d702b93c also scan by_cipher for old stuff. This requires a bit more logic, as
the names of files in there are not totally trivial to figure out from the
normal filename + sha.
-> filename will be stripped off DIST_SUBDIR
-> b64 checksums interfere with filesystem semantics, e.g., /u8//ffg
will become u8/ffg
2012-01-16 18:55:35 +00:00
espie
3a6eaf5081 add support to obey PERMIT_DISTFILES* 2012-01-14 12:26:21 +00:00
espie
cb500b33c7 remove files from distinfo when they no longer exist, so when you clean
up files and history, it won't recreate fantom stuff...
2012-01-10 19:35:36 +00:00
espie
f8050aba6f run "expire_old" as a separate job, since it may take some time.
find files under $DISTDIR, anything we don't know about should also
end up in history.
2012-01-09 17:56:28 +00:00
espie
c0aa7effe3 lock may fail 2012-01-09 11:54:30 +00:00
espie
fa990a1dba give dpb more knowledge about distfiles.
- keep a stash indexed by checksum, so dpb can identify duplicate files.
- in a full bulk, if the scan has no errors, write to a ${DISTDIR}/history
file  the files encountered in ${DISTDIR}/distinfo that seem to no longer
be needed (with full timestamp and checksum info).

Should be enough info to know when to expire old DISTDIR entries.
2012-01-08 20:28:37 +00:00
espie
52e58e485e make fetch_manually error directly visible in the lock 2012-01-08 14:40:58 +00:00
espie
3884c2f898 buglet with links and files with a DIST_SUBDIR, as found by naddy@.
people already running dpb -F: just remove /usr/ports/distfiles/distinfo
and rerun dpb.
(or at least all entries with a subdirectory)
2012-01-05 21:49:52 +00:00
espie
d9eef11aed prevent dpb from dying if make_link dies (also does not append to global
distinfo, of course)
2012-01-01 18:34:35 +00:00
espie
9938b28564 allow the fetcher part to work as a "true" mirror maker:
- keep a cache of checked checksums as ${DISTDIR}/distinfo
- read existing cache on startup, don't need to recheck those files.
- split checksum_task: update the cache after the checksum is done
- create the by_cipher/sha256/... entries
- in fetch_only mode, do a "long" checksum on existing files not in the cache.

If you ever have doubts on the integrity of already checked files, just
rm ${DISTDIR}/distinfo.
2011-12-31 11:20:00 +00:00
espie
1f6bab1e34 filter out lib-deps with the same pkgpath_and_flavors as what we're
building, like bsd.port.mk does.
won't change anything in the build, but makes pkg_add -a less confusing,
since mp ports won't try to add itself while building.0
2011-12-11 19:55:37 +00:00
espie
d179751a0f on SIGINFO, dump T/B hashes into info.log (and mark the display for refresh,
as the shell might muck it with its own information)
2011-12-10 14:48:40 +00:00
espie
c71b9ad066 expose interface to "refresh" display after some disturbing operation. 2011-12-10 14:47:36 +00:00
espie
3081c4ac9a remove "needed" debug log now that things work fine. 2011-12-05 21:27:53 +00:00
espie
03c4aa3410 ouch, don't forget to lock at front of prepare (okay, we can avoid it
if no-depends but that's complicated)

also, two distinct critical sections: we don't need to stay locked at
the end of show-prepare-result.

Even though that's not really important, simplify logic a wee little bit
2011-12-05 21:18:55 +00:00
espie
b10178048d move uninstall behavior to uninstall, unbreak !-J
as noticed by fgs@
2011-12-05 18:29:39 +00:00
espie
abb8ddf57d turn -J into an option taking a number.
always run show-prepare-results, and always serialize
depends/prepare/show-prepare-results(/junk) on a single host.
2011-12-05 16:10:00 +00:00
espie
a38035bd67 clean-up job running a bit: add methods to Grabber and PortBuilder objects
to access more stuff they need.

Remove distantshell->make, ask state instead.
Provide state with the program we run, and the parameters we pass it.

- same make for distant and local hosts
- pass BUILD_ONCE=Yes if -a.
2011-12-04 12:05:41 +00:00
espie
3858760dd5 cut up big method into smaller parts 2011-12-04 10:57:46 +00:00
espie
69df96b71c properly forget old info when rescanning... 2011-12-04 10:39:52 +00:00
espie
1674edcbbd open needed just once. 2011-12-03 11:10:48 +00:00
espie
c1a46f71e8 faster NFS: if the missing file suddenly reappears, we don't need to rescan
the path, just put it back in the queue, and forget it ever happened.
2011-12-03 11:03:07 +00:00
espie
0037a6f50e little message tweak (to be changed anyways) 2011-12-03 09:29:41 +00:00
espie
4d52ff4e54 new -J option
- refactor redirect a bit to store multi-line results to a temporary file
- new show-prepare-results stage using that
- prepare and show-prepare-results shouldn't be NoTime, since they're
predictable.
- lock host thru depends -> junk in the junk case.
acquiring the lock is done thru polling in the child, so that it doesn't block
dpb proper.
- new "junk" stage that gathers dependencies and deletes them.
2011-12-02 22:37:36 +00:00
espie
2cfd4c615d new junk option (to remove packages as they're no longer needed) to pass
around thru JobBuilder (along with locks since we need to write stuff into
that file)
2011-12-02 22:33:46 +00:00
espie
b5c447a355 don't buffer printing to locks, to be able to get dependencies early
gather needed=* lines from locks (may want to cache that info later,
but I don't think it has any actual performance impact)
2011-12-02 22:32:07 +00:00
espie
770bcd5b5a methods required to be able to lock per-host 2011-12-02 22:29:28 +00:00
espie
18a4d6f4bf remove dangerous optimization: have paths explicitly pass thru the depends
stages, even if the package is already there.

This could break EXTRA depends where the package would already be there,
and thus its dependency tree not really solved, and thus depends could
easily miss packages...

problem reported by nigel@... and not reported by ajacoutot@ :)
2011-12-02 11:52:06 +00:00
espie
f44c50300c EXTRA should (partly) be handled like DEPENDS/BDEPENDS 2011-12-02 11:40:25 +00:00
espie
ac5bbeeeae rpath without libs, that's a fun one 2011-12-01 11:13:25 +00:00
espie
2a68fab763 thinko 2011-12-01 11:11:23 +00:00
espie
34c5369453 unconfuse naming 2011-11-27 17:31:54 +00:00
espie
e0c5221e97 add a bit of debug code that logs everything that's going on... 2011-11-27 16:15:52 +00:00
espie
bc6cc54b9b move binary scanner into its own module 2011-11-27 14:49:42 +00:00
espie
5b1516ea0b unify ldd/objdump runs somewhat
tweak FileSource to return its directory separately from the filename,
so that running objdump does a chdir
adjust filename to be a relative path.
2011-11-27 11:29:33 +00:00
espie
60bc66c627 use PREPARE_CHECK_ONLY to weed out bugs in dpb.
also, missed passing dontclean to one instance of add_normal_tasks, oops
2011-11-22 16:48:01 +00:00
espie
4b801b7017 shorter lock line: user doesn't really care that multiple pkgpaths ran into
the same lock, rather that it's a lock, and what the lockname is.
2011-11-22 16:46:44 +00:00
espie
fdc7a9c9fd subtle effect of equivalence: sometimes, we do not get back the subdir
we asked for.

This can be a problem, as rescan relies on it to reenter the path in the
various queues.

So, when we build the equivalence hash, give special treatment to extra
paths that match the actual subdirs list.

Problem reported by naddy@, who sees sporadic errors linked to NFS lag,
which mean that paths sometimes get flagged as errors, then later the error
vanishes automatically when the package appears...

well, this makes darn sure the correct pkgpath does not vanish.
2011-11-22 16:44:53 +00:00
jasper
f3ebafe1ff Output an error message if we're supposed to link with a non-existent library,
instead of puking perl errors.

ok espie@
2011-11-14 22:14:38 +00:00
jasper
3dcf344837 Make output slightly more similar like gnu libtool in various
modes. Simplify Exec->command() a bit too while here as suggested by espie@

ok espie@
2011-11-14 22:12:08 +00:00
espie
2f6111d127 explain to the engine how to install stuff when it's ready (no error
checks though).

new -I option to do that.
change -C option to be more useful.
2011-11-14 21:57:47 +00:00
espie
5ffb1d6f17 allow check for local cores 2011-11-14 21:56:04 +00:00
espie
c5a81668f4 show parents when we recurse thru subdirs. a bit noisy, but useful when
you want to figure out why some big stuff you wanted to avoid is building
2011-11-14 19:03:41 +00:00
espie
aba2721637 simplify lock code so that lock/unlock actually works, even if I lose
small-grained locks for fullpkgpath
also fix a stupid parenting bug...
2011-11-13 22:18:04 +00:00
espie
2ab4b2c7e7 better FETCH_MANUALLY handling:
- expose MISSING_FILES
- treat them as a special kind of IGNORE
- handle that in DPB
that way, ports that are IGNORE'd for other reasons (not correct arch)
stay ignored, and relevant ports with missing distfiles get better error
messages
2011-11-13 10:34:35 +00:00
espie
e15fa3097d oops, fix FETCH_MANUALLY bug.
Prepare for MISSING_FILES
2011-11-13 10:28:32 +00:00
espie
8f77bf5ec5 make stuff slightly more verbose all the time 2011-11-12 13:19:26 +00:00
espie
9efee931ed smarter dump at end of build:
- try resolving RDEPENDS on tobuild, so that known dependencie are zapped.
- use equivalence info to not reprint same thing twice.
- dump tobuild AND built, since both are usually relevant in case lots of
stuff didn't build
2011-11-09 08:28:55 +00:00