Commit Graph

119 Commits

Author SHA1 Message Date
espie
1674edcbbd open needed just once. 2011-12-03 11:10:48 +00:00
espie
c1a46f71e8 faster NFS: if the missing file suddenly reappears, we don't need to rescan
the path, just put it back in the queue, and forget it ever happened.
2011-12-03 11:03:07 +00:00
espie
4d52ff4e54 new -J option
- refactor redirect a bit to store multi-line results to a temporary file
- new show-prepare-results stage using that
- prepare and show-prepare-results shouldn't be NoTime, since they're
predictable.
- lock host thru depends -> junk in the junk case.
acquiring the lock is done thru polling in the child, so that it doesn't block
dpb proper.
- new "junk" stage that gathers dependencies and deletes them.
2011-12-02 22:37:36 +00:00
espie
2cfd4c615d new junk option (to remove packages as they're no longer needed) to pass
around thru JobBuilder (along with locks since we need to write stuff into
that file)
2011-12-02 22:33:46 +00:00
espie
b5c447a355 don't buffer printing to locks, to be able to get dependencies early
gather needed=* lines from locks (may want to cache that info later,
but I don't think it has any actual performance impact)
2011-12-02 22:32:07 +00:00
espie
770bcd5b5a methods required to be able to lock per-host 2011-12-02 22:29:28 +00:00
espie
18a4d6f4bf remove dangerous optimization: have paths explicitly pass thru the depends
stages, even if the package is already there.

This could break EXTRA depends where the package would already be there,
and thus its dependency tree not really solved, and thus depends could
easily miss packages...

problem reported by nigel@... and not reported by ajacoutot@ :)
2011-12-02 11:52:06 +00:00
espie
f44c50300c EXTRA should (partly) be handled like DEPENDS/BDEPENDS 2011-12-02 11:40:25 +00:00
espie
60bc66c627 use PREPARE_CHECK_ONLY to weed out bugs in dpb.
also, missed passing dontclean to one instance of add_normal_tasks, oops
2011-11-22 16:48:01 +00:00
espie
4b801b7017 shorter lock line: user doesn't really care that multiple pkgpaths ran into
the same lock, rather that it's a lock, and what the lockname is.
2011-11-22 16:46:44 +00:00
espie
fdc7a9c9fd subtle effect of equivalence: sometimes, we do not get back the subdir
we asked for.

This can be a problem, as rescan relies on it to reenter the path in the
various queues.

So, when we build the equivalence hash, give special treatment to extra
paths that match the actual subdirs list.

Problem reported by naddy@, who sees sporadic errors linked to NFS lag,
which mean that paths sometimes get flagged as errors, then later the error
vanishes automatically when the package appears...

well, this makes darn sure the correct pkgpath does not vanish.
2011-11-22 16:44:53 +00:00
espie
2f6111d127 explain to the engine how to install stuff when it's ready (no error
checks though).

new -I option to do that.
change -C option to be more useful.
2011-11-14 21:57:47 +00:00
espie
5ffb1d6f17 allow check for local cores 2011-11-14 21:56:04 +00:00
espie
c5a81668f4 show parents when we recurse thru subdirs. a bit noisy, but useful when
you want to figure out why some big stuff you wanted to avoid is building
2011-11-14 19:03:41 +00:00
espie
aba2721637 simplify lock code so that lock/unlock actually works, even if I lose
small-grained locks for fullpkgpath
also fix a stupid parenting bug...
2011-11-13 22:18:04 +00:00
espie
2ab4b2c7e7 better FETCH_MANUALLY handling:
- expose MISSING_FILES
- treat them as a special kind of IGNORE
- handle that in DPB
that way, ports that are IGNORE'd for other reasons (not correct arch)
stay ignored, and relevant ports with missing distfiles get better error
messages
2011-11-13 10:34:35 +00:00
espie
e15fa3097d oops, fix FETCH_MANUALLY bug.
Prepare for MISSING_FILES
2011-11-13 10:28:32 +00:00
espie
8f77bf5ec5 make stuff slightly more verbose all the time 2011-11-12 13:19:26 +00:00
espie
9efee931ed smarter dump at end of build:
- try resolving RDEPENDS on tobuild, so that known dependencie are zapped.
- use equivalence info to not reprint same thing twice.
- dump tobuild AND built, since both are usually relevant in case lots of
stuff didn't build
2011-11-09 08:28:55 +00:00
espie
b0e6afe5e4 split DEPENDS into not done/done to help prepare work correctly 2011-11-08 10:26:38 +00:00
espie
cb47853a4e backout shrinkage that breaks prepare 2011-11-08 09:07:24 +00:00
espie
fb0691e921 share MULTI_PACKAGES.
simplify the default case (SUBPACKAGE=-main).
2011-11-07 16:03:19 +00:00
espie
b016e09750 modify pkgpath to store info in a more compact way (in particular,
get rid of sawflavor, and just use the existence of the key to mark the
presence of flavors).

also remove yet more info once we used it.
2011-11-07 13:23:09 +00:00
espie
8bba5b93fa remove a bit more data 2011-11-07 13:21:46 +00:00
espie
3915fb3951 don't auto-vivify info. 2011-11-06 17:56:53 +00:00
espie
7238b18cc3 make sure print_parent gets the correct path 2011-11-06 16:56:23 +00:00
espie
ebe71b8762 slightly better error handling and reporting:
- log to pkgpath whenever we can
- log FETCH_MANUALLY issues as errors, so we can fix them
2011-11-06 12:23:28 +00:00
espie
50ac27a1e8 use print_parent 2011-11-06 12:22:17 +00:00
espie
63910d84bb document why it's broken. Link more parent stuff. 2011-11-06 12:21:47 +00:00
espie
3f5146bbc7 make it simpler to print parent.
zap copy_flavors which I no longer use
2011-11-06 12:21:07 +00:00
espie
0ea853e6e2 convenience function to log to correct path more easily 2011-11-06 12:20:32 +00:00
espie
1ccfe6a255 propagate IGNOREd status to dependent ports.
display directly into engine.log.
replace all info with a stub "ignored already".
This shrinks memory usage some more.
2011-11-05 18:27:13 +00:00
espie
561e2aa812 shrink a bit: the reason for copy_flavors use is long gone
remove the debug scaffolding that duplicates depends list
2011-11-05 18:25:36 +00:00
espie
6af9b14242 repair rescan for distfiles errors 2011-10-15 10:35:41 +00:00
espie
e16ed24351 fix -R and -c. 2011-10-11 13:56:41 +00:00
espie
1be9ae7cd5 duh, for equates to work, I need to store values. I'm an idiot. 2011-10-11 13:43:25 +00:00
espie
234027bfe0 unfuck pkgpaths.
- instead of seen/unseen, have an actual constructor. Instead, mark pkgpath
for which we wantinfo/wantbuild.
- only mark EXTRA dependencies as wantinfo. So the devel/haddock,no_deps
temporary error should be gone.
- since we have FLAVOR and SUBPACKAGE available, construct as much info as
we can during vars scanning (see handle_equivalences). This avoids about 150
path rescans during a full bulk. Also, grab the timing and logsizes from
equivalent files, so that most stuff should know show % all the time.
- tweak subdirlist to be a hash, and correctly add pkgpath_and_flavors to it.
That way, we rescan avahi pseudo flavors just once, and not four or five times.
2011-10-10 18:56:50 +00:00
espie
f29db2a6fb ooh, don't merge depends from IGNOREd stuff for MULTI_PACKAGES.
this allows build to proceed faster on some arches by simply not
building stuff we don't need
2011-10-03 08:56:40 +00:00
espie
2697d6283f FLAVOR is obviously a list 2011-10-03 08:53:49 +00:00
espie
d6d7875186 cosmetic: let IGNORE string behave 2011-09-28 10:13:51 +00:00
espie
2256d26ac6 log ignored ports: with a ! in engine.log, and with full IGNORE message in
new log file ignored.log
2011-09-28 09:49:29 +00:00
espie
e8d7b584fd fix build of "x11/qt3," : for multi-packages, when we find the default
subpackage, copy full options status. So, e.g., look for "x11/qt3,"
find "x11/qt3,,-main" and equate it with "x11/qt3," NOT "x11/qt3"
2011-09-27 17:15:03 +00:00
espie
32d1fbccea deal with suspend: connection may time out, so in case of failure, if
we got suspended, try the same site again.
2011-09-27 17:12:56 +00:00
espie
7fb255ef24 make dpb more useful for people hacking on ports, allow them to not clean
the work directory after building.
2011-09-25 10:41:30 +00:00
espie
1521428320 erroring once is enough... there should be some logfile for this too ;( 2011-09-25 10:40:25 +00:00
espie
ed1516a867 beginning of a fetch-only option that is supposed to replace mirror-maker
eventually.

- fetch all files
- ignore ignores
- specific builder that doesn't look at existing packages

currently: does not stop when  fetch is finished, which is somewhat of the
remaining issue.

Also: change stats to store pid, to make sense of interleaved log files.
2011-09-13 09:46:53 +00:00
espie
9282de6556 fix thinko
thx nigel@
2011-08-30 17:44:28 +00:00
espie
7adaf18e54 move missing distinfo into not so fatal error. 2011-08-29 09:45:29 +00:00
espie
7bcd59566c code reuse 2011-07-14 11:03:49 +00:00
espie
e6a94f281b when checksumming existing files work, remove a possible .part 2011-07-14 11:03:35 +00:00