Commit Graph

79 Commits

Author SHA1 Message Date
espie
2256d26ac6 log ignored ports: with a ! in engine.log, and with full IGNORE message in
new log file ignored.log
2011-09-28 09:49:29 +00:00
espie
e8d7b584fd fix build of "x11/qt3," : for multi-packages, when we find the default
subpackage, copy full options status. So, e.g., look for "x11/qt3,"
find "x11/qt3,,-main" and equate it with "x11/qt3," NOT "x11/qt3"
2011-09-27 17:15:03 +00:00
espie
32d1fbccea deal with suspend: connection may time out, so in case of failure, if
we got suspended, try the same site again.
2011-09-27 17:12:56 +00:00
espie
7fb255ef24 make dpb more useful for people hacking on ports, allow them to not clean
the work directory after building.
2011-09-25 10:41:30 +00:00
espie
1521428320 erroring once is enough... there should be some logfile for this too ;( 2011-09-25 10:40:25 +00:00
espie
ed1516a867 beginning of a fetch-only option that is supposed to replace mirror-maker
eventually.

- fetch all files
- ignore ignores
- specific builder that doesn't look at existing packages

currently: does not stop when  fetch is finished, which is somewhat of the
remaining issue.

Also: change stats to store pid, to make sense of interleaved log files.
2011-09-13 09:46:53 +00:00
espie
9282de6556 fix thinko
thx nigel@
2011-08-30 17:44:28 +00:00
espie
7adaf18e54 move missing distinfo into not so fatal error. 2011-08-29 09:45:29 +00:00
espie
7bcd59566c code reuse 2011-07-14 11:03:49 +00:00
espie
e6a94f281b when checksumming existing files work, remove a possible .part 2011-07-14 11:03:35 +00:00
espie
22f2dd40c8 give a bigger weight to unknown paths 2011-07-14 11:03:13 +00:00
espie
2cf4d84ffd fix a long-standing bug: add LIB_DEPENDS to the RUN_DEPENDS of a port, as
there's a safeguard against self-depends anyways.
This prevents ports from getting moved from "built" to "installable"
prematurely.

For instance, devel/libsoup,-gnome doesn't directly depend on
net/glib2-networking, but it depends on devel/libsoup,-main which has
the net/glib2-networking.

Without this patch, dependencies on devel/libsoup,-gnome would often build
prematurely and build net/glib2-networking with them.
2011-06-15 10:09:31 +00:00
espie
755907c973 parse regression info if present 2011-06-15 10:06:22 +00:00
espie
91419bfb7c spaces 2011-06-04 12:58:24 +00:00
espie
64fc6d3cf2 keep a better list of depends, so that pkg_add does add exactly what it should 2011-06-04 12:56:54 +00:00
espie
8579c8b7c3 do the fetch stuff very early, in new_path. 2011-06-03 13:38:58 +00:00
espie
9ba6fb7c5c - unify code in Fetch/Build subengines
- zap heuristic#3, not critical
- don't rebuild the Fetch queue all the time, it's too expensive
- fix bugs in FetchQueue computation, correctly handle 1 distfile case
- create a quick path thru adjust* that handles only new paths.
- use it during LISTING while fetching files, to make more files available
quicker.
- model distfiles as FDEPENDS, DIST, so that they can get dumped.
- add -v option that dumps a bit more stuff in log files.

with these, dpb will fetch all distfiles in a not too surprising order,
and it won't consume 40% cpu while doing so...
2011-06-02 17:09:25 +00:00
espie
7d68d7e282 add stuff to do quick_dump of distinfo eventually 2011-06-02 11:53:55 +00:00
espie
be6847e3c2 avoid fetching distfiles where we don't know dependencies yet 2011-06-01 15:39:47 +00:00
espie
72c6360806 optimize heuristics for fetchqueue, as they take too much time 2011-06-01 12:34:29 +00:00
espie
647950c27f typo, duplicate work 2011-06-01 12:34:09 +00:00
espie
1e22893e8e if we -f -a, then scan SUPBDISTFILES too, and use the Distfile cache
to write a list of all known distfiles, that can be used to remove extra
stuff from /usr/ports/distfiles
2011-05-29 11:06:23 +00:00
espie
84266ee8ac if we're running -f, do the checksum part internally, since we already
checksummed stuff we fetched (will allow auto-refetch eventually).
2011-05-29 09:30:13 +00:00
espie
aeae163983 oops, the Builder is one indirection down now 2011-05-28 19:32:59 +00:00
espie
3f2603c2f3 fix unlocking in case of errors 2011-05-28 19:24:15 +00:00
espie
9f989ffafe fix 2011-05-28 08:21:39 +00:00
espie
f4d6937978 nicer lockname 2011-05-27 13:13:43 +00:00
espie
2fd23899dd explain the dump_dependencies/find_best optimization for quick-restarting
dpb.
2011-05-27 11:22:13 +00:00
espie
7ef05495f5 start moving the fetch/build code into separate subengine classes, to make
it more obvious how much is shared (goal is to unify a large part of these).

In particular, the locking code/stuff already active is glaringly the same...

Polish the distfiles measurements. For now, 3 heuristics that kick in
at various stages of the build (a bit of black magic).

Fix pkgpath->{has} to record both "no dependencies" and "several distfiles
missing" to avoid grabbing small patchfiles which don't lead to valid
builds while heuristic#1 is active.
2011-05-27 10:27:50 +00:00
espie
a819e7f3e8 improved fetch: do checksum as a separate process, to avoid freezing
dpb while checksumming huge tarballs.
Use ftp -C in a systematic way, more complicated logic as to when to
remove temp files, when to keep them: if core exited okay, wrong size
is very bad. If checksum failed on a partial fetch, retry same site with
an empty file...
2011-05-23 09:44:05 +00:00
espie
0c1194b39b only check checksum if we just fetched it, for now. 2011-05-22 09:19:08 +00:00
espie
d2a3a0fa83 dyslexia 2011-05-22 09:06:49 +00:00
espie
fbfc4fba47 fix checksum logging, avoid checksumming twice 2011-05-22 09:01:27 +00:00
espie
8cef021866 initial support for fetching distfiles, somewhat experimental yet.
- clean up clock handling, jobs with timer.
- pass more parameters thru state
- tweak the engine to handle distfiles
- lots of new code in Fetch for new jobs.
2011-05-22 08:21:38 +00:00
espie
afdb81839a finish refactoring options.
Deprecate: -t and -T
use -DCONNECTION_TIMEOUT=... -DDISPLAY_TIMEOUT=... instead
New feature: -DSTUCK_TIMEOUT=
kill tasks when they don't show any progress for that long.
2011-04-25 11:58:46 +00:00
espie
e7a4ca8c8a ignore whitespace 2011-04-18 11:40:23 +00:00
espie
953361b9c4 in case we log errors, go to the next iteration correctly.
log the fullpkgpath explicitly when we can.
2011-04-05 08:24:34 +00:00
espie
d76bdca371 tweak to allow subclasses of cores that get their own separate lists
(such as local fetcher jobs)
2011-03-22 19:49:56 +00:00
espie
cabe19ad60 tidy, minor 2011-03-22 19:48:53 +00:00
espie
fc9fbe61ee add the glue that parses fetch info and build distinfo for future fetching.
also let Vars report stuff it doesn't know about to the logs file.
2011-03-22 19:48:01 +00:00
espie
c6ed94f022 not sure why it happens, but log nopkgname... 2010-12-12 21:21:03 +00:00
espie
97286a5a18 better info rebuild: make Vars responsible for creating info "just in time"
by keeping local state.
That way, when we rescan, we don't have to remove any info, and all affected
infos are replaced at the same time.
2010-12-07 10:56:26 +00:00
espie
9c270c05b6 fix fix
$build_info //= 1  indeed... ;(
2010-12-06 19:35:06 +00:00
espie
ab01800701 tag dependencies discovered as pure dependencies with their parents, allow
finding out non-sensical stuff in an easier way
2010-12-06 13:20:45 +00:00
espie
d317d925fe minor warning fix 2010-12-06 13:19:44 +00:00
espie
4d8b6b0032 allow 'dpb archivers' without extra artifacts, e.g., recognize and
handle categories without classifying them as pkgpaths with errors.
2010-12-05 10:06:12 +00:00
espie
1c0a277e2c prepare to deal with new depends 2010-11-14 07:44:53 +00:00
espie
82ae79eeca in case the signature doesn't match, clean ALL packages that will be
rebuilt. Otherwise, you may suffer through 3 or 4 rebuilds of qt3...

ouchie.
2010-11-02 20:32:59 +00:00
espie
fb3a3c80f7 much better implementation of -R. In case of rebuilds, check does not "see"
pkgfiles, unless they've been registered.
the engine queues everything to build, but normal build is only an
existence/signature check. The actual build tasks only get queued when
the package does not exist, or if its signature does NOT match the ports
tree.

Should make matthieu@ happy.
2010-11-01 10:55:26 +00:00
espie
8a26786deb new file/option: save dependencies and be able to reuse them.
makes for a "quick listing" of the most important ports first (e.g., groff).
document -R, even though it's not perfect yet.
flesh out all the recent parts and update the BUGS AND LIMITATIONS section.
2010-10-31 11:07:20 +00:00