remove a few explicit (and implicit) die from Fetch: missing/out-of-sync
distinfo no longer kill dpb, instead they're properly reported as broken
paths and things still go on (note that even a missing SUPDISTFILE checksum
*will* mark a path as broken, that's totally intentional)
the names of files in there are not totally trivial to figure out from the
normal filename + sha.
-> filename will be stripped off DIST_SUBDIR
-> b64 checksums interfere with filesystem semantics, e.g., /u8//ffg
will become u8/ffg
- keep a stash indexed by checksum, so dpb can identify duplicate files.
- in a full bulk, if the scan has no errors, write to a ${DISTDIR}/history
file the files encountered in ${DISTDIR}/distinfo that seem to no longer
be needed (with full timestamp and checksum info).
Should be enough info to know when to expire old DISTDIR entries.
- keep a cache of checked checksums as ${DISTDIR}/distinfo
- read existing cache on startup, don't need to recheck those files.
- split checksum_task: update the cache after the checksum is done
- create the by_cipher/sha256/... entries
- in fetch_only mode, do a "long" checksum on existing files not in the cache.
If you ever have doubts on the integrity of already checked files, just
rm ${DISTDIR}/distinfo.
building, like bsd.port.mk does.
won't change anything in the build, but makes pkg_add -a less confusing,
since mp ports won't try to add itself while building.0
if no-depends but that's complicated)
also, two distinct critical sections: we don't need to stay locked at
the end of show-prepare-result.
Even though that's not really important, simplify logic a wee little bit
to access more stuff they need.
Remove distantshell->make, ask state instead.
Provide state with the program we run, and the parameters we pass it.
- same make for distant and local hosts
- pass BUILD_ONCE=Yes if -a.
- refactor redirect a bit to store multi-line results to a temporary file
- new show-prepare-results stage using that
- prepare and show-prepare-results shouldn't be NoTime, since they're
predictable.
- lock host thru depends -> junk in the junk case.
acquiring the lock is done thru polling in the child, so that it doesn't block
dpb proper.
- new "junk" stage that gathers dependencies and deletes them.
stages, even if the package is already there.
This could break EXTRA depends where the package would already be there,
and thus its dependency tree not really solved, and thus depends could
easily miss packages...
problem reported by nigel@... and not reported by ajacoutot@ :)
we asked for.
This can be a problem, as rescan relies on it to reenter the path in the
various queues.
So, when we build the equivalence hash, give special treatment to extra
paths that match the actual subdirs list.
Problem reported by naddy@, who sees sporadic errors linked to NFS lag,
which mean that paths sometimes get flagged as errors, then later the error
vanishes automatically when the package appears...
well, this makes darn sure the correct pkgpath does not vanish.
- expose MISSING_FILES
- treat them as a special kind of IGNORE
- handle that in DPB
that way, ports that are IGNORE'd for other reasons (not correct arch)
stay ignored, and relevant ports with missing distfiles get better error
messages
- try resolving RDEPENDS on tobuild, so that known dependencie are zapped.
- use equivalence info to not reprint same thing twice.
- dump tobuild AND built, since both are usually relevant in case lots of
stuff didn't build
- instead of seen/unseen, have an actual constructor. Instead, mark pkgpath
for which we wantinfo/wantbuild.
- only mark EXTRA dependencies as wantinfo. So the devel/haddock,no_deps
temporary error should be gone.
- since we have FLAVOR and SUBPACKAGE available, construct as much info as
we can during vars scanning (see handle_equivalences). This avoids about 150
path rescans during a full bulk. Also, grab the timing and logsizes from
equivalent files, so that most stuff should know show % all the time.
- tweak subdirlist to be a hash, and correctly add pkgpath_and_flavors to it.
That way, we rescan avahi pseudo flavors just once, and not four or five times.
eventually.
- fetch all files
- ignore ignores
- specific builder that doesn't look at existing packages
currently: does not stop when fetch is finished, which is somewhat of the
remaining issue.
Also: change stats to store pid, to make sense of interleaved log files.
there's a safeguard against self-depends anyways.
This prevents ports from getting moved from "built" to "installable"
prematurely.
For instance, devel/libsoup,-gnome doesn't directly depend on
net/glib2-networking, but it depends on devel/libsoup,-main which has
the net/glib2-networking.
Without this patch, dependencies on devel/libsoup,-gnome would often build
prematurely and build net/glib2-networking with them.
- zap heuristic#3, not critical
- don't rebuild the Fetch queue all the time, it's too expensive
- fix bugs in FetchQueue computation, correctly handle 1 distfile case
- create a quick path thru adjust* that handles only new paths.
- use it during LISTING while fetching files, to make more files available
quicker.
- model distfiles as FDEPENDS, DIST, so that they can get dumped.
- add -v option that dumps a bit more stuff in log files.
with these, dpb will fetch all distfiles in a not too surprising order,
and it won't consume 40% cpu while doing so...
it more obvious how much is shared (goal is to unify a large part of these).
In particular, the locking code/stuff already active is glaringly the same...
Polish the distfiles measurements. For now, 3 heuristics that kick in
at various stages of the build (a bit of black magic).
Fix pkgpath->{has} to record both "no dependencies" and "several distfiles
missing" to avoid grabbing small patchfiles which don't lead to valid
builds while heuristic#1 is active.
dpb while checksumming huge tarballs.
Use ftp -C in a systematic way, more complicated logic as to when to
remove temp files, when to keep them: if core exited okay, wrong size
is very bad. If checksum failed on a partial fetch, retry same site with
an empty file...
- clean up clock handling, jobs with timer.
- pass more parameters thru state
- tweak the engine to handle distfiles
- lots of new code in Fetch for new jobs.
Deprecate: -t and -T
use -DCONNECTION_TIMEOUT=... -DDISPLAY_TIMEOUT=... instead
New feature: -DSTUCK_TIMEOUT=
kill tasks when they don't show any progress for that long.
