399 Commits

Author SHA1 Message Date
espie
ebc476364a leaner messages: don't show percent on tasks where it doesn't make sense.
replace "unchanged for 28 seconds" by "frozen for 28s"

remove frozen message for "waiting-for-lock" entirely... there's already
a depend/junk task that shows this clearly, and the message tends to be
WAY too long anyways.
2013-09-18 13:26:39 +00:00
espie
51cad93976 chroot=
or
chroot=/
mean no chroot.
2013-09-18 13:07:32 +00:00
espie
8ee7ce33ed fix unlock early condition for multi-package 2013-09-16 21:29:13 +00:00
espie
82235bd10f move the shell running code up to host. Makes no sense to have the same
data structure for each Core, and a lot of this is actually tied to the
host proper.

(note that Core::Distant no longer makes a lot of sense)
2013-09-16 11:23:50 +00:00
espie
f2c8a35b64 tweak time handling a bit, introduce a "real" vs "perceived" time in stats
for people like me who do ^Z a lot.
2013-09-14 14:58:12 +00:00
espie
12be45691d remove remains of nfs wait_timeout. 2013-09-14 09:42:11 +00:00
espie
59801ba696 next step in chrooting, mostly refactor
- provide a chroot_user definition during prop finalize
- let shellclass be depend on the object built
- have all shells store prop and have access to it
- move the chrooted code into default core shell
- tweak running so that quoting/building commands happen everywhere
- change Config to recognize -B as a chroot override.

Tested thru distant chroot, local and distant !chroot
local chroot still not working, probably missing something stupid
2013-09-11 10:44:13 +00:00
espie
66d4d051d1 make it possible to run distant hosts in a chroot 2013-09-09 14:59:53 +00:00
espie
b09a05b63a common code 2013-09-08 19:30:12 +00:00
espie
3ee5353b60 explain about sudo to the Core runner. 2013-09-08 11:10:58 +00:00
espie
e4abcadef0 even with Time::Hires, always log ts in the engine without decimals,
makes for crappy logs otherwise...
2013-09-06 17:11:01 +00:00
espie
df79447c1c quick fix for dpb -x... dirty, refactor needed 2013-09-06 15:58:09 +00:00
espie
63daf24a0e reorganize code, so the periodic sleeper task no longer knows
its sleeping interval, but rather asks it from the reporter.

Give limiter customer access to timing information. Eventually,
the reporter could possibly shrink/enlarge the display timeout
automatically.
2013-09-03 09:44:05 +00:00
espie
fc9e210a0f pretty unlikely, but the Limiter measurements may straddle a ^Z/fg, so
it's better to adjust the timings there as well.
2013-09-03 09:34:23 +00:00
espie
9eae21b57a error out if we can't open log files (got a weird permission error linked
to a half-baked file-system).
2013-09-03 08:53:49 +00:00
espie
1ecc61b2c8 so new_path relies on the pkgname being there, but the engine proper doesn't.
confuses the hell out of some people (hi naddy!) as dpb restarts will notice
built packages regardless of the path.

make things more symetric by doing the check each time we go through tobuild.

I have some misgivings about this, since this is potentially expensive...
2013-08-28 12:00:39 +00:00
espie
b3962116eb some features are not tested often enough.
fix "stop-host".
problem noticed by Florian Obser
2013-08-28 11:55:06 +00:00
espie
ad7e6062f1 make sure the new nfs watch stuff works (unconfuse engine/subengine
relationship) now that landry@ can trigger this reproducibly.
2013-08-07 06:56:42 +00:00
espie
fac62d2d88 the major change from a few days ago completely broke dpb -R (reported
by fgs@, kili@, nigel@)
repair it.
- introduce end_check to cope with package files updated
(not quite satisfied with the source code, but this works, and will do for
5.4).
- adjust report to build.log to conform to the new semantics: namely, if it
didn't fail, then it's okay, don't even look at possibly missing packages
because of nfs (noticed by nigel@).
2013-07-21 16:24:32 +00:00
espie
dd5a4d32bf change the way errors are handled, fixed version:
- pass error condition from Job/Port.pm all the way to the engine
- use that to know whether we fail, instead of the existence of packages
(but still keep track of what we're doing correctly, THAT'S the fix)
- refactor error handling into OO version
- keep track of locks/errors/packages we're waiting for thx to nfs

all of these keep the lock around, and react to the lock being removed.

use case for nfs: if there was a revision bump after dpb scanned the port,
it will never find the package. Removing the lock will allow dpb to rescan
and find the correct packages.

with this, dpb no longer waits after nfs. More importantly, it does not
report nfs hangs as E:, rather as H:... (and it can "wait" for much longer
periods, since it keeps running and only checks on new jobs).
2013-07-18 05:36:54 +00:00
espie
e263c20431 note paths we cleaned in the locks, so that we're not stuck with their
depends or nonjunk properties.
2013-07-12 08:07:19 +00:00
pirofti
bca96cd921 Synchronize the synopsis with the usage.
Comments and okay espie@.
2013-07-05 13:49:50 +00:00
jasper
8e89fb0681 allow dpb to install packages with a non-standard localbase
ok espie@
2013-07-04 17:45:34 +00:00
espie
97120f6aa1 normal ports task can accurately mark that they have failed.
(so that the builder can look at them, and stop trusting the apparition
of pkgfiles to indicate the build succeeded/failed)
2013-07-01 12:35:35 +00:00
espie
d6309756b9 if core is running in "always_clean" mode, always remove the affinity marker
if we find out we got cleansed...
2013-06-30 16:45:16 +00:00
espie
b0f8404077 temporary patch: when always clean, remove the tasks we wanted to run.
(need to address the recording of errors from some tasks in a more proper
way and deal with nfs better)
2013-06-30 16:35:43 +00:00
espie
615c3ee52a belt AND suspenders. In case this triggers because BUILD_PACKAGES !=
MULTI_PACKAGES and IGNORE doesn't get set.
2013-06-25 20:21:52 +00:00
espie
505f43d1d6 sorry sebastia, that's a typo 2013-06-25 19:48:14 +00:00
espie
c6fa1b42ff tricky (problem found by sebastia@ on vax...): BUILD_PACKAGES makes
subpackages vanish too entirely, so later, when other ports want them
as dependencies, we're in trouble!

Instead, record both MULTI and BUILD, and compare them when merging depend.
Don't stub out the new paths directly, but "pre-stub them out" with an
IGNOREd message, so the engine picks them, stubs them out for real, and
logs the reason.
2013-06-25 09:05:19 +00:00
espie
c997331619 should fix some of sebastia's warnings... these are actually old 2013-06-25 07:49:52 +00:00
espie
3fdaec60c5 fix permanent log 2013-06-23 13:19:33 +00:00
espie
18ac15de39 stub_out only removes build_path_list.
equivalent paths will end up with no depends, as they actually share
info, but still on queues.

So do a sanity check when stuff has no depends: if it's stubbed out, just
drop it, as it's been ignored under another name.

Problem noticed and info provided by landry@

(it's all vax's fault anyways, as stub_out is only needed to let dpb fit
within 32M while gobbling thousands of ports info)
2013-06-23 09:04:39 +00:00
espie
b2dece358f typo, broke sthen@'s run 2013-06-22 09:16:36 +00:00
espie
7be6b7b117 reorganize option and config file parsing slightly so that variable
values can be in config files.
command line should still override config files.
to be documented shortly, but stuff like
FETCH_JOBS=n
WANTSIZE=1
MIRROR=0

should now work within a hosts file.
2013-06-21 23:13:37 +00:00
espie
b50533eff0 disentangle the config/init code from the rest.
- move the meat of handle_options from dpb into config->parse_command_line
(this means a backcall to still inherit from OpenBSD::State).
- move parse_config_files from core into config.
- move the prop handling into proper HostProperties (part of config
obviously)
- create a Core::Init file that contains all the former DPB::Host::Factory
and associated jobs.

there's still a wee little bit of cleanup to do, but this should be
easier to maintain, as all option handling is now in one place, and
startup and host confi is now easier to figure out.

- add a -DMIRROR=0/1  setup that controls whether SUPDISTFILES will be
fetched (defaults to 1 for -F and 0 for -f).

- actually allow for several host files to be parsed, as the name implies
2013-06-21 09:05:18 +00:00
espie
686b88c130 no use writing code if you don't use it, right ? 2013-06-17 10:47:44 +00:00
espie
b1e5fe1116 don't recheck the same file again and again. we can cache it properly here 2013-06-16 13:17:59 +00:00
espie
c8f035166f use redirect_fh.
add an extra unconfusing message, to say we renamed the file after it
checksummed okay.
2013-06-16 13:17:35 +00:00
espie
3ae7f059c8 zap complicated bogus logic,
fetch+immediate build should work much better now.
2013-06-15 20:06:50 +00:00
espie
8bc6278431 -DSYSLOG will syslog most activity from jobs starting/ending.
THIS IS A VERY BUSY OPTION. It's intended with a syslog.conf that
will send the log messages elsewhere, to try to pinpoint hang/panic
locations in a given port build, as system crash failures tend to leave
log files in a lagging state.

idea discussed with kurt@
2013-06-04 03:20:36 +00:00
espie
7735fdbb8e scrape redundant test 2013-05-29 09:37:17 +00:00
espie
e71445259f tweak 'dump-vars' run to include TEST values if run with -t.
THIS REQUIRES MOST RECENT bsd.port.mk, OTHERWISE THIS WILL BREAK BADLY.

The values are not used yet, but I fully intend to make it possible to
run non-regression tests in the not too distant future.
2013-05-21 20:44:08 +00:00
espie
17ac01b54d write in the lock file that we're building in memory, more possible affinity
information...
2013-05-18 17:24:46 +00:00
espie
191af91368 revisit old codes, rewrite DISTFILES handling using
.for v1 v2 in list
techniques, which make it clearer.

Add new syntax to distfiles, which is not painful to do now, and support it
for dpb fetch as well: a distfile can now be file{url} to specify a different
result from the url (this allows really weird naming scheme on http sites
to give us consistent files, which has become a nagging problem).

This still doesn't mean you shouldn't encourage upstream to do the right thing
and provide nice archive urls.

extend the fetch process in bsd.port.mk to work more like dpb does, namely
download to intermediate "part" files that don't vanish and use -C if
necessary.

Both those mean that FETCH_CMD has to work like ftp, including the -C and
-o support...

discussed with various people.
2013-05-08 08:38:36 +00:00
espie
ce625eb044 oops, if the error is dire enough, there's no info. 2013-05-01 22:39:24 +00:00
espie
c9efb7f92a add ! to ports that prevent junk from happening, so that you know why it
doesn't junk.

spell out a bit more that logs and locks do NOT need to be shared.
2013-04-30 10:22:36 +00:00
espie
f44e3e9279 forgotten commit, during debug, show more info about what's running on the
core (e.g., job contents)
2013-04-22 20:56:53 +00:00
espie
33fbbc6945 make sure things end up in a sane state. In particular, we may add the
same path several times (which is fine for hashes), but we should deal
with RDEPENDS in a consistent way...

problem "noticed" by landry.
2013-04-22 19:34:10 +00:00
espie
c08aae7a35 only show "real bugs". The ones that come from old paths don't count. 2013-04-22 11:14:17 +00:00
espie
d339778a36 dump all the paths with fishy behavior at end of equiv.
this is a bit too much, but this catches:
SUBDIR=devel/py-gobject3,python3 make dump-vars DPB=Yes
devel/py-gobject3,python.BUILD_DEPENDS=graphics/py3-cairo STEM->=0.10.38:devel/gettext lang/python/3.2 STEM->=0.41.1p0:textproc/intltool devel/gmake archivers/xz
devel/py-gobject3,python.IS_INTERACTIVE=No
devel/py-gobject3,python.SUBPACKAGE=-main
devel/py-gobject3,python.FLAVOR=python3
devel/py-gobject3,python.BUILD_PACKAGES= -main -common
devel/py-gobject3,python.FULLPKGNAME=py3-gobject3-3.8.1p0
devel/py-gobject3,python.RUN_DEPENDS=STEM->=0.10.38:devel/gettext lang/python/3.2 devel/py-gobject3,-common graphics/py3-cairo
devel/py-gobject3,python.LIB_DEPENDS=STEM->=0.10.38:devel/gettext converters/libiconv devel/gobject-introspection
devel/py-gobject3,-common.BUILD_DEPENDS=graphics/py3-cairo STEM->=0.10.38:devel/gettext lang/python/3.2 STEM->=0.41.1p0:textproc/intltool devel/gmake archivers/xz
devel/py-gobject3,-common.IS_INTERACTIVE=No
devel/py-gobject3,-common.SUBPACKAGE=-main
devel/py-gobject3,-common.FLAVOR=python3
devel/py-gobject3,-common.BUILD_PACKAGES= -main -common
devel/py-gobject3,-common.FULLPKGNAME=py-gobject3-common-3.8.1
devel/py-gobject3,-common.RUN_DEPENDS=STEM->=0.10.38:devel/gettext lang/python/3.2

(note the "wrong" flavor, which means that
devel/py-gobject3,python3,-main ends up without an associated fullpkgname)
2013-04-22 10:06:12 +00:00