Commit Graph

54 Commits

Author SHA1 Message Date
espie
9fd4058fbf also tweak the lock names to end in .lock so that they can't get confused
with core dumps ;)
2023-01-30 18:18:41 +00:00
espie
d04a59d00e fix typo 2023-01-25 13:53:48 +00:00
espie
587eb9c193 a bit more love for the control socket
- try to create it as %L/dpb-$$  by default.
If we can't, no biggie. If the user explicitly asked for a control socket,
we DO abort otherwise.
- if CONTROL is explicitly empty, do not load the extra code.
- try to remove the socket at exit

- add a new wipehost command, that will try to kill any job on that host
(if some remain), then zap all locks and affinity info related to that host
"it's dead jim"
2019-07-01 08:59:41 +00:00
espie
b7bbcb0cdf lock parsing errors should not happen, put them in debug.log along with other
generic debug info.
2019-05-12 16:37:52 +00:00
espie
dc7cd9f744 rename "finished" to "errored" so it has clearer semantics
modify the PortBuilder -> Job::Port interface so that differences
are obvious.

tweak wipe to avoid wiping what we're currently actually running,
and also clean the "right" flavor in case we gave a wrong pkgpath name.
2019-05-12 14:09:11 +00:00
espie
1d99433f5a try to avoid thinko, give me a fullpkgpath getter
mark that "cleaned" is (more or less) finished
2019-05-12 13:39:46 +00:00
espie
3c5856cf98 thinko, noticed by naddy@ 2019-05-12 07:50:00 +00:00
espie
8762bc64b8 quite a few lock-related changes, mostly internal
- create a proper writelock object, so that we can pinpoint where we
lock to it
- accordingly, new write_parent interface to deal with writing parent
to a lock
- pass the Logger through to lockinfo, so we log any errors anyhow
- properly deal with stale hostlocks
- categorize lockfile names
- tweak the find_dependencies interface to be less bizarre
- tweak find_tag as well, so that Job/Port can log accurate info for
still_untainted.
- have jobs get an explicit cleanup_after_fork interface, so in addition
to the signal handlers, port jobs also lose their fd on the lock
2019-05-11 15:31:12 +00:00
espie
a20314414c missing field (shouldn't happen too often, as the lockfile is going to vanish) 2019-05-11 11:51:00 +00:00
espie
dcfd3a4080 scan_lockdir can create the lockinfo since it contains its filename. 2019-05-11 10:31:26 +00:00
espie
9d95cfa4d6 add a proper "parser" for lockfiles, and a way to scan
through lockdir.

this simplifies all the code that scans existing locks.
2019-05-11 10:22:49 +00:00
espie
e757f47023 fix a very old bug.
I was confused in my variable names, and thus old locks
where only affecting localhost... ouch
2019-05-10 18:48:41 +00:00
espie
c7099cdbe0 way simpler to zap vim .swp files on the way... 2019-05-09 11:06:01 +00:00
espie
ba234ba60b vim swapfiles are not a problem 2018-08-10 09:50:54 +00:00
espie
719977d8ca just ran into a lock removed while scanning the dir 2018-02-18 16:44:17 +00:00
espie
de7ab17cb0 so when ports get stuck, try to obtain a bit more information as to why
(awaiting lock from the ports tree, and host locks from other dpbs)
2017-12-29 15:49:21 +00:00
afresh1
8e398553b8 Remove fcntl.ph use from DPB::Locks
Replace with a hardcoded constant as Fcntl doesn't provide it.

Fix suggested by espie@ fine with millert@
2016-11-07 21:26:06 +00:00
espie
45ed265b16 nfs shits on its pants again.
not even having readdir and open work synchronously on one single machine,
that shit is so lame.

noticed by aja@, ahaha.
2016-05-14 20:26:42 +00:00
espie
cff48ff52b tweak taintedness:
- fix actual bug in PortInfo
- restrict defined tags to !empty tags (shouldn't happen)
- track the source from locks as well.
2015-06-23 08:51:53 +00:00
espie
b364bd531d tweak find_dependencies to just return a result (deps or path)
reuse in can_be_junked. As exemplified by editors/tiled, a failing
port with nojunk set should also prevent junk tentatives, as these
will fail, but still untaint hosts...

seen by aja@ and naddy@, most probably.
2015-05-24 06:48:51 +00:00
espie
1b06c79c82 make sure we're root when killing stuff. I'm mixing up right and left,
obviously.
2015-05-07 12:30:46 +00:00
espie
8b8d117cf7 do a few more things with the correct user 2015-05-03 10:33:59 +00:00
espie
e2f35e7d02 move the user handling code to a separate file.
remove "run_as" for open, we can do things directly.

More explicit code on open.
2015-05-02 09:44:40 +00:00
espie
d40042f30e lock_user is not needed, defaults to log_user for all practical cases 2015-05-01 19:42:54 +00:00
espie
e94eae598f use UserProxy and make_path method.
Have the default host handle chroot correctly.

(still a bit of fetch to fix)
2015-04-26 18:00:19 +00:00
espie
e06efd44da make most of the creatio files code able to change identity so that logging,
locking   don't happen as root.

Still need to do the fetchfiles/package linking part.
2015-04-25 11:40:58 +00:00
espie
a91c124bcc setting the stage for further changes 2015-04-21 09:53:13 +00:00
espie
b3b46b315d jobs will register the dependencies they would want, preventing junk
from removing them (so that we do less delete/re-add cycles, the more
cores you have the more relevant this becomes).

again, found on Yandex machines, thx to 12 cores or more.
2014-03-09 20:09:53 +00:00
espie
c8f6b0a82f mark most opened files as close-on-exec, since the commands we run don't care
about them. Heck, they shouldn't even be able to access them... Ouchie.
2013-10-25 16:54:27 +00:00
espie
c17299861e on restart, if we find old locks with tags, those hosts start tainted 2013-10-08 17:40:41 +00:00
espie
b46cee4c56 update my copyright to 2013 2013-10-06 13:33:24 +00:00
espie
f2a16fdcba really need to store tags in locks, as dependencies in there will kill
me when restarting dpb...
2013-10-04 20:28:41 +00:00
espie
dd5a4d32bf change the way errors are handled, fixed version:
- pass error condition from Job/Port.pm all the way to the engine
- use that to know whether we fail, instead of the existence of packages
(but still keep track of what we're doing correctly, THAT'S the fix)
- refactor error handling into OO version
- keep track of locks/errors/packages we're waiting for thx to nfs

all of these keep the lock around, and react to the lock being removed.

use case for nfs: if there was a revision bump after dpb scanned the port,
it will never find the package. Removing the lock will allow dpb to rescan
and find the correct packages.

with this, dpb no longer waits after nfs. More importantly, it does not
report nfs hangs as E:, rather as H:... (and it can "wait" for much longer
periods, since it keeps running and only checks on new jobs).
2013-07-18 05:36:54 +00:00
espie
e263c20431 note paths we cleaned in the locks, so that we're not stuck with their
depends or nonjunk properties.
2013-07-12 08:07:19 +00:00
espie
96c787bef7 be a bit more verbose about what's going on for junk
specifically:
- log the paths that may run junk
- explicitly log nojunk behavior
2013-02-03 21:45:52 +00:00
espie
7906ac8797 if I run into lockfiles that don't contain the requisite dpb property,
just hold on until they're there.

Should happen right away, so easy to spot !
2013-01-06 14:38:14 +00:00
espie
8a23d3ca7e ask the ports tree to clean up stale locks, finally... 2013-01-06 11:59:40 +00:00
espie
7184d32670 retrieve the lists of paths locked per-host from the old lock scanner,
to eventually be able to clean them in init.
2013-01-05 19:59:43 +00:00
espie
3db19612eb forgot that one commit for nojunk. Oops 2013-01-04 12:03:06 +00:00
espie
7222a75d55 fix detection of existing dpbs. 2012-03-02 17:14:41 +00:00
espie
243f41a146 better: if old locks, don't warn.
if no locks to possibly remove, well, don't even run ps !
2012-02-28 14:23:27 +00:00
espie
56b1b3241a allow -I/-P/-C and command-line arguments to be lists of pkgpaths, or pkgpath
(it's easy to distinguish between a file and a directory under ports).

expand sequences for those files and hosts

when restarting dpb, kill locks that don't correspond to errors, but to a dpb
running on the same host that's no longer there.

do __WARN__ like __DIE__

option -DDONT_BUILD_ONCE
option -DDONT_CLEAN_LOCKS

document some
2012-02-27 14:51:37 +00:00
espie
1a4ddc336c record master dpb pid and hostname
(will be used for automatic clean-up of locks on startup)
2012-02-20 21:18:48 +00:00
espie
c1a46f71e8 faster NFS: if the missing file suddenly reappears, we don't need to rescan
the path, just put it back in the queue, and forget it ever happened.
2011-12-03 11:03:07 +00:00
espie
b5c447a355 don't buffer printing to locks, to be able to get dependencies early
gather needed=* lines from locks (may want to cache that info later,
but I don't think it has any actual performance impact)
2011-12-02 22:32:07 +00:00
espie
aba2721637 simplify lock code so that lock/unlock actually works, even if I lose
small-grained locks for fullpkgpath
also fix a stupid parenting bug...
2011-11-13 22:18:04 +00:00
espie
50ac27a1e8 use print_parent 2011-11-06 12:22:17 +00:00
espie
91419bfb7c spaces 2011-06-04 12:58:24 +00:00
espie
8cef021866 initial support for fetching distfiles, somewhat experimental yet.
- clean up clock handling, jobs with timer.
- pass more parameters thru state
- tweak the engine to handle distfiles
- lots of new code in Fetch for new jobs.
2011-05-22 08:21:38 +00:00
espie
ab01800701 tag dependencies discovered as pure dependencies with their parents, allow
finding out non-sensical stuff in an easier way
2010-12-06 13:20:45 +00:00