RunPath
Opt-in ordered step commands from a directory of numbered .py files —
RunPathCmd, register, unregister. import duho.runpath to activate the
provider; core duho never imports this module. See the
RunPath guide for the full walkthrough.
duho.runpath
RunPath: ordered step commands from a directory of numbered .py files.
Opt-in module. Core duho never imports this; you activate it explicitly with
import duho.runpath (which auto-registers a command provider as an import
side-effect) or with an explicit :func:register call. Either way it plugs into
the shipped :func:duho.register_command_provider hook and needs zero
core changes -- it is the first consumer of that seam.
What it adds
A RunPath is a directory whose NN-name.py files are steps run in order:
- each step file
NN-name.pycontributes a step namednamewith numeric ordering keyNN; - a step module may set module-level
PRIORITY: int(overridesNNfor ordering) andREQUIRED: list[str](names of steps that must run first); - a step's runnable body is its module-level entrypoint, resolved with the same
main>run>callprecedence discovery uses (duho.discovery._ENTRYPOINT_NAMES) -- no new convention is invented; - an entrypoint written 1-arg (
def main(cmd), the historical shape) is called unchanged; a 2-arg entrypoint (def main(cmd, ctx)) additionally receives thectxproduced by the directory's optional__main__.py(see below) -- arity-detected, never guessed from a flag, and ONLY when a__main__.pyactually exists (a 2-positional step with no lifecycle file keeps its own default for that parameter -- see "The optional main.py lifecycle" below). - a step may return
None(success) or a non-zeroint(failure, handled exactly like a raised exception -- see "Step return codes" below).
:class:RunPathCmd is a Cmd subclass whose __call__ runs the ordered,
selected steps, logging through self._logger_. Its one CLI field is
--rcopts (-O), a comma-separated list of fnmatch selection patterns.
Inheriting your app's shared root (register(base=...))
An app built with duho.app(root=MyLoggingArgsSubclass, commands=[...])
threads the root's DATA fields onto every subcommand's parsed instance
(argparse's own parents= mechanism), but that is namespace-copying, not
class inheritance -- a METHOD declared on the root (like LoggingArgs's
own _logger_/_set_loglevels_) is NOT callable on a class command's
instance unless the built class itself actually derives from that root.
By default every RunPath command this module builds ALSO inherits
duho.LoggingArgs (alongside RunPathCmd), so -v/_logger_/
_set_loglevels_ work out of the box with zero configuration -- -v
activates DEBUG-level colored stderr logging for a RunPath command exactly
like it does for any other LoggingArgs-based command. Call
register(base=MyAppRoot) once, early (before any RunPath command is
built), if your app's shared root is a custom subclass whose OWN methods you
want every RunPath command to inherit too -- not just its data fields.
A base that is itself an app ROOT (a Cli mixin, or any Cmd with its
own __call__) is handled specially: the built subclass ALWAYS keeps
RunPathCmd.__call__ as its own __call__ (set directly in the class
namespace, so it wins over anything base declares), and the app-root-only
attrs _subcommands_/_version_/_completion_/_config_/
_distribution_ are masked back to their neutral defaults on the built
class. Without this, a RunPath command built over your app's own
class MyApp(LoggingArgs, Cli) root would inherit the root's subcommand
tree (turning myapp rc into "pick a nested subcommand") or the root's own
__call__ (replacing the step runner entirely) -- base is meant to share
a root's methods, never its app-level identity.
The optional __main__.py lifecycle
A RunPath directory may define a __main__.py file -- the same dunder Python
already uses for "this directory's entrypoint" (as in python -m package), so
no new naming convention is invented. It is never treated as a step itself: its
leading _ already excludes it from step discovery (the same guard that skips
any _-prefixed file). It is imported before any step file (so its
module-level setup -- e.g. adding a sibling lib/ to sys.path -- is
already in effect for every step import). It defines up to three module-level
callables, all optional:
init(cmd, logger) -> ctx-- called once before any step runs; its return value is thectxhanded to every 2-arg step entrypoint. Ifinitraises, the whole run is fatal (log then re-raise unconditionally, regardless of--rcopts strict-- every step depends onctx, so there is no meaningful partial/resilient init);success(ctx, cmd, logger)-- called once, INSIDE the step loop'stry(beforefinally_, matchingdiscovery.ModuleCommand's ownmain-then-successorder), and only when no step contributed a failure (no raised exception and no non-zero return, resilient or not -- see "Step return codes");finally_(ctx, cmd, logger)-- called once unconditionally after the step loop, success or failure (a plaintry/finally); a raisingfinally_is logged and swallowed, exactly likediscovery.run_command's own guardedfinally_call, so a teardown error can never mask the real step failure or exit code.
A directory with no __main__.py behaves byte-identical to before this
lifecycle existed: ctx is never produced, and every step keeps being called
with just self regardless of its own declared arity (a step's own second,
possibly-defaulted parameter is simply never populated by RunPath in that
case -- it keeps whatever default the step itself gave it).
Steps and hooks must be synchronous: an async def step or __main__.py
hook has its coroutine result closed immediately and raises TypeError --
this is the ONLY place in duho that awaits anything other than
Cmd.__call__ itself (via duho.args._maybe_await, a separate, documented
seam), so an async step's body silently never running is a loud failure here
instead of a "coroutine was never awaited" warning nobody notices.
Step return codes
A step's return value follows the same convention discovery.ModuleCommand
uses: None means success; a non-zero int means failure. Under strict
(the default) that logs the failure, runs __main__.py's finally_ hook,
and returns the step's own code directly -- no exception, no traceback, unlike
a step that raises, which still propagates. Under resilient it is logged and
the run continues instead. Any other return value is not an exit code and is
ignored. RunPathCmd.__call__ itself returns the WORST of every step's own
code, ranked by magnitude the same way :func:duho.fanout.run_targets
aggregates target codes (a raised, swallowed failure contributes 1, since
it has no numeric code of its own) -- so a resilient run whose steps all
failed non-fatally still reports that failure via its exit code, instead of
always returning 0, and a negative step code (e.g. a step reporting a
signal-like result) is never hidden by an earlier or later 0.
__main__.py's success hook only fires when that aggregate is still
clean (no step failed, strictly or resiliently).
REQUIRED and a failed dependency
REQUIRED is more than ordering: if a step's REQUIRED names a step that
actually ran and failed (raised, or returned non-zero, resilient or not) OR
failed to import, the dependent step is skipped too (never run) rather than
running against a broken prerequisite -- fatality for the SKIPPED dependent
follows its OWN strict setting, same as any other step failure. This does not
apply when a REQUIRED dependency is simply missing or disabled (pure
presence/selection problems, unchanged -- see "Strict vs. resilient").
Filename-encoded per-step options
Before a step file's NN-name prefix is parsed, its stem is checked for a
leading ! and :/;-separated option tokens (both stripped first, so
!02-provision:key.py still yields prefix 02, name provision):
- a leading
!disables the step by default; -
everything after that is a
:- or;-separated list of tokens, eachkey(true),!key(false), orkey=value--:and;are fully interchangeable (a:banda;bparse identically), so a step wanting Windows-authorable filenames can freely use;(:is an invalid Windows filename character;;is valid on both Windows and POSIX). Two tokens are recognized specially: -
strict/!strict-- a step's own default (absent this token) is strict;!strictopts that ONE step OUT of strict (its failure is logged and resilient even when nothing else changes); enable/!enable-- an explicit alternative to the leading!;!step1andstep1:!enabledisable the same step. If BOTH the leading!and an explicit token are somehow present, the TOKEN wins (more specific than the whole-name shorthand).
Any other token is collected but not yet consumed by anything (a forward-compatible extension point).
This is the exact SAME token grammar --rcopts uses per comma-entry (see
below) -- one parser, not two.
Precedence for a step's effective strict setting (each layer overrides the
previous): the step's own filename default (strict=True absent a
!strict token), then a per-pattern --rcopts !strict token matching
that step's name -- ONLY when that pattern entry actually carries an explicit
strict/!strict token, never guessed from the mere presence of some
OTHER token (enable, a key=value extra, or a bare trailing separator
all leave a step's own strict setting untouched) -- then an EXPLICIT bare
--rcopts strict/!strict (no pattern attached -- the RUN-WIDE toggle),
which wins last of all and overrides every step uniformly. This is
symlink-transparent by construction: two symlinks (or plain copies) pointing
at the same physical step file, named differently in two different RunPath
directories, resolve to different effective enabled/strict defaults, because
the parse reads the directory entry's name, never the target file's
content.
BEFORE/AFTER soft ordering
Alongside the existing hard REQUIRED: list[str] (a step and its dependency
must both run; a missing/disabled dep is a warning or, under strict, an error),
a step module may also set:
BEFORE: list[str]-- "I run before X, if X is present and enabled" (named from the declaring step's own side);AFTER: list[str]-- "I run after X, if X runs" (the mirror direction).
Both are pure ordering hints: a BEFORE/AFTER name that is missing, or
present but disabled, is silently a no-op for ordering -- never a warning
(contrast with REQUIRED, whose missing/disabled-dep warning is unchanged).
REQUIRED's hardness stays a fully independent axis from any step's own
filename modifiers and from the run-wide --rcopts strict flag. Only
ENABLED, SELECTED steps ever enter the ordering graph or get imported at all --
a disabled or deselected step's module body never runs, and its
PRIORITY/REQUIRED/BEFORE/AFTER can never transitively reorder
an enabled step (consistent with the "silently dropped" rule above).
--rcopts selection
--rcopts is a comma-separated list of entries, each an fnmatch pattern
matched against step names, optionally followed by :/;-separated
option tokens -- the SAME grammar (and the same strict/enable special
tokens) a step's own filename uses, see above:
- a leading
!disables matching steps (!*disables everything;!*,builddisables all then re-enablesbuild);pattern:!enableis an equivalent, more-explicit spelling of!pattern(wins if both are somehow present on one entry); - a BARE entry that is exactly
strict/!strict(no pattern, no other tokens) toggles the RUN-WIDE strict mode for the run; - an entry WITH a pattern AND a
strict/!stricttoken (e.g.build:!strict) instead scopes that strict override to steps matchingbuildonly, without touching the run-wide flag or any other step.
Later entries win when several match the same step, so !*,build = "disable
all, then enable build".
Strict vs. resilient
Three independent cases, each with its own default:
- Selection/presence problems (an
--rcoptspattern matching no step, aREQUIREDname that is missing or present-but-disabled) are warnings by default; a bare--rcopts strict(no pattern) makes them errors instead. - A step's own failure (a raised exception, a non-zero return, or a
skipped run because its own
REQUIREDdependency failed) is fatal by default -- it aborts the run. A raised exception still propagates as a traceback; a non-zero return instead ends the run cleanly, returning that step's own code (see "Step return codes" above) with no exception raised. Opt ONE step out with its filename's!stricttoken or a matching--rcopts name:!strictentry; a bare--rcopts !strict(no pattern) makes every step resilient instead, and a bare--rcopts strictmakes every step fatal regardless of its own filename token (the run-wide toggle wins last). __main__.py'sinitfailing is always fatal, regardless of any of the above -- every step depends on thectxit produces, so there is no meaningful partial/resilient init.
Declared class-attribute annotations use typing.Union/Optional, not the
PEP-604 | operator, so a RunPathCmd parser builds on Python 3.9.
__all__ = ['RunPathCmd', 'RunPathRegistration', 'register', 'unregister', 'registration', 'preserve_registration', 'step_wants_context', 'is_runpath_dir']
module-attribute
step_wants_context = _step_wants_ctx
module-attribute
RunPathCmd(**kwargs)
Bases: Cmd
Run a directory of numbered NN-name.py steps in order.
A Cmd subclass built by the RunPath provider for a step directory. Its
__call__ loads the optional __main__.py lifecycle FIRST (before any
step import -- see the module docstring), loads the steps (applying
filename-encoded !/:key per-step defaults, importing only enabled/
selected ones), applies --rcopts selection, orders them (Kahn's
algorithm honoring REQUIRED plus the soft BEFORE/AFTER
relations), and runs each one's entrypoint through self._logger_ --
arity-detected so a step written (cmd, ctx) receives the
__main__.py-produced context and a step written (cmd) is
unaffected. Returns the aggregate exit code (max of every step's own,
None/success -> 0, a swallowed failure -> 1).
Not instantiated directly by users; the provider (see :func:register)
subclasses it per directory, binding the directory path and subcommand name.
A direct subclass need only set the class attrs _runpath_dir_ (the step
directory :class:~pathlib.Path) and _parsername_ (the subcommand name).
Source code in src/duho/args/_argsclass.py
298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 | |
rcopts
instance-attribute
Step selection, comma-separated fnmatch patterns; ! disables, strict errors on miss (e.g. !*,build).
__call__()
Source code in src/duho/runpath/_command.py
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 | |
RunPathRegistration
Bases: NamedTuple
What :func:register has set: the base every built command inherits,
the step adapter (None when there is none), and whether the provider
is registered.
base
instance-attribute
registered
instance-attribute
step_adapter
instance-attribute
is_runpath_dir(path)
True if path is a RunPath directory.
A RunPath directory is a directory that (a) contains at least one
NN-name.py step file and (b) is NOT a normal package (no __init__.py)
-- the "shape core doesn't understand" the provider predicate targets. A
directory with an __init__.py is a package and is left to normal import.
Source code in src/duho/runpath/_load.py
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | |
preserve_registration()
Put the registration back as it was on entry, whatever register and
unregister did inside: the base, the step adapter, and whether the
provider is registered.
For a test or a tool that changes the registration for a while::
with preserve_registration():
register(base=MyRoot, step_adapter=adapter)
...
Source code in src/duho/runpath/__init__.py
483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 | |
register(base=None, step_adapter=_KEEP)
Register the RunPath command provider (idempotent).
After this, CmdBuilder resolves a step directory (a dir with
NN-name.py files and no __init__.py) to a :class:RunPathCmd --
this is the only resolution path (discover_commands, CMDS_PATH, and
app(source=...) glob .py files directly and never consult command
providers, so pointed at a step directory they register each step FILE as
its own single-step ModuleCommand instead of an ordered RunPath).
Called automatically (with the default base/step_adapter) as a
side effect of import duho.runpath -- most apps never need to call it
at all. Call it explicitly, after the import, when you want a non-default
base/step_adapter; see :func:unregister to remove the provider
entirely instead.
base (default None -> keeps the CURRENT :data:_BASE, which is
LoggingArgs until changed) sets the class every subsequently-built
RunPathCmd subclass ALSO inherits from, alongside RunPathCmd itself --
call register(base=MyAppRoot) once, early, if your app's shared root
is a custom LoggingArgs subclass (or something else entirely) so its
methods (not just its data fields) are real, inherited members of every
RunPath command your app builds. Re-registering with a different base
on an ALREADY-active provider updates :data:_BASE for commands built
from then on (existing built classes are unaffected -- they were already
constructed). A base that is itself an app root (a Cli mixin or any
Cmd with its own __call__) is safe to pass: the built subclass
always keeps RunPathCmd.__call__ as its own, and the root-only config
attrs are masked back to neutral defaults (see :data:_MASKED_ROOT_ATTRS).
step_adapter (default: keep the current value, initially None)
sets a callable applied to every step's entrypoint just before it runs,
receiving the entrypoint and returning the callable to call in its place.
It exists so an app can accept step signatures of its own rather than
duho's (cmd)/(cmd, ctx) -- an app whose module commands take
run(client, args, logger) can let steps be written that way too,
without every step file importing a decorator::
def adapter(entrypoint):
if not getattr(entrypoint, "_wants_app_shape_", False):
return entrypoint # leave duho-native steps alone
def call(cmd, ctx=None):
return entrypoint(ctx, cmd, cmd._logger_)
return call
register(base=MyAppRoot, step_adapter=adapter)
The adapted callable is what arity detection inspects, so a wrapper is
free to change the signature (including one that uses @functools.wraps
-- see :func:_step_wants_ctx). Pass None to clear it and go back to
calling steps exactly as written; omit the argument entirely to leave it
unchanged. Unlike base, this affects every RunPath command in the
process immediately, including already-built classes -- it is consulted per
step run, not at class-build time.
Source code in src/duho/runpath/__init__.py
384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 | |
registration()
The registration as it is now. Read-only: change it with register.
Source code in src/duho/runpath/__init__.py
478 479 480 | |
unregister()
Remove the RunPath provider this module registered (idempotent).
The counterpart to :func:register -- essential for test isolation so
provider state does not leak between tests. Removes the specific
(predicate, builder) pair registered by this module; a no-op if not
currently registered.
Source code in src/duho/runpath/__init__.py
453 454 455 456 457 458 459 460 461 462 463 464 465 | |