Skip to content

Systems and providers

A system is what you are talking to; a provider is how you reach it. Keeping those separate is the point of PosixHost, WindowsHost, and IOSHost: the operating-system semantics stay put when the access mechanism changes.

IOSHost is deliberately session- and command-only: it configures no shell flavour and no path grammar, so commands must be Exec(...) (or raw text the device understands) and path() raises NotImplementedError until an IOS path grammar exists.

System versus transport

A host is not named for its transport: there is no SSH host class or WinRM host class, because a transport does not identify the target. SSH reaches Linux, BSD, network appliances, and application command environments alike, and execution and file access often travel over different channels (SSH plus SFTP, WinRM plus PowerShell file helpers, container exec plus archive).

The system class owns shell flavour, pathname flavour, and host-info normalization. Providers own only the access mechanism:

from hostctl import ExecutorProvider, PathProvider, PosixHost

host = PosixHost(
    executor_providers=(
        ExecutorProvider("ssh", ssh_executor),
        ExecutorProvider("agent", agent_executor),
    ),
    path_providers=(
        PathProvider("sftp", sftp_path),
        PathProvider("rpc", rpc_path),
    ),
)

Provider collections are immutable and ordered: list order is precedence. There is no second priority number to reconcile.

A transport config opens the system host its path flavour selects:

from hostctl import SSHConfig, WinRMConfig

posix = SSHConfig("server.example", username="root").open()
windows = WinRMConfig("server.example", username="admin").open()

Configuration and URIs

Legacy transport URIs are unchanged. System URIs use a logical authority plus ordered, percent-encoded, secret-free provider descriptors in repeated executor= and path= queries:

from hostctl import HostConfig, PathProvider, SSHConfig, register_system_provider

# `sftp` is built in; `rpc` resolves only once something registers it. A
# resolver is handed the host and returns its (executor, path) pair.
def rpc(host):
    return None, PathProvider("rpc", rpc_path)

register_system_provider("rpc", rpc)

config = HostConfig(
    "posix://node?executor=ssh&path=sftp&path=rpc",
    provider_options={"ssh": SSHConfig("node", username="root")},
)

Descriptor order round-trips exactly. Credentials are constructor-only and never appear in connection_uri or a repr(); reconstructing a URI accepts only provider_options= and initializer=.

A name register_system_provider adds is known to every system host. A host class can keep its own instead: provider_resolvers on the class maps names to resolvers, merged up the MRO ahead of the global names, and None hides one. default_providers() names what a host uses when its config names nothing (a default ssh/sftp is dropped when the ssh extra is missing), and an unknown name raises ValueError listing the names that host knows. A worked example is in Extending.

The no-replay safety rule

This is the rule that governs every fallback decision:

After dispatch, failure is terminal

Once a command or path mutation has been dispatched, a failure is never retried through the next provider. A transport that dropped mid-command may have already run it; replaying it could duplicate or corrupt state.

Falling through to the next provider is permitted in exactly two cases:

  1. The provider declines during probe or preflight — it was never dispatched to at all.
  2. The provider raises OperationNotStartedError, which a provider may raise only when it can prove no remote operation began.
from hostctl import OperationNotStartedError

def execute(command, *args, **options):
    if not transport.is_connected():
        # Provably nothing was sent, so fallback is safe.
        raise OperationNotStartedError("transport not connected")
    return transport.run(command, *args, **options)

Any other exception — ConnectionResetError, TimeoutError, OSError, or an unknown error — is treated as uncertain and propagates to the caller. Reads use the same conservative default.

A provider that declines is remembered for the current connection generation, so later operations skip it instead of re-attempting it every time. Reconnecting or replacing a provider starts a new generation and re-probes everything. The one exception is a host with nothing else left: when every other candidate is unavailable, an earlier refusal is tried again rather than failing without dialling, so one transient refusal cannot brick a single-provider host.

Paths and commands connect the same way: a composite path calls the provider's connect() before each dispatch, which is where a refused port or an untrusted host key becomes OperationNotStartedError. A path provider's connect() must therefore be idempotent and cheap once connected.

Selection traces

Every selection records why each candidate was or was not chosen. Traces are redacted: credential-like values are replaced with <redacted> before they reach a log.

path = host.path("/etc/hosts")
path.read_bytes()

for step in path.selection_trace:
    print(step.provider, step.availability, step.chosen, step.reason)

Each entry is a frozen SelectionStep: the provider name, availability, reason, capabilities, whether it was chosen, the connection generation, and the pin flag.

A provider's probe may narrow the capabilities it declared and must not add any: select(capability=...), host.capabilities, host.provider_details and a composed path's routing all use the declared set intersected with what the probe states.

host.provider_details reports availability for every provider without dispatching an operation, so capability queries are always side-effect free:

for detail in host.provider_details:
    print(detail.kind, detail.provider, detail.availability)

Each is a frozen ProviderDetail (kind, provider, availability, reason, capabilities, system_hint).

A provider that declined before dispatch reports unavailable with its refusal reason for the rest of the connection generation, rather than continuing to advertise the availability its cached probe found. Reconnecting starts a new generation and clears the decline.

Debugging

When a remote command fails, two things tell you what happened: the log, and the selection trace.

Turning on the log

Every module logs through logging.getLogger(__name__), so every logger is a child of hostctl (hostctl.provider._contracts, hostctl.system._host, hostctl.transports.ssh._transport, ...). Configure the hostctl logger: the child names follow private module paths and are not API. As a library, hostctl never installs a handler, never sets a level, and never calls basicConfig(); the application owns all of that:

import logging

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("hostctl").setLevel(logging.DEBUG)

At debug you get provider selection outcomes and why each candidate was skipped, every decline with its reason, each dispatched command, and transport connect/close lifecycle:

hostctl.provider._contracts DEBUG selected provider ssh (generation 0, pin False) after 1 candidate(s)
hostctl.provider._contracts DEBUG provider ssh dispatching: /bin/sh -c uptime
hostctl.provider._contracts DEBUG provider ssh declined before dispatch (generation 0): ssh refused: connection lost
hostctl.provider._contracts DEBUG selected provider winrm (generation 0, pin False) after 2 candidate(s)

warning is reserved for events that are recoverable but genuinely surprising, so a warning from hostctl is always worth reading. The one a system host emits is the fallback: run() or a path operation served by a provider ranked below one that is unavailable. It is logged once per host and skipped provider, on host.logger:

hostctl.host._host WARNING [nas1] run() fell back from ssh to webshell: ssh: SSH host key of nas1:22 is not trusted (Host key is not trusted for host nas1); add the key to known_hosts (known_hosts= names the file to read) or pass verify=False to skip the host-key check

The reason after the provider's name is the one it declined with, which host.last_selection also holds as SelectionStep.reason. The SSH, SFTP and WinRM providers name what refused them: an untrusted host key or TLS certificate (with the setting that trusts it), a refused login, a refused connection, a host name that did not resolve, an unreachable host, or a timeout, each with the target and the error's own message, credentials redacted.

Routing an operation to the provider that supports it, and a pinned path, are not fallbacks and log nothing.

Which host a record is about

host.name is a short label for the target (nas1, node:2222), never a credential. host.logger is a LoggerAdapter that prefixes it and sets record.host, so a handler can filter or format on it; assign your own Logger or adapter to send a host's records elsewhere:

host.logger = logging.getLogger("myapp.nas1")

Reading the run-side trace

host.last_selection is the command-side counterpart of a path's selection_trace:

try:
    host.run("systemctl restart nginx")
except Exception:
    for step in host.last_selection:
        print(step.provider, step.availability, step.chosen, step.reason)

The trace accumulates across the failover attempts of a single run(), so the call that actually failed reports every provider tried and every refusal reason — you do not have to make a second call to find out why the first one fell through:

ssh   unavailable  False  ssh refused: connection lost
winrm available    True

last_selection is () before the first run(), and () for a host whose run() does not choose between providers (QEMUHost, SerialHost). Its records are frozen, so reading it cannot disturb selector state.

The redaction guarantee, and its limits

Everything that reaches a log record or a trace entry — command text, provider names, URIs, refusal reasons — passes through ProviderSelector.redact() first.

It recognizes credentials that announce themselves: a password=, passwd=, token=, secret=, api_key=, or key= assignment (with = or :, quoted or bare), and the userinfo half of a scheme://user:secret@host URI. Those become <redacted>.

Best effort, not a guarantee of secrecy

A secret with no marker identifying it cannot be detected by inspection. A positional argument (mysql -p hunter2), a bare token passed as an argv element, or a credential embedded in a script body will appear verbatim. Treat hostctl debug output as sensitive — do not ship it to a log aggregator you would not trust with the credentials themselves.

Redaction is lazy where it can be: every log argument is a wrapper whose __str__ does the work, and logging calls that only while rendering a record it has decided to emit — so with nobody listening at debug the regexes do not run. Trace records are the exception: they are stored and read back through provider_details/selection_trace, so those are redacted as they are built.

Backend pinning

Composite paths select a provider per operation until they are pinned. .via(name) returns a derived path pinned to one backend (host.path(p, backend=name) returns the same pinned path), and .provider reports the current selection:

path = host.path("/var/lib/app/state.db")
pinned = path.via("sftp")
print(pinned.provider.name)  # 'sftp'

A pin survives every derivation — /, joinpath(), parent, parents, with_name(), with_suffix(), and the descendants produced by glob(), rglob(), walk(), and iterdir():

for child in pinned.rglob("*.log"):
    assert child.provider.name == "sftp"

Pinning is not only ergonomic, it is a safety mechanism. Every mutation pins its backend before dispatch, and each open stream owns its provider for the lifetime of the stream, so a multi-step write can never be split across two backends. str(path) is unaffected by pinning — the logical path is stable regardless of which channel served it.

Cross-backend rename() is rejected with NotImplementedError, since no backend can rename into another's namespace. move() acts on that refusal rather than failing: it falls back to a copy followed by removing the source, so moving between two providers transfers the file. copy() between providers is likewise buffered through the client.

Operation-level capabilities

Capabilities are declared per operation, not as one host-level set. A read-only provider advertises only what it can do, and a mutation against it is rejected explicitly rather than silently falling through to another backend:

download = PathProvider(
    "download",
    download_path,
    capabilities=("stat", "read", "open_read", "exists", "is_file", "is_dir"),
)

host.path("payload").via("download").write_bytes(b"...")
# NotImplementedError: provider 'download' does not support write

That explicit refusal is deliberate. Silently rerouting a mutation would hide which channel performed a stateful operation.

Composite paths without a host

A set of path providers is enough to build a composite path: CompositePosixPath.over(providers, *segments, backend=None) (or CompositeWindowsPath.over) is the constructor host.path() itself calls, so the path falls back per operation, pins with .via() or backend=, answers supports() and forwards each call exactly as a host's path does:

from hostctl import CompositePosixPath, SSHConfig, ssh_providers
from hostctl.provider import PathProvider, READ_ONLY_PATH_OPERATIONS

run, sftp = ssh_providers(SSHConfig("nas", username="admin"))
api = PathProvider("api", api_path, capabilities=READ_ONLY_PATH_OPERATIONS)

path = CompositePosixPath.over((sftp, api), "/mnt/tank/report.csv")
data = path.read_bytes()   # SFTP, or the API when SFTP is unreachable
run.close()                # the path owns no provider

providers is an ordered iterable or a ProviderSelector. Paths built over one selector share its declines and probe cache, so a provider refused once is not dialled again by the next path; an iterable gets a selector of its own on each call. Either way, two composites that can route to the same provider object are one filesystem, so copying a file onto itself is refused. The path closes nothing: close the providers when you are done (for ssh_providers(), the executor provider's close() closes the shared transport).

Authoring a provider

An executor provider needs a name, a probe, declared capabilities, and an execute() callable:

from hostctl import ExecutorProvider, ProviderProbe

def probe():
    if not endpoint.reachable():
        return ProviderProbe("unavailable", "endpoint offline")
    return ProviderProbe("available", system_hint="linux")

provider = ExecutorProvider(
    "rpc",
    execute,
    capabilities=("args", "cwd", "env"),
    probe=probe,
)

execute is any callable matching Executor. Rather than write one from scratch, subclass hostctl.executor.BufferedExecutor or StreamingExecutor and implement only the transport's primitive -- see Writing an executor.

A probe returns available, degraded, or unavailable with a reason, the capabilities it can honor, and an optional system_hint. Probes are cached for one connection generation, so they may perform real round trips.

A path provider returns a pathapi.Path; returning anything else is a TypeError:

from hostctl import PathProvider

paths = PathProvider("rpc", lambda *segments: RpcPath(*segments), probe=probe)

Declare only the operations you actually implement. Anything you leave out is refused explicitly instead of being attempted and failing deeper in the stack.

Teardown. host.close() closes each provider's transport attribute when it has one, and the provider itself otherwise (its own close(), if it has one). Providers that share one transport name it as their transport, so it is closed once, not once per provider. connect() must be idempotent: hostctl calls it again after a close and, for a path provider, before every dispatch.

For a worked example composing metadata, download, and SFTP legs with ordered selection and pinning, see examples/application_provider.py.

What is public

hostctl.provider exports the authoring contracts, and only those: ExecutorProvider, PathProvider, ProviderProbe, ProviderSelection, ProviderSelector and SessionInitializer. The exception that permits fallback, OperationNotStartedError, lives in hostctl.exceptions. All of them are also re-exported from top-level hostctl, so the spellings agree.

The adapters the built-in hosts assemble themselves from live in the private modules of their transport package (hostctl.transports.local._providers, and so on), with the capability sets describing what each built-in backend can do. Reading them is the fastest way to see a complete provider, but they are implementation detail: nothing outside hostctl constructs them, and they carry no stability promise. Write your own ExecutorProvider/PathProvider instead of subclassing one. hostctl.provider.READ_ONLY_PATH_OPERATIONS is the capability set of a content-only endpoint -- pass it as PathProvider(name, factory, capabilities=...).

Upgrading a provider to 0.4

hostctl 0.4 builds on pathapi, which replaces pathlib_next. A path provider whose factory still returns a pathlib_next path fails on first use:

TypeError: path provider 'X' returned LocalPath, expected pathapi.Path

To fix it:

  • Import Path from pathapi and SFTPPath from sftppath, and derive your own path classes from pathapi.Path.
  • For a local leg, use hostctl.local_providers(), which returns the (executor_provider, path_provider) pair for this machine, instead of building your own.
  • Call host.replace_providers() instead of assigning the host's private selectors (see below).

The changelog lists the removed names.

Inspecting and replacing providers

A host built from providers lists them in order, and can swap them:

host.executor_providers   # tuple of the command providers, in order
host.path_providers       # tuple of the filesystem providers, in order

host.replace_providers(executor=(new_executor,), path=(new_path,))

replace_providers() takes either keyword or both; None keeps that set. The replaced providers' transports are closed (once each, unless a provider you kept shares the transport), and the host starts a new connection generation: the session initializer runs again before the next operation, on a kept provider too. A set with duplicate names raises before anything is closed or changed. A path you obtained before the call still routes to the old, closed providers, so call host.path(...) again.

Session initialization

A bootstrap hook is modelled separately from both the shell flavour and the console profile, because a raw serial session may deliberately have neither:

from hostctl import SessionInitializer

host = PosixHost(
    executor_providers=(provider,),
    initializer=SessionInitializer(lambda h: h.run("sudo -v"), timeout=10),
)

The initializer runs once per connection generation, after the transport connects and before the host is handed to the caller. It never changes provider identity. If it raises, the connection is torn down and the failure propagates rather than being retried against another provider.