Sync API

pathapi.sync

Synchronizing a tree of paths onto another, across any two Path classes.

__all__ = ['PathAndStat', 'PathSyncer', 'SyncEvent'] module-attribute

PathAndStat

Bases: PathAndStat, FileStat

PathSyncer(checksum=None, /, remove_missing=False, follow_symlinks=True, symlink_mode='preserve', hook=None, ignore_error=False, quick_check=True, preserve_mtime=True, follow_mounts=True)

Bases: object

One-way checksum-driven tree sync: copies/creates in target whatever differs from source (by checksum), optionally removing files in target that are missing from source. Works across any two Path implementations (e.g. MemPath -> LocalPath, or between two remote backends) -- see sync().

The default checksum policy prefers each side's backend-native digest (protocols.checksum.NativeChecksum.checksum(), e.g. an SFTP path's check-file-handle support) over streaming the file through open("rb"), but only when BOTH sides can produce a digest under the same algorithm -- native or streamed. If either side can't (missing the protocol, or it raises NotImplementedError for the requested algorithm), both sides fall back to streaming rather than comparing a native digest to a streamed one. A custom checksum callable disables this native-preferring behavior entirely (it is called exactly as before, once per side, compared with ==).

quick_check=True (the default) adds a cheap metadata-only pre-check -- the classic rsync "quick check" heuristic -- for any pair where at least one side is non-local (is_local(); a side without an is_local() method at all, e.g. plain LocalPath/MemPath, is treated as local): if st_size AND st_mtime already match (from the listing/stat metadata PathAndStat already carries -- no extra round trip), the pair is treated as in sync WITHOUT calling checksum at all, native or streamed. Times match at the resolution of the coarser side: in whole seconds when either has no fractional part (SFTP, FTP and the object stores report seconds), to a microsecond otherwise. A mismatch on either falls through to a real checksum comparison rather than being treated as "changed" -- mtime can be unreliable across backends/clock skew, so a false "needs copy" from a mismatch is merely wasteful, while a false "in sync" would be a correctness regression. An st_mtime of 0 on either side means "unknown" and never matches. Local-to-local pairs always skip this pre-check (local reads are already cheap, and copy(preserve_metadata=True) doesn't guarantee mtime propagation on every path, see docs/divergences.md). Set quick_check=False to disable the pre-check entirely and always checksum.

preserve_mtime=True (the default) gives each file the sync copies the modification time the source reported, where the target's class can set one (LocalPath, MemPath, and FTPPath on a server that lists MFMT; a class of another distribution says so through _utime()), so that the next run's quick check can match. A source that reports no time (0), and a target that cannot set one, leave the time the copy gave the file; neither is an error. Directories and links are not touched. preserve_mtime=False leaves every copy with the time it was written at.

follow_symlinks (default True) controls whether a symlink source is resolved during traversal (content synced as if it weren't a link) or reported as a symlink (is_symlink() true). When it's False and a symlink source is reached, symlink_mode decides what happens: "preserve" (default) creates a matching symlink on target with the same raw, unresolved target string readlink() returned (dangling links and relative targets included -- never resolved/validated); "reject" raises NotImplementedError instead. If target can't create symlinks at all (most backends -- LocalPath and the SFTP backend of another distribution implement symlink_to()), "preserve" mode raises NotImplementedError too, through the same ignore_error/hook() machinery as every other branch -- decided before an existing target entry is touched, and by a dry run as by a real one. Replacing an existing entry creates the new link under a temporary sibling name first, so a runtime refusal also leaves the entry in place. A target link that already has the same raw target string (and, on a Windows target, the same file/directory link kind) is left alone. A directory link is created as one (target_is_directory), which Windows requires. A symlink or a Windows junction found inside the target, below its root, is replaced by a real directory and never listed, written or deleted through. A mount point there (a mounted volume, a bind mount) is part of the tree and is synced into, as rsync does. With follow_mounts=False it is an error for that entry instead: OSError(EXDEV) naming it, offered to ignore_error with event SyncEvent.TypeMismatch, and nothing behind it is listed, written or removed, in a dry run as in a real one. The root target is used as given either way.

remove_missing=False never deletes a non-empty target directory: when the source entry became a file or a symlink, replacing that directory raises IsADirectoryError through ignore_error (event SyncEvent.TypeMismatch) and the directory is left untouched. With remove_missing=True, or for an empty directory, it is replaced.

A changed file is written to a temporary sibling and renamed over the existing target where the target backend implements rename(), so a failed or interrupted transfer keeps the previous version. A backend without rename() is overwritten in place, with the source opened before the target is truncated. A source entry that is not a regular file, directory or symlink (FIFO, socket, device) is skipped with a SyncEvent.Skipped event and its target left untouched.

The temporary sibling is .NAME.<12 hex digits>.pathapi-tmp, NAME cut so that the whole stays within 255 bytes. A backend whose rename() refuses an existing target has the old file removed first; if the second rename then fails, the new version stays under its temporary name and the error names it. A name of exactly that form is the library's own: it is never a sync source and never removed as "missing". A stale one is removed from a target directory when a changed file or link in it is next written (and by remove_missing=True): stale means no sync of this process is writing it and its modification time is known and over a day old. A younger one, or one whose age the store does not report, stays.

Errors: ignore_error is consulted once per error, with the PathAndStat entries that failed and the event that failed (SyncEvent.Compare for a failing quick check or checksum). A name the source directory lists is never "missing": a link whose target cannot be resolved (dangling, a loop, an unmounted volume) is an error for that entry (event SyncEvent.SyncStart), and the same-named target entry is left as it is, whatever remove_missing is. An error it declines propagates without being offered again by enclosing directories. A dry run takes the same decisions as a real run (a directory that would be created is treated as empty) without changing anything. Every error the policy tolerates is logged at WARNING (with the traceback) on the pathapi.sync logger and reported to hook as SyncEvent.Error.

hook receives PathAndStat entries for every event, and the dry_run value of the call for every event, including the structural ones (SyncStart, CheckTargetChild(ren), SyncChild(ren)).

Source code in src/pathapi/sync/_syncer.py

def __init__(
    self,
    checksum: _ty.Callable[[PathAndStat], _ty.Any] | None = None,
    /,
    remove_missing: bool = False,
    follow_symlinks: bool = True,
    symlink_mode: '_ty.Literal["preserve", "reject"]' = "preserve",
    hook: _ty.Callable[[PathAndStat, PathAndStat, SyncEvent, bool], None] = None,
    ignore_error: _OnPathSyncerError | bool = False,
    quick_check: bool = True,
    preserve_mtime: bool = True,
    follow_mounts: bool = True,
) -> None:
    # `None` resolves to `_default_checksum`, which `sync()` recognizes (via `is`) to
    # decide native versus streaming across BOTH sides at once. A caller-supplied
    # callable is used as-is: `checksum(target) == checksum(source)`.
    if checksum is None:
        checksum = _default_checksum
    self.checksum = checksum
    self.remove_missing = remove_missing
    self._hook = hook
    self.follow_symlinks = follow_symlinks
    if symlink_mode not in ("preserve", "reject"):
        raise ValueError(
            f"symlink_mode must be 'preserve' or 'reject', got {symlink_mode!r}"
        )
    self.symlink_mode = symlink_mode
    self.ignore_error = _ty.cast(
        _OnPathSyncerError, _errorpolicy.as_error_handler(ignore_error)
    )
    self.quick_check = quick_check
    self.preserve_mtime = preserve_mtime
    self.follow_mounts = follow_mounts

EVENT_LOG_FORMAT = '[%s] Source:%s Target:%s DryRun:%s' class-attribute instance-attribute

__slots__ = ('checksum', '_hook', 'remove_missing', 'follow_symlinks', 'symlink_mode', 'ignore_error', 'quick_check', 'preserve_mtime', 'follow_mounts') class-attribute instance-attribute

checksum = checksum instance-attribute

follow_mounts = follow_mounts instance-attribute

follow_symlinks = follow_symlinks instance-attribute

ignore_error = _ty.cast(_OnPathSyncerError, _errorpolicy.as_error_handler(ignore_error)) instance-attribute

preserve_mtime = preserve_mtime instance-attribute

quick_check = quick_check instance-attribute

remove_missing = remove_missing instance-attribute

symlink_mode = symlink_mode instance-attribute

hook(source, target, event, dry_run, do=None, ignore_error=None, *, always_run=False)

Source code in src/pathapi/sync/_syncer.py

def hook(
    self,
    source: PathAndStat,
    target: PathAndStat,
    event: SyncEvent,
    dry_run: bool,
    do: _ty.Callable[[], None] = None,
    ignore_error: _OnPathSyncerError = None,
    *,
    always_run: bool = False,
):
    # `ignore_error` passes a per-call policy down; None keeps the instance's.
    # `always_run` runs `do` in a dry run too (the traversal steps, which only read
    # and recurse) while still reporting `dry_run`.
    if ignore_error is None:
        ignore_error = self.ignore_error
    if do and (always_run or not dry_run):
        try:
            do()
        except Exception as e:
            if self._tolerate(ignore_error, e, source, target, event, dry_run):
                return e
            raise
    if self._hook:
        self._hook(source, target, event, dry_run)
    self.log(self.EVENT_LOG_FORMAT, event, source, target, dry_run)

log(msg, *args)

Source code in src/pathapi/sync/_syncer.py

def log(self, msg: str, *args: object):
    # Overridable hook: a subclass routes sync progress elsewhere (the class has
    # `__slots__`, so an instance cannot be assigned one). `*args` go to the logger
    # lazily, so formatting is skipped unless something listens at INFO.
    _logger.info(msg, *args)

sync(source, target, /, dry_run=False, ignore_error=None)

Sync source onto target.

ignore_error overrides the instance-level policy for this call only. It accepts a bool or a callable with the same (error, source, target, event) arity as the constructor's; None (the default) means "use the policy given to __init__".

The root call is checked before anything is touched: a source that does not exist raises FileNotFoundError, and a source and target of the same implementation where one contains the other raise ValueError. Below the root, only a name the source listing no longer holds takes the remove_missing path; a listed link that cannot be resolved is an error for its entry. All go through ignore_error; a tolerated root error ends the call without changes, a tolerated entry error skips that entry. Child names that would not stay a single component inside target (.., a separator, drive, trailing dot or device name on a Windows target) raise ValueError the same way, decided on the name as the source or the target listed it, and symlinks found inside target are replaced, never written, listed or deleted through. So is a Windows junction below the root, which is another directory's second name: rmdir() removes the junction and nothing behind it. A mount point is part of the tree and is synced into. The root target the caller named is used as given.

Source code in src/pathapi/sync/_syncer.py

def sync(
    self,
    source: "Readable | PathAndStat",
    target: Path | PathAndStat,
    /,
    dry_run: bool = False,
    ignore_error: _OnPathSyncerError | bool | None = None,
):
    """Sync `source` onto `target`.

    `ignore_error` overrides the instance-level policy for this call
    only. It accepts a bool or a callable with the same
    `(error, source, target, event)` arity as the constructor's; `None`
    (the default) means "use the policy given to `__init__`".

    The root call is checked before anything is touched: a `source`
    that does not exist raises `FileNotFoundError`, and a `source` and
    `target` of the same implementation where one contains the other
    raise `ValueError`. Below the root, only a name the source listing
    no longer holds takes the `remove_missing` path; a listed link that
    cannot be resolved is an error for its entry. All go through
    `ignore_error`; a tolerated root error ends the call without
    changes, a tolerated entry error skips that entry. Child names that
    would not stay a single component inside `target` (`..`, a
    separator, drive, trailing dot or device name on a Windows target)
    raise `ValueError` the same way, decided on the name as the source
    or the target listed it, and symlinks found inside `target` are
    replaced, never written, listed or deleted through. So is a Windows
    junction below the root, which is another directory's second name:
    `rmdir()` removes the junction and nothing behind it. A mount point
    is part of the tree and is synced into. The root `target` the
    caller named is used as given.
    """
    _ignore_error = (
        self.ignore_error
        if ignore_error is None
        else _errorpolicy.as_error_handler(ignore_error)
    )
    root = target.path if isinstance(target, PathAndStat) else target
    # One commit for the whole tree where the target pays for each write;
    # a dry run writes nothing.
    with _contextlib.nullcontext() if dry_run else _write_batch_of(root):
        return self._sync(source, target, dry_run, _ignore_error, True, set(), {})

SyncEvent

Bases: Enum

Events PathSyncer.hook() fires during a sync, for progress/logging callbacks.

CheckTargetChild = _enum.auto() class-attribute instance-attribute

CheckTargetChildren = _enum.auto() class-attribute instance-attribute

Compare = _enum.auto() class-attribute instance-attribute

Copy = _enum.auto() class-attribute instance-attribute

CreatedDirectory = _enum.auto() class-attribute instance-attribute

Error = _enum.auto() class-attribute instance-attribute

RemovedMissing = _enum.auto() class-attribute instance-attribute

Skipped = _enum.auto() class-attribute instance-attribute

Symlink = _enum.auto() class-attribute instance-attribute

SyncChild = _enum.auto() class-attribute instance-attribute

SyncChildren = _enum.auto() class-attribute instance-attribute

SyncStart = _enum.auto() class-attribute instance-attribute

Synced = _enum.auto() class-attribute instance-attribute

TypeMismatch = _enum.auto() class-attribute instance-attribute