Paths and backends

The paths need no URI. Each store needs its own extra: s3, gs or az.

objpath

pathlib-style paths for S3, Google Cloud Storage and Azure Blob Storage.

S3Path, GSPath and AZPath need no URI. None of them imports its SDK until a client is built, and none is imported until it is first named (PEP 562), so import objpath needs no extra and never imports uripath. The s3:, gs: and az: schemes of uripath are objpath.uri.

S3Path(*segments, bucket=None, backend=None, **client_kwargs)

Bases: ObjectStorePath

A path in an Amazon S3 bucket (or any S3-compatible store), read and written through boto3. Needs the s3 extra.

S3Path("dir/key", bucket="bkt") is the key dir/key of the bucket bkt. The client options (region_name=, endpoint_url=, aws_access_key_id=, ...) build the path's own S3Backend; backend= shares one client between paths built separately, or plugs in a BaseS3Backend of your own. Giving backend= together with a client option is a TypeError. Without either, paths share one default backend per process. A path joined from another keeps its bucket and its backend, and bucket= moves it to another bucket on the same backend. Nothing connects when a path is built.

S3 has no real directories -- is_dir() is prefix emulation (any object key under "<path>/"), and mkdir() creates a zero-byte "<path>/" marker object (the same convention the AWS console itself uses for an empty "folder"). rename() uses server-side copy_object + delete_object (same bucket only) instead of the generic download+upload+delete move() fallback; a prefix directory is not renamed (NotImplementedError), so move() copies and removes it.

Source code in src/objpath/_s3.py

def __init__(
    self,
    *segments: "str | S3Path",
    bucket: "str | None" = None,
    backend: "BaseS3Backend | None" = None,
    **client_kwargs,
):
    text, donor = self._join(segments)
    place = check_place("bucket", bucket, donor, "S3Path")
    if backend is not None and client_kwargs:
        raise TypeError(
            "pass either backend= or the client options "
            "(region_name=, endpoint_url=, ...), not both"
        )
    if backend is None:
        if client_kwargs:
            backend = S3Backend(**client_kwargs)
        elif donor is not None:
            backend = donor._backend
        else:
            backend = _default_backend()
    self._initialize(text, place, backend)

bucket property

The name of the bucket.

S3Backend(**client_kwargs)

Bases: LazyClientBackend, BaseS3Backend

Lazily creates and caches a boto3 S3 client. A single boto3 client is reused across threads -- it's documented as thread-safe.

client_kwargs go to boto3.client("s3", ...) unchanged (region_name, endpoint_url, aws_access_key_id, config, ...). A context manager: leaving the block closes the client.

Source code in src/objpath/_s3_backend.py

def __init__(self, **client_kwargs):
    super().__init__()
    self.client_kwargs = client_kwargs

client_kwargs = client_kwargs instance-attribute

BaseS3Backend

Bases: ClientBackend

Protocol for obtaining a boto3 S3 client. Subclass this to plug in custom credential/session handling (e.g. tests mock it directly, no real AWS account needed); S3Backend is the real implementation.

client()

The boto3 S3 client every call of a path goes through.

Source code in src/objpath/_s3_backend.py

@_providers.notimplemented
def client(self):
    """The `boto3` S3 client every call of a path goes through."""

GSPath(*segments, bucket=None, backend=None, timeout=UNSET, retry=UNSET, **client_kwargs)

Bases: ObjectStorePath

A path in a Google Cloud Storage bucket, read and written through google.cloud.storage. Needs the gs extra.

GSPath("dir/key", bucket="bkt") is the key dir/key of the bucket bkt. The client options (project=, credentials=, client_options=, ...) and timeout= / retry= build the path's own GSBackend; backend= shares one client between paths built separately, or plugs in a BaseGSBackend of your own. Giving backend= together with a client option is a TypeError. Without either, paths share one default backend per process. A path joined from another keeps its bucket and its backend, and bucket= moves it to another bucket on the same backend. Nothing connects when a path is built.

GCS has no real directories -- is_dir() is prefix emulation (any object key under "<path>/"), and mkdir() creates a zero-byte "<path>/" marker object. rename() uses server-side copy + delete (same bucket only) instead of the generic download+upload+delete move() fallback; a prefix directory is not renamed (NotImplementedError), so move() copies and removes it.

Source code in src/objpath/_gs.py

def __init__(
    self,
    *segments: "str | GSPath",
    bucket: "str | None" = None,
    backend: "BaseGSBackend | None" = None,
    timeout=UNSET,
    retry=UNSET,
    **client_kwargs,
):
    text, donor = self._join(segments)
    place = check_place("bucket", bucket, donor, "GSPath")
    options = {
        name: value
        for name, value in (("timeout", timeout), ("retry", retry))
        if value is not UNSET
    }
    options.update(client_kwargs)
    if backend is not None and options:
        raise TypeError(
            "pass either backend= or the client options "
            "(project=, credentials=, timeout=, ...), not both"
        )
    if backend is None:
        if options:
            backend = GSBackend(**options)
        elif donor is not None:
            backend = donor._backend
        else:
            backend = _default_backend()
    self._initialize(text, place, backend)

bucket property

The name of the bucket.

GSBackend(*, timeout=UNSET, retry=UNSET, **client_kwargs)

Bases: LazyClientBackend, BaseGSBackend

Lazily creates and caches a google.cloud.storage Client. A single client is reused across threads -- it's documented as thread-safe.

client_kwargs go to storage.Client unchanged, client_options (dict or ClientOptions, api_endpoint included) among them. A custom endpoint still authenticates; for an emulator or fake server pass use_auth_w_custom_endpoint=False (anonymous credentials), or set STORAGE_EMULATOR_HOST yourself -- the SDK reads it.

timeout (seconds, or a (connect, read) pair) and retry (a google.api_core.retry.Retry, or None for no retry) are passed to every call a path makes; left out, the SDK's own defaults apply, which keep an unreachable endpoint waiting for about two minutes. A context manager: leaving the block closes the client.

Source code in src/objpath/_gs_backend.py

def __init__(self, *, timeout=UNSET, retry=UNSET, **client_kwargs):
    super().__init__()
    self.client_kwargs = client_kwargs
    self._options = {
        name: value
        for name, value in (("timeout", timeout), ("retry", retry))
        if value is not UNSET
    }

client_kwargs = client_kwargs instance-attribute

call_options()

Source code in src/objpath/_gs_backend.py

def call_options(self) -> dict:
    return dict(self._options)

BaseGSBackend

Bases: ClientBackend

Protocol for obtaining a google.cloud.storage Client. Subclass this to plug in custom credential/session handling (e.g. tests can override to point at a fake server); GSBackend is the real implementation.

call_options()

Keyword arguments (timeout=, retry=) added to every SDK call a path makes; none by default, which leaves the SDK's own.

Source code in src/objpath/_gs_backend.py

def call_options(self) -> dict:
    """Keyword arguments (`timeout=`, `retry=`) added to every SDK call
    a path makes; none by default, which leaves the SDK's own."""
    return {}

client()

The google.cloud.storage client every call of a path goes through.

Source code in src/objpath/_gs_backend.py

@_providers.notimplemented
def client(self):
    """The `google.cloud.storage` client every call of a path goes through."""

AZPath(*segments, container=None, account=None, backend=None, **client_kwargs)

Bases: ObjectStorePath

A path in an Azure Blob Storage container, read and written through azure.storage.blob. Needs the az extra.

AZPath("dir/key", container="cont", account="acct") is the blob dir/key of the container cont of the storage account acct, reached through https://acct.blob.core.windows.net with azure-identity's DefaultAzureCredential (an ImportError naming the extra if azure-identity is missing); paths of one account share one default backend per process. Client options (credential=, account_url=, connection_string=, ...) build the path's own AZBackend, account= among them or not; backend= shares one client between paths built separately, or plugs in a BaseAZBackend of your own. Giving backend= together with account= or a client option is a TypeError, and so is giving none of them (unless a segment is an AZPath, whose backend is kept). A path joined from another keeps its container and its backend, and container= moves it to another container on the same backend. Nothing connects when a path is built.

Azure Blob has no real directories -- is_dir() is prefix emulation (any blob key under "<path>/"), and mkdir() creates a zero-byte "<path>/" marker blob. rename() uses server-side copy + delete (same container only) instead of the generic download+upload+delete move() fallback; a prefix directory is not renamed (NotImplementedError), so move() copies and removes it.

Source code in src/objpath/_az.py

def __init__(
    self,
    *segments: "str | AZPath",
    container: "str | None" = None,
    account: "str | None" = None,
    backend: "BaseAZBackend | None" = None,
    **client_kwargs,
):
    text, donor = self._join(segments)
    place = check_place("container", container, donor, "AZPath")
    if backend is not None and (account is not None or client_kwargs):
        raise TypeError(
            "pass either backend= or account= and the client options "
            "(credential=, account_url=, ...), not both"
        )
    if account is not None:
        check_place("account", account, None, "AZPath")
    if backend is None:
        if client_kwargs:
            backend = AZBackend(account, **client_kwargs)
        elif account is not None:
            backend = _default_backend(account)
        elif donor is not None:
            backend = donor._backend
        else:
            raise TypeError(
                "an AZPath needs account=, backend= or the client options"
            )
    self._initialize(text, place, backend)

account property

The storage account, when the backend names it.

container property

The name of the container.

AZBackend(account=None, **client_kwargs)

Bases: LazyClientBackend, BaseAZBackend

Lazily creates and caches an azure.storage.blob.BlobServiceClient. A single client is reused across threads -- it's documented as thread-safe.

client_kwargs go to BlobServiceClient unchanged; with connection_string, to BlobServiceClient.from_connection_string (credential and the other options included). account, when given and neither account_url nor connection_string is, derives account_url="https://<account>.blob.core.windows.net", authenticated by azure-identity's DefaultAzureCredential unless credential is passed -- the backend an AZPath built with account= uses. Without account, the account is the one connection_string or account_url names, when they name one. A context manager: leaving the block closes the client.

Source code in src/objpath/_az_backend.py

def __init__(self, account: "str | None" = None, **client_kwargs):
    super().__init__()
    self.client_kwargs = client_kwargs
    self.account = (
        account if account is not None else _account_named_by(client_kwargs)
    )

account = account if account is not None else _account_named_by(client_kwargs) instance-attribute

client_kwargs = client_kwargs instance-attribute

BaseAZBackend

Bases: ClientBackend

Protocol for obtaining an azure.storage.blob.BlobServiceClient. Subclass this to plug in custom credential/session handling (e.g. tests can override to point at a fake server); AZBackend is the real implementation. account, when a subclass sets it, is the storage account the client reaches.

account = None class-attribute instance-attribute

copy_timeout = 300.0 class-attribute instance-attribute

read_chunk = 512 * 1024 class-attribute instance-attribute

client()

The BlobServiceClient every call of a path goes through.

Source code in src/objpath/_az_backend.py

@_providers.notimplemented
def client(self):
    """The `BlobServiceClient` every call of a path goes through."""