`syshttp.client`

A synchronous HTTP client on the standard library. syshttp.fetch is syshttp.client.fetch.

HTTPClient(transport=...) sends through any object with send() and close() (the Transport protocol), such as a transport built on requests or httpx, or a fake in a test. The client owns redirects, credentials, cookies, tokens, hooks, retries and the deadline; the transport owns routing, TLS verification, connection reuse and content decoding. A transport written elsewhere therefore handles proxies and TLS itself, as its library does; prepare_request() gives it the checks and body framing of syshttp's own, and SYSHTTP_DEFAULT_TRANSPORT makes it the default (see Settings).

`syshttp.client`

An HTTP client on the standard library, with a replaceable transport.

HTTPClient sends a request through a transport (an object with send() and close(): the default HTTPTransport on http.client, or one written against the Transport protocol) and follows redirects under one rule for credentials: Authorization, Cookie and Proxy-Authorization go to the origin they were given for and to no other scheme, host or port. A reply comes back as an HTTPResponse.

Importing this package loads http.client and ssl. Nothing here needs a third-party library; requests in particular is not used.

`CA_BUNDLE = 'CA_BUNDLE'` `module-attribute`

The setting naming a CA bundle (a PEM file) that replaces the system's trust store for origins when verification is on and the caller gave no context.

`COOKIE_HOSTS = 'SYSHTTP_COOKIE_HOSTS'` `module-attribute`

The setting declaring hosts that share cookies: hosts split by ,, groups by ;.

`DEFAULT_TIMEOUT = (10, 60)` `module-attribute`

(connect, read) timeout, in seconds, a client sends with every request unless it is given timeout (HTTPClient(timeout=...) or per request); timeout=None waits without a limit.

`DEFAULT_TRANSPORT = 'SYSHTTP_DEFAULT_TRANSPORT'` `module-attribute`

The setting naming the transport of a client given none: an object reference module:attr or <file>.py:attr (a dotted attr is allowed). The object is the transport when it has send(); otherwise it is called with no arguments to build one (a class, a factory). It names code to import: never let an untrusted party set it.

`HTTPClient(transport=None, *, headers=None, timeout=DEFAULT_TIMEOUT, deadline=None, user_agent=None, env=None, proxymap=None, ssl_context=None, proxy_ssl_context=None, verify=True, cookies=False, tokens=False, auth_provider=None, retries=3, backoff=1, status_forcelist=TRANSIENT_STATUSES, cookie_hosts=None)`

A transport with the settings every request is sent with, the rules for what happens after a redirect, and the state that belongs to a client.

transport is any object with a send() method as HTTPTransport.send(); one it is given stays the caller's. Without one the client builds the transport the SYSHTTP_DEFAULT_TRANSPORT setting names, else an HTTPTransport, when it is made, and closes it with close() or at the end of a with block. proxymap, ssl_context, proxy_ssl_context and verify configure that HTTPTransport and are ValueError beside a transport of your own; env (default HTTPEnv()) is where the client and its transport read their settings. headers are sent with every request, request-specific headers merged over them by name, over a User-Agent of user_agent (syshttp/<version> by default) and Accept: */*. Every request gets timeout=DEFAULT_TIMEOUT ((10, 60) seconds) unless timeout here or in the call says otherwise (None waits without limit). timeout bounds each wait; deadline (seconds, None for no limit, the default) bounds the whole request: redirects, retries and the reading of the body included, each wait's timeout is held to the time that remains, and TimeoutError is raised once it has passed. A read already blocked can overrun it by at most the timeout of that wait.

The client, not the transport, follows redirects. GET, HEAD and OPTIONS follow up to 30; a redirect that changes the scheme, host or port drops Authorization, Cookie and Proxy-Authorization (and nothing else; the jar's cookies are chosen again for each hop). Any other method is sent without following: a 307 or 308 (for PROPFIND also a 301, 302 or 303) to the same scheme, host and port is sent once more with the same method and body, and any other 3xx, a redirect to another origin or a second redirect raises OSError(EIO).

State, chosen by the host of the URL asked for (the origin), never by a proxy's or a gateway's. A client keeps the cookies its responses set, in memory and, with cookies, in a jar: True is a file jar under jar.store_root() by host, a string one key for every host, a jar object is used as given. cookie_hosts (default SYSHTTP_COOKIE_HOSTS) declares groups of hosts that share cookies. tokens is read the same way and becomes a per-request Authorization (Bearer <token>, or verbatim when it names a scheme) that follows no redirect to another origin. In request() (not in redirect()): auth_provider.ensure(client, url) runs first, its errors swallowed; the SYSHTTP_HOOKS_<KEY>_PY hooks run on the caller's URL, and a request a hook makes through the client runs none of them again (the guard is per thread); a status in status_forcelist, or a failed connection (ConnectionError), is retried up to retries times for a method that is safe to send twice, after backoff seconds doubled each time and never less than the Retry-After of the reply, and a status that outlasts the retries is returned, not raised.

Source code in src/syshttp/client/_client.py

def __init__(
    self,
    transport: "_ty.Any" = None,
    *,
    headers: "_ty.Mapping[str, str] | None" = None,
    timeout: "float | tuple | None" = DEFAULT_TIMEOUT,
    deadline: "float | None" = None,
    user_agent: "str | None" = None,
    env: "HTTPEnv | None" = None,
    proxymap: "_ty.Any" = None,
    ssl_context: "_ssl.SSLContext | None" = None,
    proxy_ssl_context: "_ssl.SSLContext | None" = None,
    verify: "bool | str | _os.PathLike[str]" = True,
    cookies: "bool | str | CookieJar | None" = False,
    tokens: "bool | str | TokenJar | None" = False,
    auth_provider: "AuthProvider | None" = None,
    retries: int = 3,
    backoff: float = 1,
    status_forcelist: "_ty.Iterable[int]" = TRANSIENT_STATUSES,
    cookie_hosts: "_ty.Iterable[_ty.Iterable[str]] | None" = None,
):
    if transport is not None and not callable(getattr(transport, "send", None)):
        raise TypeError("a transport is an object with a send() method")
    if isinstance(retries, bool) or not isinstance(retries, int) or retries < 0:
        raise ValueError("retries is a count of retries, zero or more")
    if isinstance(backoff, bool) or not isinstance(backoff, (int, float)) or not 0 <= backoff < float("inf"):
        raise ValueError("backoff is a number of seconds, zero or more")
    self._env = HTTPEnv() if env is None else env
    options = {
        "proxymap": proxymap,
        "ssl_context": ssl_context,
        "proxy_ssl_context": proxy_ssl_context,
        "verify": verify,
    }
    self._transport = transport
    self._built = None
    self._transport_lock = _threading.Lock()
    if transport is not None:
        if any(value is not (True if name == "verify" else None) for name, value in options.items()):
            raise ValueError(
                "proxymap, ssl_context, proxy_ssl_context and verify configure the default transport: "
                "a client given a transport leaves them to it"
            )
    else:
        self._make, self._owned = transport_source(self._env, options)
        self.transport  # made now, so a setting that cannot be used is refused now
    self._cookies = make_cookies(cookies, self._env, cookie_aliases(cookie_hosts, self._env))
    self._tokens = make_tokens(tokens, self._env)
    self._auth_provider = auth_provider
    self._retries = retries
    self._backoff = float(backoff)
    self._status_forcelist = frozenset(status_forcelist)
    self._busy = _threading.local()
    self.headers = dict(headers or {})
    self.timeout = timeout
    self.deadline = check_deadline(deadline)
    self.user_agent = user_agent or f"syshttp/{__version__}"

`transport` `property`

The transport: the one the client was given, or the one it made (again, after close()). Read once per request.

`close()`

Closes the transport this client built for itself; one it was given stays the caller's. Harmless when called again.

Source code in src/syshttp/client/_client.py

def close(self) -> None:
    """Closes the transport this client built for itself; one it was given
    stays the caller's. Harmless when called again."""
    with self._transport_lock:
        built, self._built = self._built, None
        if built is not None:
            self._transport = None
    if built is not None:
        built.close()

`redirect(method, url, resp, *, headers=None, stream=False, timeout=_UNSET, deadline=_UNSET, auth=None)`

Send method to where resp, the answer to a request for url, redirects to, and follow what it redirects to in turn. The request carries what url is sent with, except that Authorization, Cookie and Proxy-Authorization are left off whenever it goes to another scheme, host or port than url's. A redirect with no usable Location is OSError(EIO). Cookies apply to every hop; the token, hooks and retries belong to request().

Source code in src/syshttp/client/_client.py

def redirect(
    self,
    method: str,
    url: str,
    resp: HTTPResponse,
    *,
    headers: "_ty.Mapping[str, str] | None" = None,
    stream: bool = False,
    timeout=_UNSET,
    deadline=_UNSET,
    auth: "tuple[str | bytes, str | bytes] | str | None" = None,
) -> HTTPResponse:
    """Send `method` to where `resp`, the answer to a request for `url`,
    redirects to, and follow what it redirects to in turn. The request
    carries what `url` is sent with, except that `Authorization`, `Cookie`
    and `Proxy-Authorization` are left off whenever it goes to another
    scheme, host or port than `url`'s. A
    redirect with no usable `Location` is `OSError(EIO)`. Cookies apply to
    every hop; the token, hooks and retries belong to `request()`."""
    sent = self._sent_headers(headers, auth)
    location = _follow.redirect_location(url, resp)
    if location is None:
        raise _follow.refused_redirect(method, url, resp, None, "no usable Location")
    clock = self._clock(deadline, method, location, first=url)
    return self._dispatch(method, location, sent, None, None, stream, timeout, clock, first=url)

`request(method, url, *, data=None, headers=None, stream=False, allow_redirects=None, timeout=_UNSET, deadline=_UNSET, auth=None)`

Send method to url and return the final response. A data that is not bytes (a file, an iterable) cannot be sent twice, so it is not re-sent after a redirect, a dropped connection or a retry. allow_redirects=False returns the first answer of a GET, HEAD or OPTIONS as it is. auth is a (user, password) pair sent as Authorization: Basic, or a str sent as the Authorization value ("Bearer <token>"), to the origin of url unless a header names an Authorization; it beats a token of the client. ValueError for a URL (credentials in it included) or a header with a control character, before anything is sent.

The auth provider, the hooks, the token and the retries of the client apply here. A hook is called as hook(client, method, url, kwargs) with the keyword arguments given (data, headers, stream, allow_redirects, timeout, deadline, auth), which it may change in place, and may return a response to be used instead of sending.

With stream=False the body is read before this returns, and a decoded body over MAX_REPLY_BYTES is OSError(EFBIG); with stream=True the caller reads it, and closes the response.

Source code in src/syshttp/client/_client.py

def request(
    self,
    method: str,
    url: str,
    *,
    data=None,
    headers: "_ty.Mapping[str, str] | None" = None,
    stream: bool = False,
    allow_redirects: "bool | None" = None,
    timeout=_UNSET,
    deadline=_UNSET,
    auth: "tuple[str | bytes, str | bytes] | str | None" = None,
) -> HTTPResponse:
    """Send `method` to `url` and return the final response. A `data` that
    is not `bytes` (a file, an iterable) cannot be sent twice, so it is not
    re-sent after a redirect, a dropped connection or a retry.
    `allow_redirects=False` returns the first answer of a `GET`, `HEAD` or
    `OPTIONS` as it is. `auth` is a `(user, password)` pair sent as
    `Authorization: Basic`, or a `str` sent as the `Authorization` value
    (`"Bearer <token>"`), to the origin of `url` unless a header names an
    `Authorization`; it beats a token of the client. `ValueError` for a URL
    (credentials in it included) or a header with a control character,
    before anything is sent.

    The auth provider, the hooks, the token and the retries of the client
    apply here. A hook is called as `hook(client, method, url, kwargs)` with
    the keyword arguments given (`data`, `headers`, `stream`,
    `allow_redirects`, `timeout`, `deadline`, `auth`), which it may change
    in place, and may return a response to be used instead of sending.

    With `stream=False` the body is read before this returns, and a decoded
    body over `MAX_REPLY_BYTES` is `OSError(EFBIG)`; with `stream=True` the
    caller reads it, and closes the response."""
    given: "dict[str, _ty.Any]" = {
        name: value
        for name, value, default in (
            ("data", data, None),
            ("headers", headers, None),
            ("stream", stream, False),
            ("allow_redirects", allow_redirects, None),
            ("timeout", timeout, _UNSET),
            ("deadline", deadline, _UNSET),
            ("auth", auth, None),
        )
        if value is not default
    }
    self._ensure(url)
    if not self._guarded("in_hook"):
        self._busy.in_hook = True
        try:
            hooked = _hooks.call_py_hooks(self, method, url, given, env=self._env)
        finally:
            self._busy.in_hook = False
        if hooked is not None:
            return hooked
    unknown = sorted(set(given) - _REQUEST_OPTIONS)
    if unknown:
        raise TypeError("a hook set %s, which request() does not take" % ", ".join(unknown))
    return self._send(
        method,
        url,
        given.get("data"),
        given.get("headers"),
        given.get("stream", False),
        given.get("allow_redirects"),
        given.get("timeout", _UNSET),
        given.get("deadline", _UNSET),
        given.get("auth"),
    )

`HTTPResponse(status, reason, headers, url, body=b'', *, decode=True)`

One response: status, reason, headers (a read-only mapping that ignores the case of a name), url (the URL the request went to) and the body, read with read() or iter_bytes(). A context manager that closes.

The body is decoded when the response says it is gzip or deflate encoded, unless the request set its own Accept-Encoding; headers are the headers as received, so Content-Length counts the encoded bytes. A body that ends before the length the server stated is OSError(EIO), never a shorter body that reads as complete.

A transport builds one with body as bytes, or as an object with read(amount) -> bytes (empty at the end) and close(), and decode=False for a body it already decoded or that was not asked for in a coding.