Skip to content

Running your system

Both backends present the same surface: a cluster object you enter with with ... as c:, numbered nodes (1-based, matching NODE_ID), the same process faults, and the same PEERS contract. Tests written against one backend run unchanged on the other — the only thing that changes is which class your hokea_cluster fixture constructs.

Start locally with Cluster (Docker); move to KubeCluster when you deploy to the class Kubernetes cluster.

Docker backend

hokea.cluster

Cluster: run your system as a fleet of Docker containers.

You point hokea at a base image, a source directory, and a start command (or supply your own Dockerfile). hokea generates the compose configuration, builds the image, and runs N nodes attached to two networks:

  • the client network: how tests and load generators reach nodes. Never faulted.
  • the cluster network: how nodes reach each other (PEERS points here). This is the only network faults touch.

The client network is attached at higher priority so it becomes the container's primary interface: published ports and harness traffic never ride the interface that partitions and netem act on.

The contract a system must meet: read NODE_ID (1-based), N_NODES, and PEERS (comma-separated addresses on the cluster network), and answer GET /health with HTTP 200 once ready to serve.

Everything Docker-visible is prefixed hokea-<name>-; cleanup only ever touches this cluster's own compose project.

For example:

with Cluster(image="python:3.12-slim", src="./myserver",
             cmd="python server.py", nodes=5) as c:
    c.wait_healthy()
    ...

Cluster

Cluster(
    *,
    image: str | None = None,
    src: str | None = None,
    cmd: str | None = None,
    dockerfile: str | None = None,
    data: str | Path | dict | None = None,
    nodes: int = 3,
    name: str = "default",
    port: int = 8000,
    peer_port: int | None = None,
    env: dict[str, str] | None = None,
    cpus: float = 1.0,
    memory: str = "512m",
    workdir: str | Path | None = None,
    up_timeout: float = 300,
)
with Cluster(image="python:3.12-slim", src="./myserver",
             cmd="python server.py", nodes=5) as c:
    c.wait_healthy()
    ...

Either image+src+cmd (no-Dockerfile mode) or dockerfile= (a build context directory containing a Dockerfile). port is the port the system listens on inside the container; hokea publishes it per node on an ephemeral host port. peer_port (default: port) is the port written into PEERS entries.

data= mounts a dataset (a log corpus, MNIST, ...) read-only in every node: a single directory path appears at /dataset; a dict {name: directory} appears at /dataset/<name> each. It is a live bind mount of the host directory — edits on the host show up in the nodes. The k8s backend takes the same shapes at the same paths, but sources there are OCI image references (see KubeCluster). /data stays the separate per-node writable scratch tmpfs.

up

up()

down

down()

wait_healthy

wait_healthy(timeout: float = 60)

Poll every node's GET /health until all answer 200, or raise with the failing nodes' recent logs.

node

node(i: int) -> Node

nodes

nodes() -> list[Node]

make_net

make_net()

The fault injector matching this backend (the pytest plugin's net fixture calls this, so tests work unchanged on any backend).

make_chaos

make_chaos()

kill

kill(i: int)

stop

stop(i: int)

start

start(i: int)

pause

pause(i: int)

unpause

unpause(i: int)

throttle

throttle(i: int, *, cpus: float)

Make node i a compute straggler: change its CPU quota at runtime via docker update --cpus. Where Net.slow() makes a node's network slow, this makes the machine slow: it computes slowly but talks fine, the kind of slow node a MapReduce or distributed-training project has to handle. Verified by reading the applied quota back from docker inspect rather than by timing, which is unreliable on a loaded machine. A fresh up() recreates containers at the configured cpus, so a leftover throttle never survives into the next run.

unthrottle

unthrottle(i: int)

Restore node i to the cluster's configured cpus, verified the same way throttle() is.

on_state_change

on_state_change(callback)

Node

Node(cluster: Cluster, index: int)

One running container. index is 1-based, matching NODE_ID.

url property

url: str

Base URL for reaching this node from the harness (client network). Stable for the cluster's lifetime, even while the node is down or paused — clients keep a fixed address and observe the outage.

host_port

host_port() -> int

cluster_ip

cluster_ip() -> str

exec

exec(*argv: str, check: bool = True)

logs

logs(tail: int = 50) -> str

is_running

is_running() -> bool

status

status() -> str

Docker's view: "running", "paused", "exited", ... — note a paused container still counts as Running for is_running().

dataset_mounts

dataset_mounts(data) -> dict

data= as {mount_path: source}, shared by both backends: a single source mounts at /dataset (the original form), a dict {name: source} mounts each at /dataset/<name>. Names become path components and (on k8s) volume names, so they must be DNS-label-ish; what a source is — a host directory here, an OCI image reference on Kubernetes — is the backend's own validation.

Kubernetes backend

hokea.kube

Kubernetes backend: the fleet as pods, faults via Chaos Mesh.

Same surface as the Docker backend — the same tests, adapters, ops generators, and checks run unchanged; what changes is one conftest fixture:

from hokea.kube import KubeCluster

@pytest.fixture
def hokea_cluster(tmp_path):
    return KubeCluster(image="python:3.12-slim", src=HERE,
                       cmd="python server.py", nodes=3, name="myproj",
                       namespace="team-NN", port=8000,
                       workdir=tmp_path / "cluster")

How the Docker backend's concepts map:

  • N containers -> N bare pods named {prefix}-node<i>, labeled hokea/cluster={prefix} and hokea/node=<i>. restartPolicy=Never: hokea's kill()/start() are the only ways a node dies and returns, so a run's fault timeline stays the whole truth.
  • no-Dockerfile mode -> no-registry mode: the source directory is shipped as a ConfigMap mounted read-only at /app on a public base image. There is nothing to build or push. (Limits: regular files in the top level of src=, ~1MiB total. A system that outgrows this builds an image, pushes it to any registry the cluster can pull from, and passes image= alone.)
  • the PEERS contract is preserved exactly: each pod gets hostname node-<i>, subdomain {prefix}-peers (a headless Service), and a dnsConfig search entry so the bare name node-<i> resolves — your system connects to node-2:8000 identically under both backends.
  • the client/cluster network split -> two mechanisms, layered. Selector scoping first: every Chaos Mesh CR hokea creates selects only the fleet's pods as source AND target, so pod-to-pod traffic is faulted while harness traffic never matches a rule. Namespaces second, when the harness itself runs inside the cluster: server pods live in the team namespace (annotated chaos-mesh.org/inject=enabled) and the harness runs as a pod in a separate CLIENT namespace that deliberately lacks that annotation — with Chaos Mesh's namespace filter on, faults structurally cannot reach the observer, which is the same guarantee the Docker backend gets from its never-faulted client network. up() checks both annotations when RBAC allows (see _preflight_internal).
  • how the harness reaches nodes -> expose modes. expose="nodeport" publishes one NodePort Service per pod, for a harness OUTSIDE the cluster that can route to the k8s nodes. expose="internal" publishes nothing: node.url is the pod's per-pod DNS record behind the headless peers Service, written as an FQDN so it resolves cross-namespace from the client namespace. The default auto-detects — internal iff this process is itself running in a pod.
  • network faults -> NetworkChaos CRs. Chaos Mesh experiments are level-triggered: a CR keeps its fault installed until the CR is deleted, and re-applying a finished object does nothing. hokea therefore creates a FRESH CR (generateName) for every fault action and heals by deleting its CRs — it never re-applies one.
  • verification is still hokea's own: after Chaos Mesh reports AllInjected, hokea probes pod-to-pod reachability through kubectl exec and raises if the fault is not in the state the test believes.
  • process faults: kill(i) force-deletes the pod; stop(i) deletes it with a grace period; start(i) re-creates it from the saved manifest (fresh pod IP; peers re-resolve DNS, active network faults are re-established via the same state-change hook the Docker backend uses). throttle(i) shrinks the pod's CPU limit in place via the resize subresource (k8s >= 1.33) — no restart, process state survives. pause is NOT implemented here — see the class docstring for what to use instead.
  • faults hokea did NOT inject are recorded too: a watcher thread polls the namespace's Chaos Mesh CRs and logs foreign experiments (a kubectl- or dashboard-applied StressChaos, IOChaos, TimeChaos, ...) into the fault timeline as chaos/chaos_done/chaos_recovered events, so faults.jsonl stays the record of the run even when you drive Chaos Mesh by hand.
  • your own chaos -> chaos.apply: the chaos fixture (KubeChaos) takes any Chaos Mesh experiment or Workflow YAML, rewrites it to survive level-triggering (fresh generateName per apply), creates it inside the fault schedule, waits for injection, and clear() deletes it — so a chaos experiment of your own design is one step in a pytest test, correctly timed, recorded, and cleaned up.

Everything hokea creates is namespaced, labeled app.kubernetes.io/managed-by=hokea + hokea/cluster={prefix}, and removed by down() with a label selector — teardown of a whole namespace is never hokea's job.

Requires: kubectl on PATH configured for the target cluster, a namespace you may create pods in, and Chaos Mesh installed cluster-wide.

For example:

with KubeCluster(image="python:3.12-slim", src="./myserver",
                 cmd="python server.py", nodes=5,
                 namespace="team-NN") as c:
    c.wait_healthy()
    ...

KubeCluster

KubeCluster(
    *,
    image: str,
    src: str | None = None,
    cmd: str | None = None,
    nodes: int = 3,
    name: str = "default",
    namespace: str,
    port: int = 8000,
    peer_port: int | None = None,
    env: dict[str, str] | None = None,
    cpus: float = 1.0,
    memory: str = "512Mi",
    expose: str | None = None,
    node_host: str | None = None,
    workdir: str | Path | None = None,
    up_timeout: float = 300,
    takeover: bool = False,
    data: str | dict | None = None,
)
with KubeCluster(image="python:3.12-slim", src="./myserver",
                 cmd="python server.py", nodes=5,
                 namespace="team-NN") as c:
    c.wait_healthy()
    ...

Either image+src+cmd (no-registry mode: src becomes a ConfigMap mounted at /app) or image= alone (a pre-built image from a registry the cluster can pull; its own entrypoint runs). namespace must exist and allow you to create pods, services, and configmaps.

expose picks how the harness reaches nodes. "nodeport": one NodePort Service per node — for a harness OUTSIDE the cluster that can route to the k8s nodes. "internal": no Services are published at all; node.url is the pod's own DNS record (node-.-peers..svc...) — for a harness running INSIDE the cluster as a pod, typically in a separate client namespace (team-NN-clients next to team-NN). The default, None, auto-detects: internal iff this process itself runs in a pod. The client namespace is this backend's version of the Docker backend's client network — it must NOT carry the chaos-mesh.org/inject=enabled annotation, so Chaos Mesh structurally cannot fault the observer; up() verifies both namespaces' annotations when RBAC allows (_preflight_internal), and chaos.apply refuses experiments whose selectors name the harness's namespace. Inside a pod, kubectl needs no kubeconfig: the mounted ServiceAccount authenticates it.

node_host (nodeport mode only) is the address the harness reaches NodePorts at; it defaults to the first cluster node's InternalIP, which is right when the harness runs somewhere that can route to the nodes (a lab machine on the same network). Set it explicitly otherwise.

data= mounts a dataset read-only in every node, at the same paths as the Docker backend: a single source appears at /dataset, a dict {name: source} at /dataset/<name> each. The difference is what a source IS: Docker takes a host directory; here it is an OCI image reference, mounted as a pod image volume (k8s >= 1.35). A dataset image is just a filesystem shipped as an image — a two-line Dockerfile:

FROM scratch
COPY corpus/ /

built and tagged like any image. The pull policy is IfNotPresent, so a ref already in the node's containerd store works with NO registry at all (side-loaded or left over from an earlier pull), which keeps the default packaging free of any registry. containerd also dedupes by ref: one on-disk copy per cluster node no matter how many pods or teams mount it. Parity caveat: Docker's data= is a live bind mount of a host directory, so edits show up immediately; an image volume is immutable — to change the dataset, push a new tag and pass that.

Not implemented in this backend (loud NotImplementedError, with the Chaos Mesh alternative named): pause/unpause — Kubernetes has no SIGSTOP analog for a pod (PodChaos pod-failure replaces the container, losing process state, so it models a different fault).

up

up()

down

down()

Delete everything this cluster created — and only that: deletion goes by the hokea/cluster label, never by namespace.

wait_healthy

wait_healthy(timeout: float = 90)

node

node(i: int) -> KubeNode

nodes

nodes() -> list[KubeNode]

make_net

make_net()

make_chaos

make_chaos()

For faults hokea has no wrapper for: a KubeChaos that applies your own Chaos Mesh YAML (any experiment kind, or a Workflow) inside the fault schedule. The pytest plugin's chaos fixture calls this, mirroring make_net().

kill

kill(i: int)

Force-delete the pod immediately (no grace period). start(i) brings it back.

A force-delete returns when the API object is gone, which can be a second or two BEFORE the kubelet actually reaps the process — so kill_done is recorded only once the node stops answering /health: the availability window must not open while the node still serves.

stop

stop(i: int)

Graceful shutdown: SIGTERM, 5s grace, then gone. Like kill(), stop_done is recorded only once the node stops answering.

start

start(i: int)

Re-create the pod from the manifest up() built. Fresh pod IP and empty /data — a returning node proves (or disproves) durability.

The delete-first matters: if the node's process died on its own (crash, OOMKill), the pod object lingers in phase Failed and kubectl apply of the unchanged manifest would be a no-op — start() must revive a self-crashed node, not just one hokea killed.

start_done is recorded only once the node answers /health at node.url again — the mirror image of kill()/stop()'s wait-until-unreachable. Pod Ready is not enough: the re-created pod has a NEW IP, and the headless Service's DNS record for its stable name can lag behind it, so the next dial would connect-timeout. This also tightens the availability-window semantics: the window a kill/stop opened closes here, when the node truly serves again.

pause

pause(i: int)

throttle

throttle(i: int, *, cpus: float)

Make node i a compute straggler: shrink its CPU limit at runtime via the pod resize subresource (Kubernetes >= 1.33) — the pod is NOT restarted, process state survives, exactly like the Docker backend's docker update --cpus. Memory limits are untouched. Verified by reading the applied values back from the pod (spec and container status), not by timing, which flakes on a loaded cluster. A fresh up() creates pods at the configured cpus, so a leftover throttle never survives into the next run.

unthrottle

unthrottle(i: int)

Restore node i to the cluster's configured cpus, verified the same way throttle() is.

on_state_change

on_state_change(callback)

KubeNode

KubeNode(cluster: KubeCluster, index: int)

One pod. index is 1-based, matching NODE_ID. name is the pod name; the pod's hostname is node-<index> (what PEERS entries resolve to).

url property

url: str

Base URL for reaching this node from the harness. Stable for the cluster's lifetime in either expose mode:

  • nodeport: http://:. The Service selects by label, so a killed-and-restarted pod keeps the same URL.
  • internal: the pod's per-pod DNS record behind the headless peers Service, as an FQDN so it resolves from the harness's own (client) namespace. The NAME is stable; while the pod is down the record does not resolve, which the harness reads as "unreachable" exactly like a refused NodePort connection.

host_port

host_port() -> int

The NodePort publishing this node — the k8s analog of the Docker backend's per-node host port. nodeport expose mode only.

cluster_ip

cluster_ip() -> str

The pod IP (changes when the pod is re-created).

exec

exec(*argv: str, check: bool = True)

logs

logs(tail: int = 50) -> str

is_running

is_running() -> bool

status

status() -> str

"running", "pending", "failed", "succeeded", or "gone" — the pod phase, lowercased. NOT identical to the Docker backend's vocabulary: a killed node is "gone" here (the pod object is deleted) where Docker reports "exited", and "paused" cannot occur.