Faults: partitions, netem, chaos¶
Network faults come from the object the net fixture hands you — a Net on
the Docker backend, a KubeNet on Kubernetes, with the same methods and the
same rule: every fault is verified before your test continues, and
every action lands in the run's fault timeline (see the
artifact formats). Slowness faults use netem, Linux's
network-emulation tool. Process faults (kill, stop,
throttle, ...) live on the cluster object itself. On
Kubernetes only, KubeChaos (the chaos fixture) applies Chaos Mesh
experiment files (YAML) of your own design inside the fault schedule.
Docker backend¶
hokea.net
¶
Net: verified network fault injection on the cluster network.
net = Net(cluster)
net.partition([[1, 2, 3], [4, 5]]) # groups talk within, not across
net.isolate(4) # shorthand: one node vs. everyone
net.slow(3, delay_ms=200, jitter_ms=50, loss_pct=5)
net.heal() # remove every fault hokea added
Mechanism: iptables DROP rules matching cluster-network address pairs, plus
tc netem on the cluster-network interface for latency/loss. Two backends:
- host: rules in the host's DOCKER-USER chain. Used where the host has root and a real Docker daemon; less exercised than the container backend (Docker Desktop cannot run it), so it announces itself when active.
- container: rules installed inside the containers via docker exec. Needs iptables in the image and NET_ADMIN (hokea adds the capability; generated images get iptables installed). Checked at startup with a clear error.
Rules only ever match specific peer cluster-IP pairs — never the gateway or a subnet — so the harness can always observe every node through the client network. Rules go on both ends of each pair: a paused or crashed container can't run iptables, but its counterparts still drop the traffic. Every rule carries a comment tag so heal() and the startup sweep remove exactly hokea's rules and nothing else.
Faults are verified, not assumed: after installing a partition, probes prove traffic is blocked across the cut in both directions and alive within groups and from the harness — and raise if not. The controller remembers intended state; if a node restarts while a fault is active, rules are re-applied at the node's possibly-new address and re-verified (Cluster notifies Net on start/unpause). Every action lands in faults.jsonl as a requested/verified event pair.
netem is always applied in-container on the cluster-network interface (there is no "host tc" for a container's interface), so latency never touches the path tests measure on.
Net
¶
Net(cluster, backend: str = 'auto')
partition
¶
partition(groups: list[list[int]])
Cut the cluster network between groups. Groups must cover every node exactly once. Raises unless probes prove the cut is real.
slow
¶
slow(
i: int,
*,
delay_ms: int,
jitter_ms: int = 0,
loss_pct: float = 0,
)
Shape node i's egress on the cluster network with netem.
Kubernetes backend¶
KubeNet
¶
KubeNet(cluster: KubeCluster)
Verified network faults on the pod fleet, implemented as Chaos Mesh NetworkChaos CRs. Same API and same discipline as Net:
net.partition([[1, 2, 3], [4, 5]])
net.slow(3, delay_ms=200, jitter_ms=50, loss_pct=5)
net.heal()
Chaos Mesh experiments are level-triggered — a CR holds its fault until deleted, and re-applying a completed CR is a no-op — so every action creates a FRESH CR via generateName and heal() deletes this cluster's CRs (by label). After Chaos Mesh reports AllInjected, hokea still probes reachability itself and raises on any mismatch: a partition that silently didn't happen would produce confident wrong conclusions.
Scoping: every CR's source AND target select only this cluster's pods (hokea/cluster label), so harness traffic — the "client network", arriving via NodePorts in nodeport expose mode or straight from the client namespace's pods in internal mode — is never faulted. The cut covers the direct pod-to-pod path, i.e. exactly what the PEERS names resolve to — which is also the path the probes verify. Systems must reach peers via PEERS only: traffic routed through a Kubernetes Service VIP is not reliably cut (the OUTPUT-side rule fires before kube-proxy's DNAT) and such a leak is invisible to verification.
If a node is killed and re-started while faults are active, the pod returns under a new IP; Chaos Mesh pins a CR's targets at creation, so hokea deletes and re-creates its CRs (fresh objects, per the level-triggered rule) and re-verifies — the same reconciliation the Docker backend does with iptables rules.
slow
¶
slow(
i: int,
*,
delay_ms: int,
jitter_ms: int = 0,
loss_pct: float = 0,
)
Shape node i's egress toward its peers with a netem NetworkChaos. Verified as: AllInjected on the CR, plus a recorded probe RTT from a same-side peer (recorded, not asserted — loaded clusters make exact timing assertions flaky).
heal
¶
heal()
Delete every Chaos Mesh CR this cluster created, and prove recovery: probes across the former cut, and a recorded RTT to each formerly-slowed node (a chaos-daemon hiccup leaving netem behind after the CR is gone would otherwise poison every later baseline).
KubeChaos
¶
KubeChaos(cluster: KubeCluster)
Your own chaos, timed by the fault schedule: apply a Chaos Mesh experiment (or Workflow) you wrote, at a known moment in a recorded run, and clean it up — one step in a pytest test instead of a sequence of kubectl commands run by hand.
def test_my_stress(cluster, chaos, run):
h = run(ops, adapter, duration=60,
faults=[(10, lambda: chaos.apply("stress.yaml")),
(40, chaos.clear)])
apply() rewrites each document to survive Chaos Mesh's level-triggered
lifecycle (fresh generateName per call — re-running the same file always
injects), forces it into the cluster's namespace, stamps hokea's labels,
creates it, and by default waits until Chaos Mesh reports the fault
injected. In internal expose mode it first refuses any document whose
selector names the harness's own namespace — the client namespace is
the protected observer, never a fault target. Because the labels make the ChaosWatcher skip these CRs (they
are this cluster's own), apply()/clear() record the events themselves:
"chaos" at creation, "chaos_done" once injected, "chaos_recovered" at
clear() — the same shapes the watcher writes, so check.availability()
windows them identically (label chaos:<kind>:<generated-name>).
Workflows: a Workflow has no AllInjected condition — apply() waits until the workflow's entry node exists (status.entryNode) and records chaos_done with workflow=True; that window means "the workflow is running", not "a fault is in". The workflow's child experiments are created by Chaos Mesh WITHOUT hokea's label, so the ChaosWatcher records each child individually as it fires — faults.jsonl shows the workflow's actual steps with their real timestamps. Note the timing model: a workflow times its steps relative to its own start, cluster-side; anchoring that start is exactly what the fault schedule offset does.
clear() deletes every CR this KubeChaos applied (--wait=true: Chaos Mesh finalizers make deletion mean recovery complete). Deleting a Workflow cascades to its children via ownerReferences; the children's recoveries are recorded by the watcher. Anything not cleared is deleted by the cluster's labeled sweep at down() — nothing outlives the run.
Docker backend: Cluster.make_chaos() raises — Chaos Mesh only exists on Kubernetes; locally the built-in faults cover the same ground.
apply
¶
apply(
source, *, wait: bool = True, timeout: float = 60
) -> list[str]
Create the chaos object(s) in source — a YAML file path, a
dict, or a list of dicts — and return the created object refs
(e.g. ["stresschaos.chaos-mesh.org/cpu-stress-x7k2m"]).
wait=True (default) blocks until every created experiment reports
AllInjected (a workflow: until its entry node starts) or raises
after timeout seconds — so the next schedule step runs with the
fault actually in, no timing guesses. wait=False returns as soon as
the API accepted the objects.
clear
¶
clear()
Delete every CR this KubeChaos applied, waiting for Chaos Mesh's finalizers (deletion returns once recovery is complete), and record chaos_recovered for each. Safe to call with nothing applied.