Workload & runs¶
run_workload puts concurrent clients on the cluster through
your adapter and ops generator, executes the fault schedule,
and writes a run directory; record_run does the same for systems with no
request/response load (batch jobs, training). Both return a RunHandle,
which reads the run's files back for
checks and your own analysis.
Driving load¶
hokea.workload
¶
Workload driver: concurrent clients executing an operation mix you define against the cluster, with everything recorded.
You provide two pieces:
-
an operation generator:
ops(client_id, rng)returning a Python generator that yields Op objects. The driver sends each op's Result back into the generator, so reactive flows work naturally:def my_ops(client_id, rng): while True: res = yield Op("reserve", seat, {"client_id": f"c{client_id}"}) if res.outcome == "ok": yield Op("confirm", seat, {"rid": res.response["rid"]}) -
a client adapter with
execute(op, node, timeout) -> Result, speaking the system's own protocol.node.urlis the node's base URL.
Each operation's outcome records what we KNOW about whether it took effect:
ok — yes: the system acknowledged it
rejected — no: the server itself said no, and nothing changed
unknown — can't tell. When in doubt, use this.
hokea forces unknown whenever it observes a timeout or dropped connection
itself (a transport exception escaping the adapter), regardless of what the
adapter would have said — an adapter mistake can only make checks more
cautious, never wrongly confident. Any other exception from the adapter is
a bug in the adapter and crashes the run loudly. Clients never retry: each
operation is attempted exactly once and recorded exactly once.
Clients are worker processes (forked), so load is not capped by Python's threading. Each worker writes its own shard; the parent merges them into history.jsonl ordered by invocation time.
An Op is what your ops generator yields; a Result is what your adapter
returns. The fields, and the rules for choosing a Result's
ok / rejected / unknown outcome, are documented where you write that
code: the adapters guide.
Op
dataclass
¶
Op(op: str, key: str, args: dict = dict())
Result
dataclass
¶
Result(
outcome: str,
status: int | None = None,
response: object = None,
version: object = None,
error: str | None = None,
)
http_result
¶
http_result(
response,
*,
rejected: frozenset | set = frozenset(),
version=None,
) -> Result
Classify an HTTP response the safe way: 2xx -> ok, a status code you
explicitly listed in rejected -> rejected, anything else -> unknown.
The point of listing codes explicitly: "rejected" is a strong claim — the server said no AND nothing changed — so you must opt each status code into it deliberately. A code you didn't think about (a 503, a proxy's 502, an unexpected 500) can never be misfiled as rejected by accident; it lands on unknown, which is always safe.
return http_result(r, rejected={400, 409})
Script
¶
Script(adapter, *, timeout: float = 5.0)
A harness-side scripted client for choreographed steps: "at this point in the timeline, run this one operation against this one node."
script = Script(MyAdapter())
h = run_workload(cluster, ops, adapter, duration=30, script=script,
faults=[(10, lambda: net.partition([[1, 2], [3]])),
(0, lambda: script.do(Op("read", "k"), node=3))])
Steps run in the fault schedule (parent process, in order — and since e.g. net.partition() returns only after the cut is verified, a step after it needs no timing guesses). Each step goes through the adapter with the same outcome rules as worker clients, returns its Result so later steps can use ids from responses, and is recorded in history.jsonl under the reserved client_id 0. Script ops are not counted as load: they are excluded from report.json.
run_workload
¶
run_workload(
cluster,
ops,
adapter,
*,
duration: float,
clients: int = 4,
rate: float | None = None,
faults: list | None = None,
assign: str = "round_robin",
timeout: float = 5.0,
seed: int = 0,
sample: float | None = None,
state_reader=None,
script: Script | None = None,
settle: float = 0,
name: str = "run",
run_dir: str | Path | None = None,
) -> RunHandle
Drive clients concurrent workers for duration seconds, executing the
fault schedule (faults is a list of (offset_seconds, callable), run in
list order) from the parent process, and write the full run record to a
run directory.
rate is the target total ops/sec across all clients (None = closed loop:
every client issues its next op as soon as the last returns).
assign is how ops map to nodes: "round_robin", "pinned", or "random".
state_reader(node) -> dict (optional) fetches each node's final state.
script attaches a Script whose choreographed ops land in the history.
settle waits that many quiet seconds after clients finish before the
final states are captured — systems that sync in the background need it
to converge; the wait is recorded in the fault timeline and metadata.
record_run
¶
record_run(
cluster,
*,
name: str = "record",
settle: float = 0,
state_reader=None,
run_dir: str | Path | None = None,
)
A run with no client workload: server nodes + faults + events only.
For batch and training projects (a MapReduce job, distributed SGD) where nothing drives request/response load, but the fault timeline and the node event stream are still the record that checks and analysis read. Produces the same run directory as run_workload minus history.jsonl and report.json:
with record_run(cluster, name="train-sync") as r:
net.slow(3, delay_ms=200)
time.sleep(600) # nodes work, emitting events
net.heal()
check.event_join(r.events, require="commit", for_each="ack",
on="op_id")
The yielded RunHandle's accessors are valid once the block exits. The
record is written even if the body raises — but then without the
completed marker in metadata.json, so a partial run is detectable.
settle (quiet seconds before state capture, recorded in the timeline)
and state_reader behave as in run_workload.
Reading a run back¶
hokea.runs
¶
Run artifacts: what a test run writes to disk, and how it is read back.
These file formats are the stable contract of the framework — checks, exporters, and any tools of your own all read them. A run directory holds:
history.jsonl one line per client operation
faults.jsonl the fault timeline (paired requested/done|verified events)
events.jsonl server-side events collected from node stdout
states/ optional end-of-run state snapshot per node
metadata.json run configuration, clock anchor, completion marker
report.json throughput / latency / error-mix summary
All timestamps in history and faults are seconds since run start, measured on the harness host's monotonic clock — one clock, which is what makes "operation A finished before operation B started" a legitimate claim. metadata.json carries {wall_at_start, mono_at_start} to recover wall time.
RunHandle
¶
RunHandle(run_dir: str | Path)
Access to a finished run's artifacts. Everything is read from the files — any run can be re-analyzed later without re-running the cluster.
states
property
¶
states: dict[int, dict]
Per-node final state, keyed by node index. A node that could not be read appears with {"unreachable": true, ...} — checks must treat that as missing evidence, never as an empty state.