Skip to content

Workload & runs

run_workload puts concurrent clients on the cluster through your adapter and ops generator, executes the fault schedule, and writes a run directory; record_run does the same for systems with no request/response load (batch jobs, training). Both return a RunHandle, which reads the run's files back for checks and your own analysis.

Driving load

hokea.workload

Workload driver: concurrent clients executing an operation mix you define against the cluster, with everything recorded.

You provide two pieces:

  • an operation generator: ops(client_id, rng) returning a Python generator that yields Op objects. The driver sends each op's Result back into the generator, so reactive flows work naturally:

    def my_ops(client_id, rng):
        while True:
            res = yield Op("reserve", seat, {"client_id": f"c{client_id}"})
            if res.outcome == "ok":
                yield Op("confirm", seat, {"rid": res.response["rid"]})
    
  • a client adapter with execute(op, node, timeout) -> Result, speaking the system's own protocol. node.url is the node's base URL.

Each operation's outcome records what we KNOW about whether it took effect:

ok       — yes: the system acknowledged it
rejected — no: the server itself said no, and nothing changed
unknown  — can't tell. When in doubt, use this.

hokea forces unknown whenever it observes a timeout or dropped connection itself (a transport exception escaping the adapter), regardless of what the adapter would have said — an adapter mistake can only make checks more cautious, never wrongly confident. Any other exception from the adapter is a bug in the adapter and crashes the run loudly. Clients never retry: each operation is attempted exactly once and recorded exactly once.

Clients are worker processes (forked), so load is not capped by Python's threading. Each worker writes its own shard; the parent merges them into history.jsonl ordered by invocation time.

An Op is what your ops generator yields; a Result is what your adapter returns. The fields, and the rules for choosing a Result's ok / rejected / unknown outcome, are documented where you write that code: the adapters guide.

Op dataclass

Op(op: str, key: str, args: dict = dict())

op instance-attribute

op: str

key instance-attribute

key: str

args class-attribute instance-attribute

args: dict = field(default_factory=dict)

Result dataclass

Result(
    outcome: str,
    status: int | None = None,
    response: object = None,
    version: object = None,
    error: str | None = None,
)

outcome instance-attribute

outcome: str

status class-attribute instance-attribute

status: int | None = None

response class-attribute instance-attribute

response: object = None

version class-attribute instance-attribute

version: object = None

error class-attribute instance-attribute

error: str | None = None

http_result

http_result(
    response,
    *,
    rejected: frozenset | set = frozenset(),
    version=None,
) -> Result

Classify an HTTP response the safe way: 2xx -> ok, a status code you explicitly listed in rejected -> rejected, anything else -> unknown.

The point of listing codes explicitly: "rejected" is a strong claim — the server said no AND nothing changed — so you must opt each status code into it deliberately. A code you didn't think about (a 503, a proxy's 502, an unexpected 500) can never be misfiled as rejected by accident; it lands on unknown, which is always safe.

return http_result(r, rejected={400, 409})

Script

Script(adapter, *, timeout: float = 5.0)

A harness-side scripted client for choreographed steps: "at this point in the timeline, run this one operation against this one node."

script = Script(MyAdapter())
h = run_workload(cluster, ops, adapter, duration=30, script=script,
                 faults=[(10, lambda: net.partition([[1, 2], [3]])),
                         (0,  lambda: script.do(Op("read", "k"), node=3))])

Steps run in the fault schedule (parent process, in order — and since e.g. net.partition() returns only after the cut is verified, a step after it needs no timing guesses). Each step goes through the adapter with the same outcome rules as worker clients, returns its Result so later steps can use ids from responses, and is recorded in history.jsonl under the reserved client_id 0. Script ops are not counted as load: they are excluded from report.json.

do

do(op: Op, *, node: int) -> Result

run_workload

run_workload(
    cluster,
    ops,
    adapter,
    *,
    duration: float,
    clients: int = 4,
    rate: float | None = None,
    faults: list | None = None,
    assign: str = "round_robin",
    timeout: float = 5.0,
    seed: int = 0,
    sample: float | None = None,
    state_reader=None,
    script: Script | None = None,
    settle: float = 0,
    name: str = "run",
    run_dir: str | Path | None = None,
) -> RunHandle

Drive clients concurrent workers for duration seconds, executing the fault schedule (faults is a list of (offset_seconds, callable), run in list order) from the parent process, and write the full run record to a run directory.

rate is the target total ops/sec across all clients (None = closed loop: every client issues its next op as soon as the last returns). assign is how ops map to nodes: "round_robin", "pinned", or "random". state_reader(node) -> dict (optional) fetches each node's final state. script attaches a Script whose choreographed ops land in the history. settle waits that many quiet seconds after clients finish before the final states are captured — systems that sync in the background need it to converge; the wait is recorded in the fault timeline and metadata.

record_run

record_run(
    cluster,
    *,
    name: str = "record",
    settle: float = 0,
    state_reader=None,
    run_dir: str | Path | None = None,
)

A run with no client workload: server nodes + faults + events only.

For batch and training projects (a MapReduce job, distributed SGD) where nothing drives request/response load, but the fault timeline and the node event stream are still the record that checks and analysis read. Produces the same run directory as run_workload minus history.jsonl and report.json:

with record_run(cluster, name="train-sync") as r:
    net.slow(3, delay_ms=200)
    time.sleep(600)              # nodes work, emitting events
    net.heal()
check.event_join(r.events, require="commit", for_each="ack",
                 on="op_id")

The yielded RunHandle's accessors are valid once the block exits. The record is written even if the body raises — but then without the completed marker in metadata.json, so a partial run is detectable. settle (quiet seconds before state capture, recorded in the timeline) and state_reader behave as in run_workload.

Reading a run back

hokea.runs

Run artifacts: what a test run writes to disk, and how it is read back.

These file formats are the stable contract of the framework — checks, exporters, and any tools of your own all read them. A run directory holds:

history.jsonl   one line per client operation
faults.jsonl    the fault timeline (paired requested/done|verified events)
events.jsonl    server-side events collected from node stdout
states/         optional end-of-run state snapshot per node
metadata.json   run configuration, clock anchor, completion marker
report.json     throughput / latency / error-mix summary

All timestamps in history and faults are seconds since run start, measured on the harness host's monotonic clock — one clock, which is what makes "operation A finished before operation B started" a legitimate claim. metadata.json carries {wall_at_start, mono_at_start} to recover wall time.

RunHandle

RunHandle(run_dir: str | Path)

Access to a finished run's artifacts. Everything is read from the files — any run can be re-analyzed later without re-running the cluster.

history property

history: list[dict]

faults property

faults: list[dict]

events property

events: list[dict]

states property

states: dict[int, dict]

Per-node final state, keyed by node index. A node that could not be read appears with {"unreachable": true, ...} — checks must treat that as missing evidence, never as an empty state.

metadata property

metadata: dict

report property

report: dict