Writing an adapter (and an operation generator)¶
The adapter is a piece of code you write to allow hokea to drive your project; it's critical that this code is correct as a mistake here can poison your results. Thankfully, adapters should only be about 30-40 lines of code. Read this whole page before writing your own!
The contract¶
class MyAdapter:
def execute(self, op: Op, node, timeout: float) -> Result:
...
Execute exactly one operation against exactly one node (node.url is its
base URL, node.index its 1-based id), using your system's own protocol —
HTTP, gRPC, your client library, anything. Return a Result.
The three outcomes — what they mean and how to choose¶
The outcome is not a status code. It records what you know about whether the operation took effect on the system: yes, no, or can't tell.
ok— yes: the server acknowledged it ("done"). Not merely "the request got a response": a 500 ("internal server error") is a response but is not an acknowledgment.rejected-- no, provably: the server itself said no, and nothing changed, and nothing ever will because of this request: e.g., HTTP 409 "seat taken", 400 bad request, or a validation error.unknown— can't tell. Timeouts, dropped connections, any 5xx server error, "no quorum", anything ambiguous. When in doubt, use this. Lots of unknowns during faults is a run behaving normally, not dirty data.
To classify a reply, try asking yourself three questions, in order:
- Did you get an answer at all? If not -- timeout, dropped connection --
don't classify anything, just let your
execute()raise an unhandled underlying exception. hokea will catch it and recordunknownitself. - Did the server say "done"? Then
ok. - Did the server say "no" — and are you certain nothing changed? Both
parts, then
rejected. Any hesitation about the second part:unknown.
Why the caution concentrates on rejected: checks and exporters treat a
rejected operation as one that never happened — e.g., the Porcupine exporter
literally deletes them from the history it hands the checker. A wrong
unknown just makes conclusions more cautious; a wrong rejected
fabricates evidence, and every later conclusion built on it is confidently
wrong.
These mistakes can be subtle. During testing of hokea, I accidentally let hokea
classify a server's 503 Service Unavailable response (which reflects a
temporary server error) as a rejection: the server said it wasn't able to
handle the request, after all. But then the recorded runs said otherwise: a
write that had been rejected with the 503 showed up on all replicas because the
server that sent the 503 had applied it locally before noticing it did not have
the required quorum, and then a later resync applied the write elsewhere. If
your server can say "no" after doing something, its no is an unknown.
For HTTP-based protocols, http_result makes the safe choice the default:
from hokea import http_result
...
return http_result(r, rejected={400, 409})
2xx becomes ok; only the status codes you deliberately listed become
rejected; everything else — including the 503 you didn't think about —
becomes unknown. Build a Result by hand for non-HTTP protocols or
special cases.
Two more rules:
- Never catch transport errors (question 1 above). If your client library wraps transport failures in its own exception type, let that escape too. Any non-transport exception crashes the run with an error, which is better than a silent misclassification.
- A read that observes "not found" is
ok. The read happened and observed an absence.rejectedwould erase a legitimate observation.
Extras on Result:
response— the verbatim decoded reply. Never summarize it; checks and exporters need the whole thing.version— if your system returns a global version/revision token (like etcd's revision) pass it through; it enablescheck.session. Omit it otherwise.status— the protocol status code, for availability error mixes.
There are no retries: hokea attempts each operation exactly once and records it exactly once. If a client retried after a timeout, the history could no longer say whether an operation happened once or twice, and no check could reason about it. If your protocol needs retries, model each retry as its own operation in the generator.
The operation generator¶
def my_ops(client_id: int, rng: random.Random):
while True:
result = yield Op("reserve", seat, {"client_id": f"c{client_id}"})
if result.outcome == "ok":
yield Op("confirm", seat,
{"rid": result.response["reservation_id"]})
- It's a plain Python generator, one per client, run in that client's worker
process. hokea sends each op's
Resultback in, so flows that depend on a previous response (ids, tokens) read naturally. If you don't need the result, justyield Op(...). - Use only the passed-in
rngfor randomness — that's what makesseed=reproduce a workload shape. Op(op, key, args):keyis the logical object the operation touches (seat id, cart id, document key). Checks group by it; choose it carefully.time.sleep()inside the generator is fine and useful (think time — it also means operations are in flight when your faults hit).
Choreographed steps: the Script client¶
Some bugs need a specific sequence rather than random load: "cut the
network, then delete on the majority side, then write on the minority
side, then look." A random workload only produces such a sequence by
luck. For these, use a Script, a scripted client that hokea itself runs,
whose steps run inside the fault schedule:
script = Script(MyAdapter())
h = run_workload(cluster, my_ops, MyAdapter(), duration=30, script=script,
faults=[(10, lambda: net.partition([[1, 2], [3]])),
(0, lambda: script.do(Op("delete", "cart-7"), node=1)),
(0, lambda: script.do(Op("read", "cart-7"), node=3))])
Schedule entries run in list order, each no earlier than its offset, and
each only after the previous returned — net.partition() returns only once
the cut is verified, so an offset of 0 means "immediately after that",
with no guessing how long installation takes. script.do returns the live
Result (use ids from result.response in later steps), applies the same
outcome rules as worker clients, and is recorded in history.jsonl under
the reserved client_id 0, so checks and exporters see scripted ops like
any others. Script operations are not counted as load: report.json
excludes them.
Settling before judgment¶
Systems that sync in the background need some quiet time before their
nodes agree; that is normal, not a bug. run_workload(..., settle=5) waits five
quiet seconds after the clients finish before final states are captured,
and records the wait in the fault timeline. Without it, a perfectly healthy
eventually-consistent system can fail check.converged purely because the
snapshot came too early.
The state reader (optional, recommended)¶
def read_state(node) -> dict:
... # this node's view of the data, WITHOUT contacting its peers
Used at the end of a run to snapshot every node for check.converged and
check.acked_visible. What those checks consume is what each individual
server believes — divergence between servers is exactly what they detect.
Keys of the returned dict are the same key space your ops use.
If your servers answer client reads from their own local state (many designs do), point the state reader at the ordinary client API and you're done. That is the most trustworthy option, because it is the same interface real clients use, and there is no separate endpoint that could report something different from what clients see.
If a server interacts with its peers to return a result to a client
(quorum reads, a coordinator merging replica views), the client API will
not reflect the state held by the server you asked — which is important
for the convergence and durability checks, because the merged answer hides
exactly the per-server differences they exist to find, and during a
partition the coordinator may refuse to answer at all. In these cases,
your implementation should expose endpoints with identical semantics to
the client read API (e.g. GET /debug/cart/<id> mirroring
GET /cart/<id>) that answer only from state held on that node at the
moment of the call — never by contacting peers. Mirror the client API
one-for-one: same paths under a /debug prefix, same response shapes,
ideally the same serialization code with the peer coordination skipped —
that way the state reader is just your client reader with a different URL
prefix, and there is no second schema to invent or to drift. This may not
be necessary depending on the type of system you are building, but for any
system where answering a client request requires responses from multiple
server nodes, it is strongly recommended.
A debug state endpoint must meet three requirements, each of which you can test:
- It never contacts peers. It must answer, quickly, while fully partitioned. A test can call it during a verified partition, and if it stalls or errors there, it is contacting peers after all.
- It covers the same key space your operations use.
- It agrees with the client API when the system is healthy. After a heal and a settle, what clients read must match what the per-node states predict. Put that comparison in one test, so the endpoint cannot drift away from what clients see without you noticing.
Worked example¶
examples/toyserver/test_toy.py contains a complete adapter and operation
generator for the introductory lab's toy key-value server, which
the lab walks through, outcome classification included.
Start from it: swap in your system's endpoints and status codes, then go
through the three outcome rules above for every reply your server can
produce.