Checks¶
Functions that read only a finished run's files — no cluster needed, so any run can be re-examined later. Every verdict is three-way (violated / clean / insufficient evidence) and a violation carries the recorded operations that prove it. Before trusting a clean verdict, read what the checks can and cannot see.
Verdicts¶
hokea.check
¶
Checks: functions that read only a run's files, answering one question each.
Every verdict is three-way — violated / clean / insufficient_evidence — and the third is reported explicitly: missing data must never look like a pass. A violation carries the exact recorded operations that prove it, so you can verify the conclusion by hand.
There is no linearizability checker here. Deciding linearizability requires
searching over orderings of concurrent operations, which is easy to get
subtly wrong (we will cover linearizability in lecture). If you claim
linearizability, export the history to Porcupine instead
(hokea export --format porcupine).
Verdict
dataclass
¶
Verdict(
check: str,
status: str,
reason: str,
evidence: list = list(),
details: dict = dict(),
)
What every check returns: a three-way status (violated / clean /
insufficient_evidence), a one-sentence reason, and the recorded
operations that prove it in evidence.
InsufficientEvidence
¶
Bases: Exception
Raised when a metric is read from data that cannot support it.
The five checks¶
converged
¶
converged(states: dict) -> Verdict
Do all nodes report the same final state? states is {node_index: state} as captured by the run's state reader.
acked_visible
¶
acked_visible(
history: list,
states: dict,
*,
write_ops: list[str],
visible=None,
) -> Verdict
Is the effect of every acknowledged write present in the final state?
By default this checks existence: for each op in write_ops with
outcome "ok", the op's key must appear in every node's final state. Use
that for writes whose effect is "this thing now exists" (a confirmed
sale, a created record). Existence is the safe default because nothing
that happens later — except a delete — can legitimately erase it, so
the check holds no matter how concurrent operations interleaved.
visible replaces the existence test with your own: a function taking
(one node's state, one history record) and returning True if that
write's effect is present there. Examples:
# the value this write set is what the node ended up with
visible=lambda state, op: state.get(op["key"]) == op["args"]["value"]
# an acknowledged delete stayed deleted
visible=lambda state, op: op["key"] not in state
WARNING — read before writing a value-level visible: a write's value
only has to survive until some later write legitimately replaces it.
Under concurrent random traffic you usually cannot know which write came
last (two overlapping writes have no defined winner), so checking "my
value is still there" against every acked write will flag correct
systems. A value-level visible is only sound for writes you arranged
to be last: issue them with a Script step after the main workload, give
the system settle time, then check. For everything else, stick with the
existence default — and pick write_ops so "acknowledged" implies
"should still exist" (not holds that expire, not things later deleted).
session
¶
session(history: list) -> Verdict
Within each client, server-supplied versions never go backwards — a plain comparison, catching a client reading older state than it already saw. Only for systems whose version token is global (like etcd's revision); meaningless for per-key versions.
event_join
¶
event_join(
events: list, *, require: str, for_each: str, on: str
) -> Verdict
Every event of type for_each has a matching event of type require
with the same value of field on — e.g. every "ack" is eventually
followed by a "commit" for the same op id.
availability
¶
availability(history: list, faults: list) -> Availability
Slice the history by the fault timeline: descriptive numbers per fault window and per side, for tests to assert against.
Each fault produces a window starting when it was VERIFIED/done — operations during the multi-second install gap belong to neither the baseline nor the fault — and ending when the matching recovery was requested. Sides split operations by the node they targeted: majority/minority for a 2-group partition (group-1/group-2 when sizes tie), faulted-node/others for process faults.
Foreign chaos (experiments applied outside hokea, recorded by the k8s
backend's watcher as chaos/chaos_done/chaos_recovered) windows the same
way: open at chaos_done, closed by the first chaos_recovered for the
SAME CR (matched on chaos_name), labeled chaos:<kind>:<name>, e.g.
"chaos:stresschaos:my-stress". No node attribution — all operations
fall on the single side "all".
Availability results¶
availability() returns an Availability; its windows and sides are what
your test asserts against:
a = check.availability(h.history, h.faults)
side = a.window("partition").side("majority")
assert side.success_rate > 0.9
Window
dataclass
¶
Window(
label: str,
start: float,
end: float,
sides: dict = dict(),
)