Skip to content

Checks

Functions that read only a finished run's files — no cluster needed, so any run can be re-examined later. Every verdict is three-way (violated / clean / insufficient evidence) and a violation carries the recorded operations that prove it. Before trusting a clean verdict, read what the checks can and cannot see.

Verdicts

hokea.check

Checks: functions that read only a run's files, answering one question each.

Every verdict is three-way — violated / clean / insufficient_evidence — and the third is reported explicitly: missing data must never look like a pass. A violation carries the exact recorded operations that prove it, so you can verify the conclusion by hand.

There is no linearizability checker here. Deciding linearizability requires searching over orderings of concurrent operations, which is easy to get subtly wrong (we will cover linearizability in lecture). If you claim linearizability, export the history to Porcupine instead (hokea export --format porcupine).

Verdict dataclass

Verdict(
    check: str,
    status: str,
    reason: str,
    evidence: list = list(),
    details: dict = dict(),
)

What every check returns: a three-way status (violated / clean / insufficient_evidence), a one-sentence reason, and the recorded operations that prove it in evidence.

check instance-attribute

check: str

status instance-attribute

status: str

reason instance-attribute

reason: str

evidence class-attribute instance-attribute

evidence: list = field(default_factory=list)

details class-attribute instance-attribute

details: dict = field(default_factory=dict)

InsufficientEvidence

Bases: Exception

Raised when a metric is read from data that cannot support it.

The five checks

converged

converged(states: dict) -> Verdict

Do all nodes report the same final state? states is {node_index: state} as captured by the run's state reader.

acked_visible

acked_visible(
    history: list,
    states: dict,
    *,
    write_ops: list[str],
    visible=None,
) -> Verdict

Is the effect of every acknowledged write present in the final state?

By default this checks existence: for each op in write_ops with outcome "ok", the op's key must appear in every node's final state. Use that for writes whose effect is "this thing now exists" (a confirmed sale, a created record). Existence is the safe default because nothing that happens later — except a delete — can legitimately erase it, so the check holds no matter how concurrent operations interleaved.

visible replaces the existence test with your own: a function taking (one node's state, one history record) and returning True if that write's effect is present there. Examples:

# the value this write set is what the node ended up with
visible=lambda state, op: state.get(op["key"]) == op["args"]["value"]
# an acknowledged delete stayed deleted
visible=lambda state, op: op["key"] not in state

WARNING — read before writing a value-level visible: a write's value only has to survive until some later write legitimately replaces it. Under concurrent random traffic you usually cannot know which write came last (two overlapping writes have no defined winner), so checking "my value is still there" against every acked write will flag correct systems. A value-level visible is only sound for writes you arranged to be last: issue them with a Script step after the main workload, give the system settle time, then check. For everything else, stick with the existence default — and pick write_ops so "acknowledged" implies "should still exist" (not holds that expire, not things later deleted).

session

session(history: list) -> Verdict

Within each client, server-supplied versions never go backwards — a plain comparison, catching a client reading older state than it already saw. Only for systems whose version token is global (like etcd's revision); meaningless for per-key versions.

event_join

event_join(
    events: list, *, require: str, for_each: str, on: str
) -> Verdict

Every event of type for_each has a matching event of type require with the same value of field on — e.g. every "ack" is eventually followed by a "commit" for the same op id.

availability

availability(history: list, faults: list) -> Availability

Slice the history by the fault timeline: descriptive numbers per fault window and per side, for tests to assert against.

Each fault produces a window starting when it was VERIFIED/done — operations during the multi-second install gap belong to neither the baseline nor the fault — and ending when the matching recovery was requested. Sides split operations by the node they targeted: majority/minority for a 2-group partition (group-1/group-2 when sizes tie), faulted-node/others for process faults.

Foreign chaos (experiments applied outside hokea, recorded by the k8s backend's watcher as chaos/chaos_done/chaos_recovered) windows the same way: open at chaos_done, closed by the first chaos_recovered for the SAME CR (matched on chaos_name), labeled chaos:<kind>:<name>, e.g. "chaos:stresschaos:my-stress". No node attribution — all operations fall on the single side "all".

Availability results

availability() returns an Availability; its windows and sides are what your test asserts against:

a = check.availability(h.history, h.faults)
side = a.window("partition").side("majority")
assert side.success_rate > 0.9

Availability

Availability(windows: list)

window

window(label: str) -> Window

Window dataclass

Window(
    label: str,
    start: float,
    end: float,
    sides: dict = dict(),
)

label instance-attribute

label: str

start instance-attribute

start: float

end instance-attribute

end: float

sides class-attribute instance-attribute

sides: dict = field(default_factory=dict)

side

side(name: str) -> SideStats

SideStats dataclass

SideStats(
    name: str,
    n: int,
    outcomes: dict,
    errors: dict,
    _ok_latency_ms: list,
    _end_latency_ms: list,
)

name instance-attribute

name: str

n instance-attribute

n: int

outcomes instance-attribute

outcomes: dict

errors instance-attribute

errors: dict

success_rate property

success_rate: float

latency_ms property

latency_ms: dict

answer_ms property

answer_ms: dict

Latency to any answer, including failures and timeouts — asserts like "failures are fast, not hangs" read this.