Testing your claims — from proposal to falsification experiments¶
Your project proposal makes four kinds of claims about the system you're going to build — safety, liveness, fault model, performance — and for each claim it describes an experiment that would falsify it: show it wrong if it is wrong. This page maps each claim type to the hokea machinery that runs that experiment. By the end, "can be falsified by partitioning two replicas and..." is a test file in your repo that either passes or hands you the evidence, rather than just a sentence in a document.
The worked examples below use the example scenario from the project spec — a replicated shopping cart where every reachable replica accepts adds, temporary divergence is fine, but a silently lost item is not. The hokea parts of each example are real, runnable code; the parts marked your adapter here are the ~30 lines you write for your own system — Adapters is the page for those.
Safety claims¶
A safety claim names something that must never happen. When it is violated, it is violated at a specific moment by a specific event, and a recorded run should let you find that event in a log. hokea's job is to record enough for that to be possible; these are the tools you use to find the event:
check.acked_visible— every effect your system acknowledged is still present later. Use it for "never silently lost" claims.check.converged— all replicas report the same state. For "replicas agree (after syncing)" claims.check.event_join— every event of one kind has its matching event of another kind (everyackhas acommit, every task has a completion), read from your servers' own logs.Scriptsteps — for claims about a specific point in time ("during the partition, a read on the other side must not see this"), choreograph exact operations at exact moments in the fault schedule instead of hoping random load hits the window.
Checks documents what each one can and — just as important — cannot see.
Worked example. The cart proposal's safety claim: concurrent adds from different sessions all survive being merged. Its falsification experiment, as a real hokea test: partition the replicas into two groups, add a different item to the same cart on each side, heal, give the system quiet time to merge, then check that the replicas agree and that both items survived:
from hokea import Op, Script, check
from my_cart import CartAdapter, cart_ops, read_cart_state # your adapter here — see docs/adapters.md
def test_concurrent_adds_survive_the_merge(cluster, net, run):
script = Script(CartAdapter()) # your adapter here
h = run(cart_ops, CartAdapter(), duration=30, script=script, settle=5,
state_reader=read_cart_state, # your adapter here
faults=[(5, lambda: net.partition([[1, 2], [3]])),
# offset 0: runs right after partition() returns, i.e. cut verified
(0, lambda: script.do(Op("add", "cart-9", {"item": "boots"}), node=1)),
(0, lambda: script.do(Op("add", "cart-9", {"item": "socks"}), node=3)),
(20, net.heal)])
v = check.converged(h.states)
assert v.status == "clean", v
in_cart = lambda state, op: op["args"]["item"] in state.get(op["key"], [])
v = check.acked_visible(h.history, h.states, write_ops=["add"], visible=in_cart)
assert v.status == "clean", v
Reading it: the schedule runs in order — partition (verified real before
anything proceeds), one scripted add on each side of the cut, heal at
t=20; settle=5 gives the merge five quiet seconds before states are
captured. If the merge drops an item, acked_visible comes back
violated naming the exact add and the node it's missing from — the
recorded evidence the falsification experiment promised. (in_cart
assumes the state reader
returns each cart as a list of items; adapt it to your state shape.) The
state_reader= is the third ~5-line piece of your adapter (see
Adapters): it fills
h.states with each node's final
state, and without it both state checks return insufficient_evidence —
even on a correct system, because there is nothing to judge.
Where this file lives, and how to run it
Put this in a test_claims.py next to your conftest.py — the fixture
from Data & durability or the
introductory lab, pointed at your server. Then run
uv run pytest -q test_claims.py on your laptop, or
uv run hokea test --namespace team-NN -q test_claims.py to ship the
same file to the class cluster and run it there (the lab's
Act 3 covers hokea test).
One design point from the Checks page: cart_ops in
this test should only add items, never remove them. An acknowledged add
is then an effect nothing can legitimately erase, which is exactly what
makes the visible= comparison sound — with no removes in play, "is the
item in the cart?" is itself an existence test, so acked_visible's
warning about value-level visible (which is about writes you must
arrange to be last) doesn't apply here. Mix removes into the same
workload and it does: the check can flag a correct system for an item a
client legitimately removed.
Liveness claims¶
A liveness claim names something that must eventually happen — requests get answered, replicas converge, jobs finish — including a statement of when your system is allowed to stop (refusing writes during a partition can be a legitimate design choice, but it must be a choice you wrote down, not a behavior you discovered).
In hokea, liveness claims become availability windows:
check.availability slices the recorded history by the fault timeline, so
"any reachable replica accepts writes" turns into success-rate asserts on
each side of a partition window, and "replicas converge after healing"
turns into settle= plus check.converged. The cart proposal's liveness
falsifier, same shape:
def test_reachable_replicas_accept_writes(cluster, net, run):
h = run(cart_ops, CartAdapter(), duration=30, settle=5, # your adapter here
state_reader=read_cart_state, # your adapter here
faults=[(8, lambda: net.partition([[1, 2], [3]])),
(22, net.heal)])
part = check.availability(h.history, h.faults).window("partition")
assert part.side("majority").success_rate > 0.95 # both sides keep
assert part.side("minority").success_rate > 0.95 # accepting writes
v = check.converged(h.states)
assert v.status == "clean", v # and re-merge after heal
If your design deliberately refuses minority-side writes (choosing consistency over availability, as a consensus-based design like Raft or Paxos must, since a side without a majority cannot accept writes; we will cover why in lecture), your falsifier asserts the opposite there: clean rejections, not successes. Your proposal should say so. Asserting high success on a side your design promises to refuse would fail a correct system; see the Checks page's notes on availability.
Fault-model claims¶
Your proposal must say which failures your claims hold under, and, just as importantly, which are out of scope. Pick from the faults hokea can actually inject, so that every in-scope fault has a falsifier that uses it:
cluster.kill(i)/stop(i)/start(i)— crash-stop: the node halts with no warning and no flush. A restart brings back a new node with the old name — empty memory, empty/data. If your design assumes storage that survives a crash, read Data & durability before writing this section.net.partition([[...], [...]])/net.isolate(i)— cut the cluster network between groups, or cut one node off from all its peers. Verified real before your test proceeds.net.slow(i, delay_ms=..., jitter_ms=..., loss_pct=...)— a slow or lossy network path: delay, jitter (random variation in that delay), and packet loss on one node's traffic.cluster.throttle(i, cpus=...)— a compute straggler: the node talks fine but computes slowly (wherenet.slowmakes it slow to talk).cluster.pause(i)/unpause(i)— Docker backend only: freeze the process and resume it later with memory intact — a very long garbage-collection pause, not a crash.chaos.apply("your.yaml")— on the class cluster, anything Chaos Mesh can inject: CPU/memory stress, disk I/O delays, clock skew, HTTP tampering, multi-step Workflows. The lab's Act 4 shows the pattern.
Whichever you use, the run's record (faults.jsonl) captures all of it
with timestamps — including chaos experiments injected outside your test —
so the claim "holds under X" is always checked against evidence that X
actually happened. One scope note to copy into your proposal if it applies:
a malicious replica, one that lies rather than fails (called a Byzantine
fault; we will cover this in lecture), is out of scope for the course tooling. hokea's
built-in faults inject crashes, cuts, and slowness, never corruption;
Chaos Mesh's HTTP tampering (reachable through chaos.apply) can
corrupt traffic in flight, but even that is a hostile network, not a
replica that lies about its own state.
Performance claims¶
The proposal doesn't ask for numbers yet — it asks you to name the metrics, define the benchmarks, and say how you'll generate the load. hokea maps onto those directly:
- Metrics: every run writes
report.json— throughput, latency percentiles (p50/p99), outcome and error mixes (Artifacts has the format; the lab's measurement interlude defines the terms). Naming your metrics mostly means picking which of these you'll report, plus any system-specific ones your nodes emit as events. - Benchmarks: the lab's
measurement interlude is the template — separate
read-heavy and write-heavy workloads as their own test files, and
HOKEA_NODES=6to re-run the same benchmark at a different fleet size and see which numbers scale (a conftest convention, not built into hokea — your conftest must read the variable; the sample in Data & durability does). - Load: the same
run(...)that drives fault tests is the load generator —clients=200, rate=2000runs two hundred concurrent simulated clients at a target operation rate. - Under-fault vs. baseline: performance claims like "convergence time
after a partition heals" or "write latency during a partition versus
normal operation" fall out of availability windows — compare
window("partition")againstwindow("baseline")in one recorded run, as the lab's CPU-stress test does with p99 latency.
Your claims will get sharper — keep the falsifiers¶
Your proposal will change over the semester, and these tests should change with it: each falsifier is the precise version of one claim. Keep them in your repo next to your code, and run them, in CI (automated tests on every push) if your team has it set up, or at least by hand before every milestone. When a claim changes ("we now refuse minority writes"), the falsifier changes in the same commit, and the history of your test files over the semester is the record of your claims getting sharper, which is what I will be evaluating.