Introductory lab — run it, break it, ship it, break it in prod¶
This is a lab to introduce hokea, the distributed systems orchestration framework we will use, in four acts (plus a short measurement interlude). The times are generous — budget up to about two hours in all:
- Run it locally (~15 minutes) — a 3-node toy service in containers on
your laptop, tests passing, evidence recorded.
- Interlude — measure it (~10 minutes): put the service under load, read its throughput and latency, double the fleet to six nodes, and see which numbers improve — and which don't.
- Break it locally (~15 minutes) — kill a node and cut the network while clients are hammering it, and read what hokea caught.
- Ship it to the class cluster (~40 minutes) — one command,
hokea test, runs the same tests on real machines, in your team's own space on the cluster; then measure it at 3 and 6 nodes and turn the numbers into one graph. - Break it in prod (~15 minutes) — the same faults, now injected by Chaos Mesh on the class cluster.
Before any of that comes Part 0 — Setup (~20 minutes), the first thing you'll do in the lab: install the tools on your own laptop and download your cluster credentials. The Docker Desktop download is large and can be slow on lab WiFi, so start that download first, then read on while it comes in.
The system you'll run and break in every act is a key-value store: a service other programs talk to over the network to store and fetch values by name — put a value under a key, get it back later, like a Python dictionary served to many clients at once. Many real systems you've heard of are distributed key-value stores — Redis, etcd, MongoDB, DynamoDB — exposing similar interfaces while making different tradeoffs in consistency and in how they handle machine and network failures. The toy server in this lab is a bare-bones one, small enough to read top to bottom.
Each part ends at a checkpoint — a concrete thing you can show. Nothing is handed in; this is an in-class exercise. Acts 1 and 2 need only your laptop; Acts 3 and 4 need the cluster access you set up in Part 0.
Part 0 — Setup¶
On your own laptop: install five tools (you may already have some), check each one works, and download your cluster credentials. Budget about 20 minutes — more if you're installing Docker for the first time. If you haven't already, start the Docker Desktop download (0.1 below) right now: it is the biggest download and can be slow on lab WiFi, and the rest of Part 0 can happen while it comes in. Part 0 ends with a checklist; every line of it must pass before you move on to Act 1.
Every command in this document assumes a Unix shell: macOS, Linux, or — on Windows — WSL (the Windows Subsystem for Linux, which gives you a real Linux environment inside Windows). A plain Windows Command Prompt or PowerShell will not work for this course.
0.1 Docker¶
Docker runs programs inside containers — a container is a lightweight box with its own filesystem and network, like a very cheap virtual machine. The lab runs a three-node distributed system in containers on your laptop.
Install Docker Desktop for your operating system (official pages):
- macOS: https://docs.docker.com/desktop/setup/install/mac-install/ (pick Apple silicon or Intel to match your Mac)
- Windows: https://docs.docker.com/desktop/setup/install/windows-install/
— during setup, keep the WSL 2 option enabled (it's the default);
that's what lets your WSL shell talk to Docker. If you don't have WSL
yet, install it first: open PowerShell and run
wsl --install, then reboot. - Linux: https://docs.docker.com/engine/install/ for your distribution (Docker Engine is enough; Docker Desktop is optional).
Verify — with Docker Desktop running (start it once; it can start on login afterwards), in your terminal:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
A table — even an empty one — starting with CONTAINER ID means you're
done.
Troubleshooting
command not found— Docker isn't installed (or, on Windows, you're not in a WSL terminal).Cannot connect to the Docker daemon— Docker is installed but not running: start Docker Desktop (or the Docker service) and retry.permission deniedaboutdocker.sock(Linux) — add yourself to the docker group:sudo usermod -aG docker $USER, then log out and back in, and retry.
0.2 Python¶
You need Python 3.11 or newer. Verify:
python3 --version
Python 3.12.3
(Anything from 3.11 up is fine.)
0.3 uv¶
uv is a fast, modern Python package manager. If you know pip: uv replaces that workflow — it creates isolated per-project environments and pins exact dependency versions, so a project behaves the same on every teammate's machine. You're strongly encouraged to use it in this course if you work in Python. Install it, then restart your terminal:
curl -LsSf https://astral.sh/uv/install.sh | sh
Verify:
uv --version
uv 0.11.9 (x86_64-unknown-linux-gnu)
(Any version number is fine.) If it prints command not found, you
skipped the "restart your terminal" step.
0.4 git¶
You probably have it already. Verify:
git --version
git version 2.43.0
(Any version is fine.) If not: macOS offers to install it the first time
you run the command; on Linux, sudo apt install git (or your
distribution's equivalent); on Windows, install it inside WSL the Linux
way.
0.5 kubectl¶
kubectl is the Kubernetes command-line tool — Kubernetes is the system
the class cluster uses to run programs across its machines, and the
second half of the lab talks to it.
Install kubectl on your laptop by following
https://kubernetes.io/docs/tasks/tools/ for your operating system (on
Windows, install the Linux version inside WSL). That page's main
install command uses sudo; if you can't use sudo on your laptop,
follow the page's note on installing without root into a folder on your
PATH (such as ~/.local/bin) instead.
Verify:
kubectl version --client
Client Version: v1.35.8
Kustomize Version: v5.7.1
(Any versions are fine. kubectl doesn't need the cluster to exist yet —
--client only checks the tool itself.)
0.6 Cluster access¶
Two things connect you to the class cluster: your team namespace name and a kubeconfig file you download yourself. The cluster is reachable from any ordinary internet connection — campus WiFi, home, a phone hotspot.
- Your team namespace name — a namespace is your team's own named
compartment on the cluster, and you'll type its name in several
commands below. We write
team-NNfor it throughout this document; your team's real name is zero-padded, e.g.team-03, andteam-3will fail withForbiddenorNotFound. Your instructor announces team assignments; if you don't know your team number, ask. -
Your kubeconfig (the credentials file the cluster tools read). Download it yourself — about two minutes:
- In a browser, go to https://launch.cs.vt.edu and sign in with your CS department account (the same login you use for CS department machines — not your main VT PID).
- Find the cluster named hokea — click its name to open it, then
click the Download KubeConfig button (a download icon near the
top right; hovering over the icons shows their names). A file named
hokea.yamllands in your downloads. -
Put it where kubectl looks. kubectl only reads
~/.kube/config, and the file must be named exactlyconfig— so rename it as you copy:mkdir -p ~/.kube && cp ~/Downloads/hokea.yaml ~/.kube/config(On Windows, your browser saves into the Windows Downloads folder, which your WSL shell sees at a different path: run
ls /mnt/c/Usersto see the folder names, pick yours, and usecp "/mnt/c/Users/YOUR-WINDOWS-NAME/Downloads/hokea.yaml" ~/.kube/config— keep the quotes; they matter if the folder name has a space. And on any machine: if a~/.kube/configalready exists from another course or job, this overwrites it — save the old one first withmv ~/.kube/config ~/.kube/config.backup.)
If you have both, verify (with your own team's namespace) — the command asks the cluster to list your team's pods (running programs on the cluster; you haven't started any yet, which is the point):
kubectl -n team-NN get pods
No resources found in team-NN namespace.
That empty answer is success.
Troubleshooting
- Signing in to the website fails — that's your CS department account, not this course. If you don't have one or forgot the password, tell your instructor, who will point you to CS tech staff.
- Signed in fine, but no cluster named
hokeaappears — you haven't been added to your team on the cluster yet. Tell your instructor; once you're added, everything above works. command not found— kubectl isn't installed (0.5 above).- The verify command answers
ForbiddenorNotFound— you typedteam-NNliterally, or a namespace that isn't yours (remember the zero-padding:team-03, notteam-3). If the name is right, you may not be added to your team yet — tell your instructor. The connection to the server localhost:8080 was refused— kubectl can't find your credentials file: step 3 above didn't happen, or the copy grabbed the wrong file. Redo step 3, then retry.Unable to connector a long hang — check your internet connection. If it persists on a connection that otherwise works, re-download the kubeconfig (steps 1–3) — the file may be stale or damaged.
0.7 The checklist — before moving on to Act 1¶
Before moving on to Act 1, confirm all of these. Run each; if any fails, its fix is in the section above.
docker ps
python3 --version
uv --version
git --version
kubectl version --client
kubectl -n team-NN get pods
-
docker psprints a table (even an empty one) -
python3 --versionprints 3.11 or newer -
uv --versionprints a version -
git --versionprints a version -
kubectl version --clientprints a version - team namespace known; kubeconfig at
~/.kube/config;kubectl -n team-NN get podsanswersNo resources found
Checkpoint — all six lines pass (or: five pass, a TA is flagged about the cluster one, and you start Acts 1–2 while it's sorted out — they don't need the cluster).
Act 1 — Run it locally¶
When this act is done you will have a 3-node replicated key-value store running in containers on your laptop, three tests passing against it, and a directory of recorded evidence you can open and read. The store itself is the toy key-value server from the intro — a dictionary served over the network, replicated across three nodes.
1.1 A clean project directory¶
Pick a folder you'll keep your course work in (say cd ~/cs4284), clone
the course repository there, then make a fresh directory next to it and
set it up as a uv project:
git clone https://github.com/spin-vt/hokea
mkdir toy-demo && cd toy-demo
uv init --bare
uv add --editable ../hokea
uv add --dev pytest
What each uv command did. uv init --bare created one file,
pyproject.toml — the standard file describing a Python project and what
it depends on. uv add records a dependency in that file and installs it
into the project's own private environment (uv creates the environment
automatically the first time; nothing you install here touches the rest of
your machine). --editable ../hokea means hokea is installed from your
clone, live: when course fixes land, a git pull inside hokea/ updates
hokea for you — no reinstall. And --dev pytest adds pytest as a
development tool — something you test with, not part of the project
itself.
The two uv add lines end with the list of what was installed:
Installed 7 packages in 14ms
+ certifi==2026.7.22
...
+ hokea==0.1.0 (from file:///home/you/hokea)
...
Installed 5 packages in 42ms
...
+ pytest==9.1.1
(The rest of this document assumes toy-demo sits next to your hokea
clone.
If you cloned it somewhere else, use your path in the uv add --editable
line instead — the path is recorded in pyproject.toml, so if you move
things around later, re-run uv add --editable <new-path> to update it.)
From here on, commands that run tests start with uv run, which runs a
command inside this project's private environment, finding it
automatically — nothing to activate, in any terminal; just be in the
toy-demo directory.
1.2 Copy the toy project — three files¶
cp ../hokea/examples/toyserver/server.py .
cp ../hokea/examples/toyserver/conftest.py .
cp ../hokea/examples/toyserver/test_toy.py .
What each file is:
server.py— the system under test: a ~100-line replicated key-value store. You will run multiple instances of the server, one per node. Each node keeps a Python dict in memory and forwards every write to its peers. It follows the same two-part contract your course project will follow: it reads the environment variablesNODE_ID/PEERS(an environment variable is a named value the operating system hands a program as it starts — Python reads them withos.environ; hokea also providesN_NODES) to learn who it is and who its peers are, and it answersGET /healthwith HTTP 200 (the web's "success" status code) once it's ready.conftest.py— one pytest fixture (a named piece of setup that pytest runs for you and hands to any test that asks for it; pytest automatically reads any file namedconftest.pyin the test directory) telling hokea how to run the system: which ready-made base image to build on (an image is a frozen snapshot of a filesystem plus a program — the template containers are started from), which source directory, what command starts a node, how many nodes, which port. That's the entire deployment description — no Dockerfile, no compose file.test_toy.py— a small adapter (about 15 lines teaching hokea to speak the server's HTTP protocol and classify every reply as ok / rejected / unknown), an operation generator (what the simulated clients do: write keys, read them back), and three tests. The adapter and the operation generator are exactly what you'll write for your own project — about 30 lines, total.
1.3 Run it¶
uv run pytest -q
The first run downloads the Python base image and builds your container image from it, so it takes a few minutes. Every run after that takes about a minute and a half. Expected:
... [100%]
3 passed in 80.63s (0:01:20)
3 passed is the result this document promises — that's the lab's first checkpoint. If Docker isn't running you'll see errors about failing to connect to Docker — start it and re-run. A run much slower than the times above — the first run's image download, or a laptop on battery-saver; the exact times don't matter, passing does. Anything else — re-run once, and if it repeats, flag a TA with the full output.
1.4 What just happened¶
Each test brought up three containers — hokea-toydemo-node1 through
node3, one per node of the toy server — connected by two private Docker
networks: a cluster network the nodes use to talk to each other (this is
where faults will be injected), and a client network the test's own
traffic uses (this one is never faulted, so hokea can always tell the
difference between "the system is down" and "I broke my own view of it").
Then hokea started several concurrent client workers, each running your
operation generator against the nodes, exactly like real users hitting a
real service.
While all that ran, hokea wrote everything down: every client operation with the server's verbatim reply, every fault with timestamps, every JSON line the servers printed, and a final snapshot of each node's state. The tests' verdicts are computed from those files, not from anything live — which means you can re-read the evidence yourself, right now.
The three tests you just ran, by name:
test_other_nodes_stay_available_when_one_diestest_every_received_write_was_appliedtest_replication_is_not_durable_and_hokea_catches_it
The names hint that more went on than a smoke test — nodes were killed mid-run, and that third test passed because hokea caught something. Act 2 walks through each of them slowly. For now the takeaway is simpler: it's green, and everything that happened is written down in files you can read.
1.5 Look inside a run directory¶
Every test run left a directory under runs/:
ls runs/
20260815-185656-test_other_nodes_stay_available_when_one_dies
20260815-185726-test_every_received_write_was_applied
20260815-185748-test_replication_is_not_durable_and_hokea_catches_it
One reassurance while you're looking around: hokea removes its containers
when a run ends, so docker ps shows an empty table again after every
test — the runs leave nothing behind on your laptop except the runs/
folders themselves.
Open the last one. A word on picking "the last one": run directories
are named by their start time, but later in this lab the runs that come
back from the class cluster are stamped in UTC while local runs use your
laptop's clock — so name order isn't always time order. Sort by the time
the directory was written instead (ls -td = directories, newest first)
and keep the winner's name in a shell variable for the next commands
(if you open a new terminal later, run the RUN= line again):
RUN=$(ls -td runs/*catches_it* | head -1)
ls "$RUN"
events.jsonl faults.jsonl history.jsonl metadata.json report.json shards states
What's in each, and why you care:
history.jsonl— one line per client operation: which node, what was asked, the verbatim reply, when it started and returned, and the outcome (ok/rejected/unknown). This is the ground truth every check reads.faults.jsonl— the fault timeline: every kill, restart, and partition, with a timestamp for when it was requested and another for when it was actually done. Lining this file up againsthistory.jsonlis how a check knows which operations happened "during the fault".events.jsonl— every JSON line your servers printed to stdout (their normal console output), with which node printed it and when. Your system's own testimony about what it did internally.states/— one JSON file per node: a snapshot of that node's store at the end of the run. This is how hokea proves a write is gone — it's acknowledged inhistory.jsonlbut missing from a node's state file.metadata.jsonandreport.json(the run's configuration and a summary of how fast the run went — the Interlude after this act is all about reading those numbers) andshards/(raw per-client files that were merged intohistory.jsonl) — you can ignore these for now.
Look at one line of real evidence:
head -1 "$RUN/history.jsonl"
{"client_id": 1, "seq": 0, "node": 1, "op": "put", "key": "c1-0", "args": {"value": "0"}, "invoke_ts": 0.004848, "return_ts": 0.013239, "outcome": "ok", "status": 200, "response": {"ok": true}, "version": null, "error": null}
Every verdict in this course is reproducible from files like this one. You never have to take a test's word for what happened: the evidence is on disk, and you can re-read it yourself.
1.6 About the toy server: incomplete on purpose¶
The toy server is deliberately incomplete. It is correct when nothing goes wrong: writes land, replication works, reads return what was written. But it has no retries (a replication attempt to a dead peer is simply dropped), no re-sync (a node that restarts comes back empty and nobody catches it up), and everything lives in memory. Nobody planted a bug in it — these are the gaps that every distributed system has until someone deliberately closes them.
In your course project, your task is to build a system where the same types of faults that break the toy server (as well as others) don't break your own.
Interlude — measure it¶
Act 1 asked "is it correct?". Before we start breaking things, a ten-minute detour into the other question every distributed system exists to answer: how much load can it take — and what happens when you add machines? You'll put the toy store under load, read two numbers off the result, then double the fleet from three nodes to six and see which number improves — and which doesn't.
The two numbers, defined once (the rest of this document uses them too):
- Throughput — how many operations the system completes per second, counted across all clients together. Bigger is better.
- p50 / p99 latency — sort every response time from fastest to slowest; we'll track both the p50 (median) and p99 (slowest 1% of requests) latencies. Ask yourself: why would we track both, and why these percentiles? What do you think these represent? We'll discuss this more in a few weeks!
Tiny nodes on purpose¶
The conftest.py you copied in Act 1 gives every node half a CPU and
256 MB of memory — deliberately puny. A production service saturates at
tens of thousands of requests per second, which you can't generate (or
afford) from a laptop. So instead of scaling the load up, we scale the
servers down: on half-CPU nodes, thirty simulated clients are enough to
drive the system to its limit, and you get to watch production behavior —
saturation, queueing, latency spikes, capacity that grows (or doesn't)
with the fleet — at toy scale. The behavior is the same as at production
size; only the numbers are smaller. (The
server also has a WORK knob — a simulated per-request cost, set in
conftest.py — standing in for the parsing, validation, and disk work a
real request handler does; see the docstrings in both files.)
The fourth file¶
cp ../hokea/examples/toyserver/test_measure.py .
Inside are two "experiments" written as pytest tests. Each drives 30 concurrent simulated clients for 20 seconds — no faults — and prints the throughput and latency that the run recorded. The asserts in them are deliberately loose (the run happened, operations succeeded): with measurements, you read the numbers, the asserts just keep the file runnable as a test.
test_read_throughput— writes a small set of keys once, waits for them to replicate everywhere, then every client hammersGETs across all nodes, round-robin (taking the nodes in turn).test_mixed_throughput— the same write+read mix Act 1's tests used: every client writes its own keys and reads them back.
Experiment 1 — reads, then more machines¶
Run the read experiment (-k read_throughput selects that one test by
name; -s tells pytest not to swallow the printed summary):
uv run pytest -q test_measure.py -k read_throughput -s
---- reads only: 3 nodes, 30 clients, 20s ----
throughput 240 ops/s
latency p50=11.75ms p99=412.57ms
outcomes {'ok': 4809}
.
1 passed, 1 deselected in 34.54s
Your numbers will differ — they depend on your machine — but the shape is
what matters. Now the fun part. The node count comes from an environment
variable, HOKEA_NODES (default 3). Writing it in front of a command sets
it for that one command only — that's deliberate, so plain
uv run pytest -q keeps meaning "3 nodes"; don't export it. Doubling
the fleet therefore needs no editing (and set it only for
test_measure.py: the fault tests in this lab assume 3 nodes):
HOKEA_NODES=6 uv run pytest -q test_measure.py -k read_throughput -s
---- reads only: 6 nodes, 30 clients, 20s ----
throughput 476 ops/s
latency p50=11.23ms p99=347.12ms
outcomes {'ok': 9512}
.
1 passed, 1 deselected in 37.41s
Read throughput roughly doubled. This is the promise that makes people build distributed systems in the first place — reads can be served by any node that has a copy of the data, so six half-CPU nodes really do have twice the read capacity of three. This is what "scaling out" means: add machines, serve more users.
Experiment 2 — writes don't scale the same way¶
Same doubling, this time with the write+read mix (half of every client's operations are writes):
uv run pytest -q test_measure.py -k mixed -s
HOKEA_NODES=6 uv run pytest -q test_measure.py -k mixed -s
---- writes+reads: 3 nodes, 30 clients, 20s ----
throughput 80 ops/s
latency p50=20.2ms p99=2090.65ms
outcomes {'ok': 1602}
.
1 passed, 1 deselected in 34.10s
---- writes+reads: 6 nodes, 30 clients, 20s ----
throughput 70 ops/s
latency p50=8.92ms p99=4269.11ms
outcomes {'ok': 1386, 'unknown': 6}
.
1 passed, 1 deselected in 37.31s
Twice the machines — and throughput went down, while the p99 more than
doubled (those unknowns are requests that timed out entirely). The p50
actually improved: with six nodes each one has fewer clients queued on
it, so a typical request waits less — even as the tail got worse and
total throughput fell. On your machine the mixed numbers may dip less, or
just sit flat; what they will not do is double the way the reads did.
The reason is arithmetic. The toy server replicates every write to
every other node: with 3 nodes, one client write costs the fleet 3
requests' worth of work (the write plus 2 replica copies); with 6 nodes
it costs 6. Adding servers adds read capacity, but it adds write work
at exactly the same rate — so the capacity you added gets spent copying
writes around, and the extra replication traffic is also what drives the
p99 up. This is a real tradeoff, not a toy quirk: full replication buys
read scaling by making every write more expensive. Sharding, quorums,
and primary/backup schemes exist to improve exactly this tradeoff — and
you now have the instrument to measure whether your project's design
does.
Act 2 — Break it locally¶
When this act is done you will have understood exactly what Act 1's three green tests proved — including the one that passed by catching real data loss — watched hokea cut the network under live load, and written your own fault schedule.
2.1 The kill test, slowly¶
Act 1's first and third tests are two questions about the same fault — node 2 dies at second 4 and comes back at second 11, while three clients write and read the whole time:
- Does everyone else keep working?
check.availabilityslices the operation history by the fault timeline: before the kill, all three nodes' clients succeed; during it, node 1's and node 3's clients keep succeeding while node 2's fail (correctly — their node is dead). That check comes back clean: the property holds. - Did we lose anything? While node 2 was dead, clients of nodes 1 and 3
kept writing, and the server acknowledged those writes — it said "ok,
saved". Node 2 never got them: replication to a dead peer is dropped, and
nothing re-syncs a returning node.
check.acked_visiblecompares every acknowledged write against the final state snapshot of every node, and returns violated — along with the exact operations that are missing and which node they're missing from. The test then asserts the violation: it passes because hokea caught the loss and produced the evidence.
Act 1's second test, test_every_received_write_was_applied, runs with
no faults at all. The
server prints a recv event and an applied event for every write it
handles (that's the events.jsonl file from Act 1), and check.event_join
confirms every recv has its matching applied — the system's own
testimony about its internals, audited line by line.
For client-server projects like this one, that is the pattern: the checks compare what the system acknowledged with what actually happened to the data. Batch-style projects (a MapReduce, distributed training) audit the same kind of promises through the events their nodes emit instead of client histories — hokea records both.
2.2 Add the partition test¶
A network partition is when machines are all still running but some can no longer talk to each other — think a cut network cable, though a variety of faults can cause this situation. The fifth and last file you'll copy tests how the toy server handles one. Copy it and run everything again:
cp ../hokea/examples/toyserver/test_partition.py .
uv run pytest -q
...... [100%]
6 passed in 231.44s (0:03:51)
Checkpoint — 6 passed.
(Six now: Act 1's three tests, the Interlude's two measurements, and the
new partition test. The measurement tests print nothing without -s, but
they still run — that's the extra minute.)
2.3 What the partition test proves¶
This test splits the cluster network into {node 1, node 2} on one side
and {node 3} on the other, from second 8 to second 50 of a 60-second
run. (The run is longer than the kill test's because on the class
cluster proving that a cut is really in place can take tens of seconds,
and the cut has to be in place while the clients are still working.)
Three things, in order:
- The cut is real. hokea injects the fault and proves the cluster is actually partitioned before the test proceeds — there are subtle ways a cut can silently fail to happen, and hokea manages those for you, erroring out rather than letting a test draw conclusions from a fake fault. The test adds its own demonstration: a key written to node 1 during the cut still isn't readable from node 3 seconds later (a 404 — without a cut, replication lands in milliseconds).
- Everyone still serves their own clients. Client traffic rides the
never-faulted client network, and the toy server acknowledges writes
locally, so during the cut every node — both sides — keeps answering
its own clients.
check.availabilityshows success rates above 90% on the majority side and the minority side. - The damage is permanent. Writes acknowledged on one side during the
cut never crossed it, and the toy never retries or re-syncs — so even
after the network heals, they're still missing from the other side.
check.acked_visiblereturns violated with the lost operations, exactly like the kill test.
Don't take the test's word for it; read the evidence yourself. Inside
the run's directory, history.jsonl is the ordered log of every client
operation with the server's verbatim reply, and states/node-<i>.json
is each node's entire store at the end of the run. The key the test
writes during the cut is named cut-probe:
RUN=$(ls -td runs/*partition* | head -1)
grep cut-probe "$RUN/history.jsonl"
{"client_id": 0, "seq": 0, "node": 1, "op": "put", "key": "cut-probe", "args": {"value": "x"}, "invoke_ts": 13.706348, "return_ts": 13.775434, "outcome": "ok", "status": 200, "response": {"ok": true}, "version": null, "error": null}
{"client_id": 0, "seq": 1, "node": 3, "op": "get", "key": "cut-probe", "args": {}, "invoke_ts": 16.775656, "return_ts": 16.783771, "outcome": "ok", "status": 404, "response": {"error": "not_found", "key": "cut-probe"}, "version": null, "error": null}
Line 1: node 1 said ok — saved ("outcome": "ok", HTTP 200). Line
2: three seconds later, node 3 has never heard of the key (404) —
without a cut, replication lands in milliseconds. Now check the final
states, captured after the network healed:
grep -c cut-probe "$RUN/states/node-1.json" "$RUN/states/node-3.json"
runs/20260829-102147-test_partition_splits_replication_but_not_clients/states/node-1.json:1
runs/20260829-102147-test_partition_splits_replication_but_not_clients/states/node-3.json:0
1 and 0: node 1 still has the write it acknowledged; node 3 never
got it — and never will, because nothing in the toy re-syncs. An
acknowledged write, permanently missing from a replica: that's a
violation, caught and pinned to evidence.
Checkpoint — paste the "outcome": "ok" put line and the 1/0
state counts. Together they are the violation: the server's promise,
and the node that never received it.
Troubleshooting
ls: cannot access 'runs/*partition*': No such file or directory(and then agrepcomplaint) — there is no partition run yet: the test didn't pass (scroll up for pytest's error) or you're not intoy-demo.- You ran the test more than once — that's fine; the
RUN=line picks the most recently written run.
2.4 Write your own fault schedule¶
Every test you've run so far asked for three helpers by naming them as
arguments (that's the fixture mechanism from conftest.py again). One
sentence each: cluster controls the nodes as processes — cluster.kill(2),
cluster.start(2); net controls the network between them —
net.partition(...), net.heal; and run(...) drives a recorded client
workload against the nodes and hands back the recorded files.
A fault schedule is just a list of (seconds_after_start, action) pairs
passed to run(...), and you can mix process and network faults freely.
Here is a complete test — try writing one like it in a new file (say
test_mine.py) next to the others:
from test_toy import ToyAdapter, toy_ops
from hokea import check
def test_my_first_fault_schedule(cluster, net, run):
h = run(toy_ops, ToyAdapter(), duration=30, clients=3,
faults=[(5, lambda: cluster.kill(2)), # t=5s: node 2 dies
(10, lambda: cluster.start(2)), # t=10s: it returns (empty!)
(15, lambda: net.partition([[1], [2, 3]])), # t=15s: cut node 1 off
(25, net.heal)]) # t=25s: reconnect everyone
a = check.availability(h.history, h.faults)
assert a.window("baseline").side("all").success_rate > 0.9
Change the schedule and the assertions to the question you want
answered, and run uv run pytest -q again. The times are "earliest
start": each
action fires once its offset has passed and the previous action finished.
Everything still lands in faults.jsonl with requested/done timestamps.
From here on, your counts will be one higher than the ones I show —
7 passed where I show 6 passed, one more file shipped — because of
test_mine.py.
2.5 A check answers one question¶
Look at what happened inside the same partition run:
check.availability came back clean (every node kept serving) while
check.acked_visible came back violated (acked writes were lost).
A check isn't a thumbs up or down on your system, it's an
answer to one specific question about one recorded run. The toy server
really is available under partition, and really does lose data — both are
true, and the checks say exactly that, each carrying its evidence. (There's
a third answer a check can give — insufficient evidence — when the run
didn't record enough to decide; missing data never silently counts as a
pass.)
Act 3 — Ship it to the class cluster¶
When this act is done, the same six tests will have run on the class
Kubernetes cluster — your servers on real machines, driven and checked
from inside the cluster itself — with one command; and, optionally,
you'll have poked at a live deployment with curl and kubectl.
What "deploying to Kubernetes" means¶
So far the three nodes were containers on your laptop. The class cluster is a set of real machines running Kubernetes — the standard system for running containers across many machines: you describe what you want (three copies of this program, listening on this port) and Kubernetes finds machines for them, starts them, and wires up the networking. A running container on Kubernetes is called a pod (for us: one pod = one node of your system).
Everything your team creates lives in your team's namespaces — a namespace is a named compartment of the cluster that keeps your team's stuff separate from every other team's. Your team gets two of them, and the split matters:
team-NN(use your team's; we writeteam-NNhere and everywhere below) — where your servers run. This is the room where faults happen: kills, partitions, CPU stress.team-NN-clients— where your tests run. Faults can never happen in here, by construction.
This split is Act 1's two networks, rebuilt out of namespaces. On your laptop, hokea kept the test's own traffic on a client network it never faulted, so it could always tell "the system is down" from "I broke my own view of it". On the cluster, the same guarantee comes from running the observer in a separate namespace that the fault machinery is not allowed to touch.
Which leads to the biggest change from Act 1: your test run itself now
happens inside the cluster. When you run hokea test, your laptop
packs up your project's files, ships them to the cluster, and pytest runs
in team-NN-clients — right next to your servers in team-NN, on the
cluster's fast internal network. Your laptop doesn't drive the load; it
just launches the run, streams the live pytest output to your screen, and
collects the recorded runs/ directories when the tests finish. (This
mirrors real practice: production load tests run near the servers, not
from a laptop on the campus WiFi.)
No packaging or uploading is involved for a project like the toy server: your handful of Python files rides along inside the request itself, and the cluster runs them on the same stock Python image Act 1 used. Call that Approach A — it's the course default, and the rest of this act uses it. Alternatively, you could package and publish your own container images (Approach B, in Appendix A at the end of this document), which may be useful for your actual projects as they get more complex. For this introductory lab, you will only need to learn Approach A.
Two rules Approach A imposes, both worth knowing before you write your
own project's server. First, the server must be plain Python that
imports only the standard library: the cluster runs your files on the
stock python:3.12-slim image exactly as they are — there is no
pip install step anywhere (Act 1's local runs build on that same image
and don't install anything either). A server that needs Flask, FastAPI,
gRPC, or any other package — or that isn't Python at all — needs
Approach B. Second, your test and adapter files can import only the
standard library plus requests (and hokea and pytest themselves):
the runner that executes pytest on the cluster has exactly those
installed, nothing else, so an adapter that imports anything more works
on your laptop and fails on hokea test. And one rule about layout:
only the files at the top level of your project directory ship to the
cluster — subdirectories don't. This differs from your laptop, where
Act 1's runs mount your whole directory, subdirectories included: code
laid out as a package (myserver/raft.py) passes every local test and
then fails on the cluster with an import error. Keep everything at the
top level, or use Approach B.
Are you still connected?¶
You installed kubectl and put your kubeconfig in place back in
Part 0. Re-run Part 0's proof command before going on — it takes a
second and saves a confusing failure later:
kubectl -n team-NN get pods
No resources found in team-NN namespace.
Anything else: Part 0's Troubleshooting under Cluster access.
Before your first run: put your name on your cluster¶
One small edit. Open the conftest.py you copied in Act 1 in your
editor and search for NAME = (about 40 lines down):
NAME = "toydemo" # e.g. "toydemo-alice" — see the lab, Act 3
Change it to your own name (lowercase letters, digits, and hyphens only, starting and ending with a letter or digit — the name ends up inside pod names, which Kubernetes is picky about):
NAME = "toydemo-alice"
Why: on your laptop, toydemo named a cluster on a machine only you use.
On the class cluster, your whole team shares team-NN, and hokea names
everything it creates after the cluster name — so if you and a teammate
both run tests as toydemo at the same time, the second run refuses to
start rather than trample the first (you'd see an error naming the
already-running pods; more on it below). With toydemo-alice and
toydemo-bob, you never collide. The rule worth internalizing, for this
course and afterwards: in a shared space, everything you create carries
your name.
Ship it: hokea test¶
From your toy-demo directory (any terminal — uv run finds the project
environment on its own):
uv run hokea test --namespace team-NN -q
Everything after the hokea options (-q here) is passed to pytest
untouched, so all your pytest habits work: -k partition selects tests
by name, -x stops at the first failure, -s shows printed output.
(If you'd rather not type the namespace every time, writing
HOKEA_NAMESPACE=team-NN in front of the command works too — it's the
same variable your conftest reads. In front of the one command only:
don't export it, or your plain local uv run pytest -q would start
trying to talk to the cluster too.)
What you'll see, in order — the whole run is streamed to your terminal live:
hokea: shipping 7 file(s) and starting the runner Job hokea-test-20260829-141530-3f2a in team-NN-clients (servers will run in team-NN)
hokea: runner pod is being created...
hokea: streaming pytest output from pod hokea-test-20260829-141530-3f2a-abc12 (Ctrl-C detaches; the Job keeps running)
hokea runner: running: pytest -q
...... [100%]
6 passed in 366.20s (0:06:06)
hokea runner: pytest exited 0; artifacts ready — holding up to 600s for the launcher to copy runs/ out
hokea: copied 6 run(s) into ./runs/: 20260829-141612-test_other_nodes_stay_available_when_one_dies, ...
hokea runner: copy-out confirmed; exiting
hokea: tests passed
(A Job is the Kubernetes object for a task that runs to completion —
here, your pytest run. Your Job name and timestamps will differ, and the
runner pod is being created line only shows up when the start takes a
moment. Expect the full suite to take about six minutes end to end — in
the same ballpark as the local run, with some extra startup per test
because pods on real machines take longer to come up than containers on
your laptop. During long stretches with no pytest output, hokea prints a
reassurance line about once a minute —
hokea: still waiting for pytest to finish (pod phase Running) — so a
long suite never just looks hung. Above the 6 passed line you'll see
a block headed warnings summary, and the line itself ends with
something like , 75 warnings — the count varies from run to run.
Warnings are advisory notices from Python (one mentions fork() and
deadlocks — a Python deprecation notice, not a report that anything
went wrong); they are not failures, and only failed or error mean
trouble. And you may
see a line about skipping ['__pycache__'] — that's hokea leaving out a
cache folder Python creates, which your tests don't need.)
Same six tests, same files, unchanged. The conftest.py you copied in
Act 1 checks an environment variable, HOKEA_NAMESPACE: hokea test
sets it inside the cluster-side run, so the fixture builds a
Kubernetes-backed cluster instead of a Docker one. Kills become pod
deletions, partitions become Chaos Mesh rules (Act 4 introduces Chaos
Mesh), and the recorded files and checks are byte-for-byte the same kind
of evidence.
Notice the second to last line: your runs/ came back. The tests ran
on the cluster, but every recorded run directory was copied into the same
local runs/ folder Act 1 used. You can open them exactly like before, with
ls and head. The results from the run will live on your laptop, even though
the run itself was on the cluster. Note that for large runs these could become
large.
Checkpoint — 6 passed and hokea: tests passed, and six new run
directories in your local runs/.
A few things worth knowing about this command:
- Only the files at the top level of your directory ship. The ride to
the cluster is a ConfigMap — a Kubernetes object that holds a small
bundle of files. Some files are skipped automatically, and hokea names
them in one line: the obvious machinery (
.venv,runs/, caches), lockfiles such asuv.lock, and image and archive files (.png,.pdf,.zip, and the like). Whatever is left must total no more than about 1 MiB — source code fits easily; if you go over, hokea refuses to start the run and names your biggest files. hokea also names anything else it skips out loud — if you seehokea: only top-level files ship to the cluster; skipping ['lib'] — tests that import these will fail on the cluster, it means exactly that: code you keep in that subdirectory won't be there. - Ctrl-C detaches, it doesn't kill. The run keeps going on the
cluster, and hokea prints the exact commands to re-attach to the log
or clean the run up. One catch, printed right there too: a detached
run's
runs/are not copied back automatically — the finished runner holds them for about 10 minutes for you to collect, then they are gone. Stay attached for the whole run unless you have a reason not to. - The exit code is pytest's own.
hokea: tests passedmeans the suite passed on the cluster; a red run prints pytest's real failure output (it was streaming to you all along) and exits nonzero, exactly like a local run.
Troubleshooting
hokea: which team? Pass --namespace <your team's namespace> ...— you left off--namespace.hokea: could not create the test Job in 'team-77-clients': ... Does that namespace exist, and are you a member of the team?— the namespace name is wrong (typo, or another team's). This one fails within seconds, before any test runs, and cleans up after itself.- An error inside the streamed pytest output saying
a cluster named 'toydemo' already has running pods in namespace 'team-NN' ...— the name collision from the previous section. If those pods are a teammate's live run, this is the system working: pick your ownNAMEand re-run. If they're yours (a crashed or Ctrl-C'd earlier run left them behind), the fix — the message points at it too — isHOKEA_TAKEOVER=1 uv run hokea test --namespace team-NN -q, which replaces them. (And if things are ever so tangled you just want your namespace empty,teardown.shat the end of this act sweeps everything hokea ever created out of it.) - The run starts, then sits for several minutes with no test output
at all (stuck before the first test finishes setting up) — tell
your instructor, and include the Job name (the
hokea-test-...token) from thehokea: shipping ...line. This failure is a course-infrastructure problem, not a mistake in your setup.
Same knobs, real machines¶
Two of the six tests were the Interlude's measurement experiments — which
means you just load-tested a deployment on real machines. Everything from
the Interlude works here, including the fleet-doubling variable, which
hokea test passes through to the cluster-side run. The numbers you get
here differ from the Interlude's, because these are real machines with the
cluster's internal network between them, not containers sharing one
laptop's CPU — the shape (reads scale, writes don't) is what should
match.
While a measurement runs, kubectl -n team-NN top pods (in a second
terminal) shows each server pod's live CPU use pinned at its half-CPU
limit while the p99 climbs — no setup, the cluster's metrics service is
already installed. If your team wants real dashboards with graphs over
time, you're free to run a monitoring stack like Prometheus+Grafana
inside your own namespace.
Scale it on the cluster: 3 nodes vs 6¶
You copied test_measure.py in the Interlude and ran its two experiments
on your laptop. Now the same two workloads, at both fleet sizes, on the
cluster — four runs. Start them back-to-back, since most of the time is
waiting (-k read / -k mixed selects one experiment by name, -s shows
its printed summary, and HOKEA_NODES=6 in front of a command runs six
server nodes for that one command only):
uv run hokea test --namespace team-NN -q test_measure.py -k read -s
HOKEA_NODES=6 uv run hokea test --namespace team-NN -q test_measure.py -k read -s
uv run hokea test --namespace team-NN -q test_measure.py -k mixed -s
HOKEA_NODES=6 uv run hokea test --namespace team-NN -q test_measure.py -k mixed -s
Each one looks like the hokea test launch above, with the experiment's summary in
the middle. The first, for example:
---- reads only: 3 nodes, 30 clients, 20s ----
throughput 222 ops/s
latency p50=15.33ms p99=439.24ms
outcomes {'ok': 4450}
.
1 passed, 1 deselected in 35.51s
hokea runner: pytest exited 0; artifacts ready — holding up to 600s for the launcher to copy runs/ out
hokea: copied 1 run(s) into ./runs/: 20260829-144210-test_read_throughput
hokea runner: copy-out confirmed; exiting
hokea: tests passed
(1 deselected is the other experiment -k filtered out — expected.)
One number worth understanding if you see it: outcomes may include an
'unknown' count, like {'ok': 1453, 'unknown': 9}. An unknown is an
operation whose answer never came back before the client gave up — hokea
can't tell whether it landed or not, so it refuses to guess and records
exactly that. Under heavy write load a few unknowns are normal (they're
the timeouts hiding inside that huge p99), and they're worth a close
look: timeouts under load are classic distributed-systems behavior, and
you'll meet them again in your own project.
Budget a minute or two per run. Watch the numbers as the four summaries go
by: read throughput should roughly double at 6 nodes; the mixed
workload's should not — the toy replicates every write to every
node, so extra machines bring exactly as much new replication work as
new capacity. The next section turns this into the graph.
Confirm all four runs came back — ls -t lists newest first, so the
Interlude's local throughput runs sit further down and head -4 cuts
them off:
ls -t runs/ | grep throughput | head -4
20260829-144210-test_read_throughput
20260829-144655-test_read_throughput
20260829-145120-test_mixed_throughput
20260829-145601-test_mixed_throughput
Checkpoint — the four newest throughput run directories are your
cluster runs (two read, two mixed; timestamps yours — remember
these are stamped in UTC).
Troubleshooting
- No summary printed but the test passed — you dropped
-s; the run still recorded everything (the graph below reads the files, not the screen), but re-run with-sif you want to see it live. HOKEA_NODES=6had no effect, or every later run silently became six-node —HOKEA_NODES=6must be in front of the command, as shown; it sets the variable for that one command only. Don'texportit.- A run collides with a teammate's — the
HOKEA_TAKEOVER/NAMETroubleshooting above applies.
One graph¶
Copy plot_scale.py into your toy-demo directory, uv add matplotlib
(one-time), uv run python plot_scale.py, open scale.png — reads
should roughly double, reads+writes should not. This picture is the
evidence you'd show to back up a claim about how your system scales.
cp ../hokea/examples/toyserver/plot_scale.py .
uv add matplotlib
uv run python plot_scale.py
(8 runs from other tests left off the chart — it only plots the measurement experiments)
reads + writes at 3 nodes: p99 latency 1076.18 ms
reads + writes at 6 nodes: p99 latency 5194.0 ms
reads only at 3 nodes: p99 latency 439.24 ms
reads only at 6 nodes: p99 latency 326.67 ms
wrote scale.png — open it to see how throughput scaled
(Your numbers and skipped-run count will differ; if you repeated an
experiment, a using your most recent ... run line tells you the newest
one won.) Open the picture — macOS: open scale.png; Linux:
xdg-open scale.png; WSL: explorer.exe scale.png. Two side effects
to know about: uv add matplotlib makes uv.lock grow, and
plot_scale.py leaves scale.png in your project folder — both are
skipped automatically when hokea ships your directory to the cluster,
so the cluster runs later in this lab are unaffected.
What you should see: two bars per fleet size. The reads only bar roughly doubles from 3 nodes to 6 — reads can be served by any node that has a copy, so more machines really do mean more capacity. The reads + writes bar stays roughly flat (or drops) — every write must be copied to every node, so the machines you added spend their new capacity copying writes around. That is the whole lesson in one picture: adding servers buys capacity only when the work per operation stays fixed, and full replication makes write work grow with the fleet. Making that tradeoff come out better — sharding, quorums, primary/backup — is what your course project is for.
Checkpoint — scale.png, showing read throughput roughly doubling
and mixed throughput staying put.
Troubleshooting
- An error asking you to run
uv add matplotlib— do exactly that, then re-run. run the scale experiments first — see the introductory lab, Act 3— the four runs above aren't inruns/(are you intoy-demo?).- A bar pair missing at one fleet size — one of the four runs didn't finish; re-run that command from the previous section.
The optional poke-at-it path: hokea k8s up¶
Test runs tear everything down when they finish. To get a deployment that
stays up so you can poke at it by hand, use the hokea command-line
tool directly (run it from your
toy-demo directory — and note the --name: same shared-space rule,
name it after yourself):
uv run hokea k8s up --namespace team-NN --name alice --src . --cmd "python server.py"
After half a minute or so:
hokea: ConfigMap source mode ships top-level regular files only; skipping ['.hokea', '.venv', 'runs']
Your cluster is up: 3 node(s) in namespace 'team-NN', and it stays up until you take it down.
node 1's last log lines:
toyserver node 1 listening on 8000
reach a node from here (each forward keeps running in its own terminal until Ctrl-C):
node 1: kubectl -n team-NN port-forward pod/hokea-alice-node1 8000:8000
then, in another terminal: curl http://localhost:8000/health
node 2: kubectl -n team-NN port-forward pod/hokea-alice-node2 8001:8000
then, in another terminal: curl http://localhost:8001/health
node 3: kubectl -n team-NN port-forward pod/hokea-alice-node3 8002:8000
then, in another terminal: curl http://localhost:8002/health
see your pods: kubectl -n team-NN get pods
take it down: hokea k8s down --name alice
(The skipping line is normal, and the exact list varies: anything in
your directory that isn't a top-level source file — the project
environment, runs/, caches, and by now uv.lock and scale.png —
is left out of the shipment, which is exactly what you want. One quirk of the printed hints: they omit the
uv run prefix, so type the last one as
uv run hokea k8s down --name alice.)
Your deployed nodes have no public address — pods on the class cluster
are reachable only from inside it (your tests, running in
team-NN-clients, reach them directly; that's the point of hokea test).
To reach one from your laptop, you open a temporary private tunnel with
kubectl port-forward — exactly the lines the output above printed: the
command connects a port on your laptop (localhost:8000) to the pod's
port, and keeps the tunnel open until you press Ctrl-C. So, in a
second terminal:
kubectl -n team-NN port-forward pod/hokea-alice-node1 8000:8000
Forwarding from 127.0.0.1:8000 -> 8000
Leave that running, and in a third terminal talk to your deployed node through the tunnel:
curl http://localhost:8000/health
{"ok": true, "node_id": 1}
Then look at your three nodes as Kubernetes sees them:
kubectl -n team-NN get pods
NAME READY STATUS RESTARTS AGE
hokea-alice-node1 1/1 Running 0 61s
hokea-alice-node2 1/1 Running 0 61s
hokea-alice-node3 1/1 Running 0 61s
(Test runs create and destroy their own fresh pods, named after your
conftest's NAME — hokea-toydemo-alice-... — and never touch this
standing deployment's hokea-alice-... pods; if you run tests while
this is up and see six pods at once, that's normal.)
That's your code, running on the class cluster. Write a key to node 1 and
read it back from node 2 — replication, across real machines. Open node
2's tunnel too, in a fourth terminal (its line is in the up
output — note it uses local port 8001, so both tunnels can stay open at
once), then, back in the third terminal:
curl -X PUT http://localhost:8000/kv/hello -d world
curl http://localhost:8001/kv/hello
{"ok": true}
{"key": "hello", "value": "world"}
uv run hokea k8s status --name alice re-prints the port-forward lines
and pod table any time. When you're done (cluster machines are shared —
always take your deployment down):
uv run hokea k8s down --name alice
removed .hokea/k8s-alice.json
deleted cluster 'alice' (pods, services, configmaps, chaos rules) from namespace 'team-NN'
If a run ever crashes half-way and leaves pods behind, sweep everything hokea created out of your namespace with:
../hokea/examples/toyserver/teardown.sh team-NN # team-NN → your team's namespace
No resources found
No resources found
Two lines of No resources found is the answer you want: the script
asks the cluster to delete everything hokea labeled in your namespace
(pods, services, and files in one sweep, chaos experiments in a second),
and each sweep reports that there was nothing left to remove — nothing
was left behind.
Limits of the no-image Approach A. The shipment (for hokea test
and hokea k8s up both) carries only the files at the top level of
your project directory (no subdirectories — unlike local runs, which
mount everything). Lockfiles (uv.lock) and image or archive files are
skipped automatically, and the launcher names what it skipped; whatever
remains must total no more than about 1 MiB, or the launcher refuses to
run and names your biggest files. Source code fits easily; a compiled
binary, a big dependency, or a bundled dataset does not. When your
project outgrows this, that's what Approach B is for.
Differences you'll notice on the cluster These matter when you write your project's server — the toy already follows them, so there's nothing to do right now:
- A restarted node comes back empty and with a new address — keep
node-local scratch in
/data(wiped on restart on both backends) and treat only replicated state as durable. You cannot assume there is a persistent "disk" available; it dies when the node dies. - Always reach peers by their
PEERSnames (node-2:8000), never by a cached IP address — fault injection targets those name-to-name paths. - Pod restarts take 5–20 seconds, not milliseconds.
- One local-only fault you haven't met yet —
cluster.pause()(freeze a node's process without killing it) — exists only on the Docker backend; on the cluster it stops with an error naming the Chaos Mesh alternative.cluster.throttle()(slow a node's CPU down) works on both backends: on the cluster it shrinks the pod's CPU limit in place (no restart) and verifies the applied value by reading it back.
Approach B — build and publish your own image¶
The class project doesn't require this, and nothing in the rest of this lab uses it. When your project outgrows Approach A — a server that needs packages beyond the standard library, isn't Python, keeps its code in subdirectories, or ships more than about 1 MiB — the answer is to package it as a container image and publish it where the cluster can download it. The full recipe, using the toy server, is Appendix A at the end of this document; skip it for now.
Act 4 — Break it in prod¶
When this act is done you will have watched real fault-injection machinery cut the network between pods on the class cluster, seen the experiment objects appear and vanish, and injected a fault of your own design from your own test — plus seen that hokea records even faults it didn't inject.
The same partition, injected by Chaos Mesh¶
On your laptop, hokea created partitions with firewall rules inside the containers. On the class cluster it delegates to Chaos Mesh, an industry-standard open-source fault-injection system for Kubernetes, used to test real production systems. Chaos Mesh injects a range of realistic faults by manipulating pods from the inside: network rules, CPU load, clock skew. On the class cluster (where I have installed it cluster-wide), hokea drives Chaos Mesh for you — it creates the fault experiments and then independently verifies each fault took hold, erroring out if not. In addition, you're not limited to what hokea itself can inject as faults: anything else you inject through Chaos Mesh yourself is still captured in the run's record — more on that at the end of this act.
Run just the partition test against the cluster — same launcher as Act 3, with a pytest filter on the end:
uv run hokea test --namespace team-NN -q test_partition.py
While it runs, watch the experiment object come and go in a second
terminal (-w means watch: the command keeps running and prints a new
line whenever an experiment appears or changes; Ctrl+C stops it):
kubectl -n team-NN get networkchaos -w
Be patient: nothing appears for the first minute or two — the runner and
then the test's server pods have to start before the workload's clock
begins. Then, around the workload's 8-second mark, hokea's experiment
shows up — a NetworkChaos object Chaos Mesh enforces for as long as it
exists (the name carries your conftest's NAME; yours will match your
own):
NAME ACTION DURATION
hokea-toydemo-alice-part0-x7k2m partition
(More lines for the same object may follow as its status updates, and the
blank DURATION is on purpose: hokea heals by deleting the object at
the 50-second mark rather than giving it a lifespan.) After the test
ends, press Ctrl+C and confirm nothing is left:
kubectl -n team-NN get networkchaos
No resources found in team-NN namespace.
That's the whole lifecycle: hokea creates a fresh experiment object to install a fault, verifies it took effect, and deletes it to heal. Back in your first terminal:
. [100%]
1 passed in 86.30s
hokea runner: pytest exited 0; artifacts ready — holding up to 600s for the launcher to copy runs/ out
hokea: copied 1 run(s) into ./runs/: 20260829-150312-test_partition_splits_replication_but_not_clients
hokea runner: copy-out confirmed; exiting
hokea: tests passed
Same test file as Act 2, and the run's evidence is back in your local
runs/ as always. Same verdicts, now proven against real machines with
more realistic fault machinery.
You can re-run Act 2's two greps on the new run directory and find the
same acked-then-lost write, this time across real machines. Several
*partition* runs match by now, and the cluster one is stamped in UTC,
so pick it by time written, as in Act 2:
RUN=$(ls -td runs/*partition* | head -1)
grep cut-probe "$RUN/history.jsonl"
grep -c cut-probe "$RUN/states/node-1.json" "$RUN/states/node-3.json"
Checkpoint — hokea: tests passed for the partition test on the
cluster (and, if you watched, you saw the networkchaos object appear
and vanish).
Troubleshooting
- The watch shows nothing ever — most likely the test failed before the fault fired (read the first terminal), or you watched the wrong namespace.
- The watch command itself errors (e.g.
the server doesn't have a resource type "networkchaos") — tell your instructor; the watch is a bonus, the test itself is unaffected.
Everything from Act 3's hokea test Troubleshooting applies here too.
Injecting a fault of your own design: CPU stress¶
Here's how you inject a fault hokea has no wrapper for: making one node's CPU busy, so it responds slowly — a straggler. You describe the experiment in a small file, and hokea runs it as a step in a normal recorded test run.
Create a file named stress.yaml — a manifest, a small text file
describing the experiment you want (change team-NN to your namespace and
toydemo-alice to your conftest's NAME; the format is YAML, which is
indentation-sensitive, so copy it exactly, spaces not tabs):
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
name: cpu-stress-node1
spec:
mode: all
selector:
namespaces: [team-NN]
labelSelectors:
hokea/cluster: hokea-toydemo-alice
hokea/node: "1"
stressors:
cpu:
workers: 2
load: 100
Reading it: find every pod labeled as node 1 of the hokea cluster named
toydemo-alice in your namespace (hokea labels every pod it creates with
hokea/cluster and hokea/node, exactly so you can target them like
this), and run 2 CPU-burning workers at full load inside it. Note the
label's value is hokea- + your conftest's NAME — hokea adds that
prefix, so keep it when you substitute your own name. Notice also what
you did not write: no duration — your test decides when the stress
ends.
Now hand the file to hokea inside a test. The chaos fixture takes your
YAML as a fault-schedule step, exactly like the built-in faults. Put the
test in a new file, test_stress.py, next to the others, and start it
with the two-line skip marker shown below — this test is cluster-only:
the chaos fixture doesn't exist on the Docker backend, and without the
marker pytest would still collect the test locally, build and start the
whole Docker cluster, and then error out asking for chaos. The marker
tells pytest to skip the test unless HOKEA_NAMESPACE is set — which
hokea test does inside the cluster-side run, and your plain local
uv run pytest -q never does — so locally you'll see 6 passed, 1
skipped and the suite stays green. Run it with
uv run hokea test --namespace team-NN -q test_stress.py — both new files
sit at the top level of your directory, so they ship with everything
else, and chaos.apply("stress.yaml") finds the YAML right there on the
cluster side:
import os
import pytest
from test_toy import ToyAdapter, toy_ops
from hokea import check
pytestmark = pytest.mark.skipif(not os.environ.get("HOKEA_NAMESPACE"),
reason="Chaos Mesh exists only on the class cluster")
def test_cpu_stress_makes_node1_a_straggler(cluster, chaos, run):
# clients=2 keeps the load light on purpose: this experiment measures
# how slow the stressed node's responses get, and we don't want the
# queueing delays of a saturated cluster (which the Interlude created
# deliberately) drowning that signal out.
h = run(toy_ops, ToyAdapter(), duration=60, clients=2,
faults=[(10, lambda: chaos.apply("stress.yaml")),
(40, chaos.clear)])
a = check.availability(h.history, h.faults)
stressed = a.window("chaos:stresschaos")
# The stressed p99 typically lands several times above the baseline's;
# assert the direction, not a specific factor.
assert stressed.side("all").latency_ms["p99"] > \
a.window("baseline").side("all").latency_ms["p99"]
The test itself runs for a bit over a minute (duration=60 plus the
usual startup), ending in the familiar
1 passed ... hokea: tests passed tail. At the 10-second mark
chaos.apply creates your experiment and waits until Chaos Mesh
confirms the stress is actually running inside the pod before the
schedule moves on; at 40, chaos.clear deletes it and waits for the
recovery to finish. The run's faults.jsonl shows the whole lifecycle:
{"event": "chaos", "kind": "stresschaos", "chaos_name": "cpu-stress-node1-wr6gq", "source": "applied", ..., "ts": 10.32}
{"event": "chaos_done", "kind": "stresschaos", "chaos_name": "cpu-stress-node1-wr6gq", "ts": 11.36}
{"event": "chaos_recovered", "kind": "stresschaos", "chaos_name": "cpu-stress-node1-wr6gq", "ts": 40.09}
Two details worth noticing. The experiment's name grew a random suffix:
hokea creates a fresh experiment object on every apply, because Chaos
Mesh ignores a re-applied used one — so your test can re-run forever
without you renaming anything. And check.availability turns the
chaos_done→chaos_recovered span into a window (here labeled
chaos:stresschaos:cpu-stress-node1-wr6gq; a.window(...) matches by
prefix, which is why the test can ask for just chaos:stresschaos), so
the assert reads the latency of exactly the operations that ran under
stress.
Troubleshooting
chaos.applyerrors out saying Chaos Mesh never confirmed the fault was injected — the usual cause is a selector matching zero pods: yourstress.yaml's label doesn't match yourNAME(check withkubectl -n team-NN get pods --show-labelswhile a test runs).- The p99 assert itself fails — re-run once before digging: with
only two light clients the signal is real but small, and one quiet
run can happen. Two failures in a row means look at
faults.jsonland check the stress actually landed on a node your clients were talking to.
Don't expect every response in that window to be slow — the effect is spikier than that: most responses stay near the run's normal p50, but again and again the stressed pod uses up its CPU allowance for the current scheduling slice and everything inside it — your server included — waits for the next one. That's why the assert reads the p99 rather than the p50 (both defined back in the Interlude): the spikes move the p99 hard while the p50 barely notices — exactly the straggler behavior this experiment set out to create.
You can still kubectl apply -f stress.yaml by hand (or build an
experiment in the Chaos Mesh dashboard — Appendix B) while a test runs —
hokea watches the
namespace and records experiments it didn't create the same way, so it
lands in faults.jsonl too. chaos.apply just spares you the fiddly
parts: the timing (the pods must exist when the experiment is created,
and inside the schedule they always do), the namespace (hokea fills in
yours), the re-run renaming, and the cleanup — anything you forget to
clear() is deleted when the cluster comes down.
Where this leaves you¶
Count what you watched hokea catch in these four acts: writes acknowledged and then lost when a node died; a node that came back empty with nobody re-syncing it; writes that never crossed a partition and were never retried; a node that a little CPU pressure turned into a straggler. Every one of those is a gap in the toy server — no retries, no re-sync, no durability, no defenses — and every one has a known remedy that real systems implement.
Your course project is to build a system where these same types of tests -- the faults, the correctness checks, etc -- come up clean on both your laptop and the cluster because your design actually accounts for their possibility, unlike the toy example we've used here. Remember, hokea isn't a grading script: it records what your system did, and your role is to use it to write tests to show that your claims hold given the evidence hokea collects.
Appendix A — Approach B: build and publish your own image¶
The class project doesn't require this. But "package my program as an image anyone can run" is a skill worth having, and it's the industry answer to what Approach A does with a file shipment. It works on the class cluster, too: the cluster downloads publicly published images from the internet exactly like your laptop does, so once your image is published — and marked public — the cluster can run it. Here's the whole path, using the toy server.
Why is there an "upload" step at all? A machine that will run your image
must be able to download it, and your laptop isn't reachable from other
machines — so you push the image to a registry, a public server that
hosts images (like a package index, but for container images). We'll use
GitHub's free one, ghcr.io.
1. Write a Dockerfile. In your toy-demo directory, create a file
named Dockerfile (a recipe for building an image) with exactly this:
FROM python:3.12-slim
WORKDIR /app
COPY server.py .
EXPOSE 8000
CMD ["python", "server.py"]
Line by line: start from the official slim Python image; work in /app;
copy the server code in; document that it listens on port 8000; run it.
2. Build and test it locally.
docker build -t toyserver:v1 .
...
=> => naming to docker.io/library/toyserver:v1
Run one container from it and check it answers (this will sit in the foreground; that's expected):
docker run --rm -p 8000:8000 toyserver:v1
toyserver node 1 listening on 8000
In a second terminal:
curl http://localhost:8000/health
{"ok": true, "node_id": 1}
Press Ctrl+C in the first terminal to stop it.
3. Create a registry account and log in. On GitHub: Settings →
Developer settings → Personal access tokens → Tokens (classic) → Generate
new token (classic), check the write:packages scope, and copy the
token it shows you. Then (replace YOURNAME with your GitHub username,
here and everywhere below):
docker login ghcr.io -u YOURNAME
Paste the token when it asks for a password.
Login Succeeded
4. Tag and push. A registry image name includes where it lives:
docker tag toyserver:v1 ghcr.io/YOURNAME/toyserver:v1
docker push ghcr.io/YOURNAME/toyserver:v1
...
v1: digest: sha256:… size: …
One last click: on GitHub, go to your profile → Packages → toyserver →
Package settings → Change visibility → Public — so machines can
download it without logging in.
5. Point hokea at it. In conftest.py, the Kubernetes branch changes
from image= + src= + cmd= to image= alone — your code is inside
the image now, and the image's own start command runs:
return KubeCluster(image="ghcr.io/YOURNAME/toyserver:v1",
nodes=nodes, name=NAME,
namespace=namespace, port=8000, env=ENV,
cpus=0.5, memory="256Mi",
workdir=tmp_path / "cluster")
On any cluster that can pull public images — the class cluster
included — hokea test then works exactly as before, with the pods
running your published image. Two caveats. First, the Public click in
step 4 is required: the cluster has no login for your registry
account, so a private image fails to download — pods sit in
ImagePullBackOff, Kubernetes-speak for "couldn't fetch the image".
If your image genuinely can't be made public (say it contains something
you may not publish), ask your instructor: the course staff can load an
unpublished image directly onto the cluster's machines by hand, and
your conftest change above works unchanged once they have. Second, the
hokea k8s up command always ships a --src directory, so the
image-alone mode is used from conftest.py rather than the CLI.
That's the entire industry workflow — Dockerfile, build, push, deploy — and now you've done it once. For the class, Approach A remains the default: with Approach B, every code change means rebuilding and re-publishing the image before you can re-run, which is safer but slower. (Real teams automate exactly that step — a GitHub Action that rebuilds the image on every push is easy to set up, and left as an exercise.)
Appendix B — The Chaos Mesh dashboard and Workflows (optional; needs a per-team token)¶
Nothing in the lab needs this appendix. The dashboard needs a per-team login token that I hand out separately, later in the course. It's here so you know what's coming.
The Chaos Mesh dashboard¶
Chaos Mesh also has a web dashboard — your instructor gives you the URL and
a login token. It shows every experiment in your namespace on a timeline
(including the ones hokea creates during your test runs, appearing and
disappearing live), and it can inject fault types hokea doesn't wrap:
CPU/memory stress, disk I/O delays, clock skew, and more. And they land in
your run's record too: hokea watches your namespace for chaos experiments
it didn't create, so a fault you inject by dashboard or by hand shows up
in faults.jsonl as chaos / chaos_done / chaos_recovered events
named after your experiment, and check.availability() gives it a window
labeled chaos:<kind>:<name>. (hokea doesn't attribute hand-made faults
to specific nodes — the window has a single side, all.)
Chaos Mesh Workflows¶
So far, one file meant one fault. Chaos Mesh also has Workflows: a
single YAML that chains several experiments — run this CPU stress for 30
seconds, then that network delay, or both at once — like a fault schedule
that lives on the cluster. One thing to understand before using one: a
workflow times its steps relative to its own start, cluster-side,
knowing nothing about your client load — by itself it can't line up with
your run. Applying it through hokea closes that gap: hokea anchors the
workflow's start to a known moment in your schedule, and as each step's
experiment fires, it lands in faults.jsonl individually, with real
timestamps — the record shows what actually happened and when, not what
the workflow intended.
# a sketch — workflow.yaml is a file you'd write yourself (link below)
faults=[(10, lambda: chaos.apply("workflow.yaml")),
(50, chaos.clear)]
How to write one: https://chaos-mesh.org/docs/create-chaos-mesh-workflow/