Data & durability¶
What survives a crash (nothing), and where data lives¶
This page answers three questions every project hits sooner or later: what actually happens to a node's data when hokea kills it, where your nodes can write scratch files, and how to give every node a read-only dataset (a log corpus, MNIST) to work on. Read the first section before you write any durability claim for your project; everything else on this page follows from it.
The fault model: crash-stop, and what it does to your data¶
When hokea kills a node — cluster.kill(2), or a fault schedule doing the
same — the node simply halts. It doesn't flush anything, doesn't say
goodbye to its peers, doesn't get a chance to run cleanup code. This style
of failure has a textbook name, crash-stop: a failed node does nothing
further, ever.
A test may restart the node later, and this is the part that matters for
how you design your system: a restarted node is best understood as a
brand-new node that happens to reuse the old one's name. It comes back
with empty memory and an empty /data directory. Nothing it knew before
the crash survives. On the Kubernetes backend (the class cluster) this is
literal — the old pod (the node's running container: the isolated box its
process lives in) is deleted and a fresh one is created; on the Docker
backend hokea gives you the same behavior, so that a test passing on your
laptop means the same thing on the cluster. (One Docker difference: files
written outside /data, into the container's own filesystem, can
survive a restart there. More on that later in this page.)
The consequence for your safety claims: under this fault model, durability can only come from replication (or from recomputing the data from inputs that still exist). A write is durable when enough other nodes know about it, never because a disk has it. "We saved it to a file before acknowledging" is not a durability mechanism here, because the file dies with the node. When you write a claim like "a confirmed reservation survives any single server dying", the mechanism backing it has to be "at least one other node also has the reservation before we confirm".
A caveat if you're building on a consensus protocol. Textbook crash-recovery designs assume something hokea deliberately doesn't give you: storage that survives a crash. Raft (a consensus protocol we will cover in lecture), for example, requires each node to persist its log and its vote so that a restarted node rejoins remembering what it promised. Under hokea's fault model you have two legitimate options, and the course spec accepts either — but it must be a choice you write down in your fault model, not a behavior you discover during a demo:
- Treat a restarted node as a new member that must be (re-)admitted to the group and caught up from its peers, exactly like adding a fresh machine.
- Scope restarts out of your fault model: your claims hold under nodes
crashing and staying dead (and you never call
cluster.starton a killed node in your falsification tests).
One more fault to file correctly: cluster.pause() (Docker backend only)
freezes a node's process without killing it, and unpause resumes it with
all its memory intact. That is not crash-recovery — it models a very
slow node, or a long garbage-collection pause: the node vanishes for a
while and then continues as if nothing happened, still believing everything
it believed before.
/data: the per-node scratch space¶
Every node has a private writable directory at /data. It belongs to that
one node alone, and it is wiped whenever the node dies or restarts — on
both backends. What backs it differs: on the cluster it is a plain
directory on the machine's own local disk (Kubernetes calls this an
emptyDir) — fast, and as big as the disk allows; on
the Docker backend it is kept in RAM (a tmpfs) and counts against the
node's memory limit — fine for caches and small spills, but don't test
bigger-than-memory spill files on your laptop and expect the space to be
there.
Use it for anything you can afford to lose: spill files when a computation
doesn't fit in memory, caches, a SQLite file you treat as a disposable
local index. Do not put anything there that a safety claim depends on.
As the section above explains, the moment the node crashes /data is
gone, and a check like acked_visible will show you the
missing writes.
(The rest of the container filesystem is technically writable, but don't
use it: on the Docker backend files written there can even survive a
restart, while on the cluster a restarted pod keeps nothing — so a habit of
writing outside /data lets your laptop tests pass where the cluster
fails. /data behaves the same on both backends: gone on restart.)
Datasets: read-only inputs with data=¶
A dataset — a log corpus, MNIST, anything your nodes read but never
write — is passed to your cluster as data=. One source appears
read-only at /dataset in every node (hokea mounts it — attaches it
into each node's filesystem); a dict {name: source} mounts each source
at /dataset/<name>.
What a source is depends on the backend:
- Docker backend: a host directory, mounted live — edit a file on your laptop and the nodes see the change immediately.
- Kubernetes backend: an OCI image reference — the name of a
container image (a frozen filesystem snapshot, the kind Docker builds),
like
ghcr.io/you/my-corpus:v1— the same kind of name you'ddocker pull. A dataset image is justFROM scratch+COPY corpus/ /: a filesystem shipped as an image, no code, nothing to run — built and tagged like any image. The cluster attaches it read-only, keeping one on-disk copy per cluster machine no matter how many teams use it.
The paths are identical on both backends, so your node code reads
/dataset/... the same everywhere. The difference is mutability: the
Docker mount is live, the image is immutable — to change a dataset on the
cluster, push a new tag (v2) and pass that.
Here is the pattern in a conftest.py, with the same HOKEA_NAMESPACE
switch as the introductory lab's toy project. You never set that
variable yourself: hokea test ships your files to the cluster and sets
it for the cluster-side run — it always names your server namespace
(team-NN: a namespace is your team's named compartment on the cluster,
and the server one is where your nodes run); plain pytest on your laptop
leaves it unset and gets the Docker branch:
import os
from pathlib import Path
import pytest
HERE = Path(__file__).parent
@pytest.fixture
def hokea_cluster(tmp_path):
# HOKEA_NODES=6 pytest ... reruns the same tests at a different fleet size
nodes = int(os.environ.get("HOKEA_NODES", "3"))
namespace = os.environ.get("HOKEA_NAMESPACE")
if namespace:
from hokea.kube import KubeCluster
return KubeCluster(image="python:3.12-slim", src=HERE,
cmd="python server.py", nodes=nodes,
# your team shares its namespace: make the name
# yours alone, e.g. "myproj-alice" (see the
# introductory lab, Act 3)
name="myproj", namespace=namespace, port=8000,
workdir=tmp_path / "cluster",
# k8s: an image reference. This is one of the
# course dataset images (see the Datasets page).
data="ghcr.io/spin-vt/datasets/wikitext-2:v1")
from hokea import Cluster
return Cluster(image="python:3.12-slim", src=HERE,
cmd="python server.py", nodes=nodes, name="myproj",
port=8000, workdir=tmp_path / "cluster",
# Docker: a plain directory on your machine
data=HERE / "corpus")
Node code then just opens /dataset/wiki.train.tokens — same line on your
laptop and on the cluster. Need more than one dataset? The dict form mounts
each under its name:
data={"corpus": "ghcr.io/spin-vt/datasets/wikitext-2:v1",
"logs": "ghcr.io/spin-vt/datasets/weblogs-synth:v1"}
# node code: /dataset/corpus/wiki.train.tokens
# /dataset/logs/events-000.jsonl
(On the Docker backend the dict values are directories, one per name.)
Sharding convention. Every node sees the same /dataset — hokea does
not split it for you, because how to split is a design decision your
project owns. One convention you might use: node i picks its own slice using NODE_ID
(the 1-based id every node reads from its environment — see the contract in
the introductory lab):
shards = sorted(Path("/dataset").glob("*.jsonl")) # same sorted list on every node
# every N-th shard, starting from this node's own offset
mine = shards[int(os.environ["NODE_ID"]) - 1::int(os.environ["N_NODES"])]
Every node runs the same code, each takes every N-th file starting at its own offset, and together they cover the dataset exactly once.
Tying this back to the fault model above: /dataset is read-only and
comes back intact after a restart. It is the one kind of "storage that
survives a crash" you do have, precisely because nobody can write to it.
Recomputing lost state from /dataset (re-reading your shard after a
restart) is a legitimate durability mechanism; a MapReduce worker that
re-runs its task from the input files is the classic example.
Course datasets¶
The course ships four ready-made dataset images — fashion-mnist,
mnist, wikitext-2, and weblogs-synth — and any team can publish its
own the same way. The Datasets page is the catalog:
contents, sizes, licenses, and the two-line Dockerfile recipe for
publishing yours, and it lists the exact image references to put in your
conftest.