Skip to content
clstr.io

Crash Recovery

Your server survives clean shutdowns but loses everything if it crashes. In this stage, you’ll add a write-ahead log so data survives unexpected failures too.

Write-Ahead Logging

Section titled “Write-Ahead Logging”

Implement a Write-Ahead Log (WAL) that records operations before they’re applied to memory. If the process crashes between a write and its memory update, the log survives and you can replay it on restart to reconstruct state.

Log Format

Section titled “Log Format”

Your log should record operations in append-only fashion. The format is up to you: JSONL (one JSON object per line), binary serialization, or plain text all work.

Each log entry needs enough information to replay the operation:

  • Operation type (e.g., set, delete, clear)
  • Key
  • Value
  • Any other metadata you need for replay

Durability

Section titled “Durability”

After appending an operation to the log, ensure it’s physically written to disk before responding to the client. Use your language’s file sync mechanism (fsync, flush, etc.) to force the operating system to persist the write.

Without fsync, the OS may buffer writes in memory and you’ll lose data on crash.

Syncing on every write is slow: you’re blocking the response on a disk round-trip, and holding locks during that I/O serializes concurrent writers. This is the right trade-off for a simple implementation, but if the concurrent write tests start failing, consider batching multiple operations into a single fsync to amortize the cost.

Recovery Procedure

Section titled “Recovery Procedure”

When your server starts:

  • Load the most recent snapshot (from the persistence stage) if one exists
  • Replay all operations from the WAL that occurred after the snapshot
  • Resume serving requests

If no snapshot exists, replay the entire log from the beginning.

Checkpointing

Section titled “Checkpointing”

As your log grows, replaying from the beginning becomes slow. Periodically create snapshots of your in-memory state and truncate the log.

When to checkpoint is up to you: after N operations, every M seconds, when the log reaches a certain size, etc. The test doesn’t care about your checkpoint strategy, only that recovery works correctly.

After creating a snapshot:

  • Write the snapshot to a new file
  • Truncate or create a new WAL file
  • Continue logging operations

On recovery, load the latest snapshot and replay only the operations logged after that snapshot.

Storage

Section titled “Storage”

You now have two types of files:

  • Snapshot: full state at a point in time (from the previous stage)
  • WAL: operations logged since the last snapshot

Organize these in the data directory however makes sense.

Testing

Section titled “Testing”

The test harness mounts a persistent volume at /app/data and sets the DATA_DIR environment variable to /app/data, same as the previous stage.

Your server will be tested with unexpected crashes:

Terminal window
$ clstr test crash-recovery
Testing crash-recovery: Data Survives SIGKILL


✓ Data Survives a Hard Crash (464ms)
✓ All Data Survives Repeated Hard Crashes (1.76s)
✓ Rapid Sequential Writes Survive a Hard Crash (806ms)
✓ Rapid Concurrent Writes Survive a Hard Crash (496ms)
✓ CLEAR Survives a Hard Crash (351ms)


PASSED ✓


Run clstr next to advance to the next stage.

The tests will:

  • Store data in your server (through HTTP API calls)
  • Kill the server process (SIGKILL) without warning
  • Restart your server
  • Verify all data that was acknowledged before the crash is still present

Debugging

Section titled “Debugging”

Your server’s output (stdout/stderr) is captured during testing and viewable with clstr logs. The [KILL] and [START] events show exactly when the crash and restart occurred:

Terminal window
$ clstr logs n1
[n1]  +0.000s     [START]
[n1]  +0.127s     Server listening on 0.0.0.0:8080
[**]  +0.377s     [CLUSTER READY]
[**]  +0.378s     [TEST: Data Survives a Hard Crash]
[n1]  +0.389s     PUT /kv/canada:capital accepted, value=Ottawa
[n1]  +0.511s     PUT /kv/brazil:capital accepted, value=Brasilia
[n1]  +0.693s     PUT /kv/australia:capital accepted, value=Canberra
[n1]  +0.875s     PUT /kv/japan:capital accepted, value=Tokyo
[n1]  +0.876s     Appended 4 entries to /app/data/wal.log
[n1]  +0.877s     [RESTART: KILL]
[n1]  +1.127s     Server listening on 0.0.0.0:8080
[n1]  +1.128s     Replaying 4 entries from /app/data/wal.log
[n1]  +1.217s     GET /kv/canada:capital returning 200
[n1]  +1.401s     GET /kv/brazil:capital returning 200
[n1]  +1.584s     GET /kv/australia:capital returning 200
[n1]  +1.768s     GET /kv/japan:capital returning 200
[**]  +1.769s     [TEST: All Data Survives Repeated Hard Crashes]
[n1]  +1.781s     [RESTART: KILL]
[n1]  +2.031s     Server listening on 0.0.0.0:8080
[n1]  +2.032s     Replaying 4 entries from /app/data/wal.log
[**]  +2.965s     [TEST: Rapid Sequential Writes Survive a Hard Crash]
[n1]  +3.249s     [RESTART: KILL]
[n1]  +3.499s     Server listening on 0.0.0.0:8080
[n1]  +3.500s     Replaying 100 entries from /app/data/wal.log
[**]  +3.771s     [TEST: Rapid Concurrent Writes Survive a Hard Crash]
[n1]  +3.810s     [CONCURRENTLY: 1000 req, 0.00% err · p50=1ms p95=4ms p99=5ms max=207ms]
[n1]  +3.810s     [RESTART: KILL]
...