Skip to content
clstr.io

Persistence

In this stage, you’ll make your key-value store survive clean shutdowns. When the server restarts, it picks up exactly where it left off.

Graceful Shutdown

Section titled “Graceful Shutdown”

When your server receives a termination signal (SIGTERM or SIGINT), it should:

  • Wait for in-flight requests to complete (within 5 seconds)
  • Save all key-value pairs to disk
  • Exit with status code 0

Startup Recovery

Section titled “Startup Recovery”

When your server starts, it should:

  • Check the DATA_DIR environment variable for the data directory path
  • Load any previously saved key-value pairs
  • Continue serving requests with the restored data

If no previous data exists, start with an empty store.

Storage

Section titled “Storage”

Save your in-memory state to disk during shutdown and restore it on startup. Create your data files in the directory given by the DATA_DIR environment variable. The serialization format and file naming are up to you: JSON, binary (Protocol Buffers, MessagePack, BSON, gob, pickle), plain text, whatever.

This approach survives clean shutdowns but not crashes. If the process dies unexpectedly, you’ll lose any data that wasn’t saved. That’s fine for this stage. You’ll add crash recovery in the next one.

Testing

Section titled “Testing”

The test harness mounts a persistent volume at /app/data and sets the DATA_DIR environment variable to /app/data.

You can test your implementation using the clstr command:

Terminal window
$ clstr test
Testing persistence: Data Survives SIGTERM


✓ Data Survives a Graceful Restart (456ms)
✓ All Data Survives Repeated Graceful Restarts (1.73s)
✓ Rapid Sequential Writes Survive a Graceful Restart (813ms)
✓ Rapid Concurrent Writes Survive a Graceful Restart (781ms)
✓ CLEAR Survives a Graceful Restart (354ms)


PASSED ✓


Run clstr next to advance to the next stage.

The tests will:

  • Store data in your server (through HTTP API calls)
  • Send SIGTERM to trigger graceful shutdown
  • Restart your server
  • Verify all data is still present

Debugging

Section titled “Debugging”

Your server’s output (stdout/stderr) is captured during testing and viewable with clstr logs. The [STOP] and [START] events show exactly when the server was restarted:

Terminal window
$ clstr logs n1
[n1]  +0.000s     [START]
[n1]  +0.127s     Server listening on 0.0.0.0:8080
[**]  +0.379s     [CLUSTER READY]
[**]  +0.380s     [TEST: Data Survives a Graceful Restart]
[n1]  +0.393s     PUT /kv/kenya:capital accepted, value=Nairobi
[n1]  +0.511s     PUT /kv/germany:capital accepted, value=Berlin
[n1]  +0.512s     Flushing 2 keys to /app/data/store.json
[n1]  +0.513s     [RESTART: STOP]
[n1]  +0.576s     Server stopped
[n1]  +0.963s     Server listening on 0.0.0.0:8080
[n1]  +0.964s     Loaded 2 keys from /app/data/store.json
[n1]  +1.052s     GET /kv/kenya:capital returning 200
[n1]  +1.236s     GET /kv/germany:capital returning 200
[**]  +1.237s     [TEST: All Data Survives Repeated Graceful Restarts]
[n1]  +1.249s     [RESTART: STOP]
[n1]  +1.312s     Server stopped
[n1]  +1.575s     Server listening on 0.0.0.0:8080
[n1]  +1.576s     Loaded 2 keys from /app/data/store.json
[**]  +2.969s     [TEST: Rapid Sequential Writes Survive a Graceful Restart]
[n1]  +3.252s     [RESTART: STOP]
[n1]  +3.289s     Server stopped
[n1]  +3.513s     Server listening on 0.0.0.0:8080
[n1]  +3.514s     Loaded 100 keys from /app/data/store.json
[**]  +3.781s     [TEST: Rapid Concurrent Writes Survive a Graceful Restart]
[n1]  +3.810s     [CONCURRENTLY: 1000 req, 0.00% err · p50=1ms p95=4ms p99=204ms max=205ms]
[n1]  +3.810s     [RESTART: STOP]
[n1]  +3.842s     Server stopped
...