# Linux systems — NetWatch Labs kit

A small, inspectable set of experiments for the foundations and advanced pathways.
Use a Linux machine or VM. All core checks are bounded and run without root.

## Start here

Requirements: C11 compiler with pthreads, make, Python 3, Rust 1.74 or newer.
The Rust server uses only the standard library; no crates or package downloads.
Optional lesson tools: strace, perf, bpftrace. Their permissions and availability
are independent of whether the core kit builds and passes.

```sh
make
make check
```

`make check` runs ten C experiment modes and checks the Rust service's protocol,
fragmentation, bounded frames, malformed commands, shutdown and journal recovery.
It reports observations for timing and fault counts, not universal pass thresholds.
The pidfd check reports SKIP when unavailable or restricted.

## What is included

| File | Purpose |
|---|---|
| probe.c | fd, pipe, cow, memory, schedule, shared, local, level, edge, pidfd |
| pocketdb.rs | Four-worker loopback TCP service with an append journal |
| client.py | One-command client and bounded closed-loop benchmark |
| verify.py | Functional checks with temporary scratch storage |
| report-template.md | Investigation structure and capstone rubric |
| Makefile | Build, check and clean |

Read each mode's implementation before editing it. Keep an unmodified baseline
or use version control so you can compare your prediction with the original.
The probes check semantic invariants and print measured data where appropriate.

## Run the service

Terminal A:

```sh
LAB_DIR=$(mktemp -d)
# Write this path in your notebook; reuse it for a recovery exercise.
./pocketdb --dir "$LAB_DIR" --port 0 --sync always --delay-ms 0
```

The first line prints `LISTEN PORT ...`. Port zero asks the OS to choose a free
port. In terminal B, replace 12345 with that number:

```sh
PORT=12345
python3 client.py "$PORT" PING
python3 client.py "$PORT" "SET topic linux"
python3 client.py "$PORT" "GET topic"
python3 client.py "$PORT" --bench 100 --workers 4
python3 client.py "$PORT" STOP
```

STOP returns BYE, stops new accepts and lets queued workers finish or time out.
This is a local teaching control command, not authentication. The server binds
only to 127.0.0.1. Do not expose it as a public service. Signal-driven graceful
shutdown is an extension; the reference implementation uses the STOP protocol.

## Protocol and limits

- Newline-delimited PING, SET key value, GET key and STOP.
- Keys/values: 1–128 ASCII letters, digits or `_ - . /` (no spaces).
- Maximum frame: 4096 bytes including its terminating newline.
- At most 1024 distinct keys. Existing keys may be updated at capacity.
- Four workers; sixteen pending connections in a bounded channel.
- Full pending queue: BUSY when a reply can be written.
- Two-second socket read/write timeout; a total request deadline is an extension.
- Connections can carry multiple commands. This is not a request-fair scheduler.
- The in-memory key count is bounded, but the journal grows until you remove
  scratch data or implement compaction. Core tests use small temporary logs.

## Journal contract

`--sync always` appends the record and calls sync_data before publishing its map
update and replying OK. The existing scratch directory is supplied by the caller;
startup syncs the journal and its directory entry. Durability of newly created
ancestor directories is outside the kit's contract.

`--sync none` omits per-SET sync. It is a different acknowledgement contract, not
an equivalent optimisation. A process-restart test does not prove power-loss
survival: the OS and device can still retain data.

Recovery replays complete valid lines, truncates an incomplete final suffix and
refuses malformed complete records. It has no checksums or arbitrary corruption
repair. A journal write/sync failure puts subsequent writes into a failed state.
No WAL compaction, multi-key transaction or concurrent process sharing is provided.
Run only one server against a scratch directory at a time.

## Benchmark interpretation

The Python client is **closed-loop**. Each worker waits for a result before
sending more work. Server delay reduces offered rate. It is not a fixed-arrival
rate SLO test and can mask queue growth that an external arrival process causes.

Percentiles use successful requests and nearest-rank selection. Always report
attempt/success/failure counts alongside them. Record worker count, sync mode,
request count, injected delay, kernel, hardware and repetitions. A single maximum
or a p99 over very few samples is not a stable service guarantee.

The local counter loop can be simplified by the optimiser. Its comparison to a
shared atomic measures whole algorithms, not isolated cache-coherence latency.
The schedule probe includes timer and scheduling effects. The memory probe's
fault counts depend on huge-page and other policies.

## Troubleshooting and cleanup

- Missing rustc: build `make probe` first; install Rust >= 1.74 for the server.
- Permission denied from perf/BPF: record the limitation; do not interpret it as
  an empty profile. The core kit requires neither facility.
- Connection refused: verify the server is running and copy its new port.
- ERR command: check newline framing and restricted token characters.
- Error on recovery: preserve the scratch journal before investigating; malformed
  complete data is intentionally rejected.
- Kernel lessons: use a separate kernel checkout and follow its own prerequisites.
  They are not run by `make check`, which does not build or boot a kernel.

`make clean` removes only the two binaries in this directory. `verify.py` removes
its own temporary data. Remove manually created scratch directories only after
checking their paths and preserving anything you want in your report.
