feat(rig): the audio server can die alone — supervise scsynth, not sclang (#115)

The rig went silent overnight and NOTHING recovered it. Seven layers watched it
happen. From the journal:

    10:37:36  PM: suspend entry (s2idle)
    10:48:38  systemd-coredump: scsynth terminated abnormally
    10:48:38  sclang: "Server 'localhost' exited with exit code 0."
    11:18:47  Bridge: "scsynth process not found"   x63, to the journal only

scsynth is a CHILD of sclang and systemd's MainPID is sclang, so the unit reported
`active (running)` for two days while there was no audio server at all. A green unit
is not sound.

Two findings behind it:

* `parvagues-sc.service` was DISABLED. Six user units start with the session —
  gig-log, bridge, perf-tray, tidal-ardour-autoroute, lcxl-leds-watch,
  midi-autoconnect — and the only one that produces audio was not among them. It had
  been alive since Jul 28 purely as a manual leftover, which is why its death was
  permanent. Now enabled, and symlinked to the repo like the others so it cannot
  drift from the tracked copy.
* The d8346672 fix (QT_QPA_PLATFORM=offscreen) worked exactly as designed: sclang
  survived the suspend perfectly. Suspend killed scsynth instead. The mitigation had
  been applied one layer too shallow — and because sclang stayed up, the unit never
  failed, so no Restart= setting of any kind would have fired.

So: Restart=on-failure (bounded by StartLimitBurst=3/5min, honouring the unit's own
"audio gear should never flap" comment) for the sclang case, plus an external
watchdog for the scsynth case. Deliberately outside the boot path — a bug in
start_and_midi.scd is gig-fatal, a bug in a watchdog is merely annoying. Folding
this into an sclang ServerQuit handler is the right long-term shape; not seven days
before a gig.

THE FIRST VERSION FLAPPED ON PLN'S DESK, and the bug is the interesting part.
The unit had `PartOf=parvagues-sc.service`, so that stopping SuperDirt would stop
the watchdog. But PartOf propagates RESTARTS. Three lines, 90ms apart:

    11:35:46.265  sc-watchdog[1102323]: scsynth GONE — restarting
    11:35:46.321  systemd: parvagues-sc.service: Consumed 1min CPU time
    11:35:46.354  sc-watchdog[1123545]: watching scsynth        <- NEW PID

The watchdog killed itself issuing the restart, Restart=always revived it, and its
in-memory rate-limit counter reset to zero. MAX_RESTARTS was unreachable BY
CONSTRUCTION: the state enforcing the limit was destroyed by the action it was
limiting. It would not have stopped at four notifications; it would have gone
forever. A supervisor must outlive its supervisee.

Fixed three ways, defence in depth:
  - no PartOf on the watchdog unit (the actual fix)
  - restart log persisted to $XDG_RUNTIME_DIR, so being killed cannot launder it
  - await the server's return (up to 100s > TimeoutStopSec=90s) instead of sleeping
    a guessed 25s and judging a boot that was still in flight

Also: every notification was urgency=critical, which Plasma treats as never-expire,
so recovery buried the screen in sticky popups. Stickiness is now the signal —
normal+timeout for "handled itself", critical+sticky only for "this needs your
hands".

Validation. tools/tests/test-sc-watchdog.sh runs the real script against a FAKE unit
holding a FAKE child server, reproducing the green-unit-no-sound topology exactly.
7/7 pass in ~90s and scsynth's PID is unchanged across the run — it touches no real
audio. It pins the flap directly: "watchdog survived issuing the restart", and the
log shows (0 prior) -> (1 prior) -> (2 prior) -> GIVING UP. Plus a 25s --dry-run
against the live unit: zero false positives on a healthy rig.

That testability is the real lesson. v1 could only be exercised by killing the
actual audio server five days before a gig, so its flap was found by a human hearing
four notifications instead of by a test. UNIT and PROC are now injectable.

The watchdog is committed but NOT enabled: it needs one real-rig recovery test
before it earns a place in the boot set.

parvagues-sc.service

@@ -22,6 +22,10 @@
 Description=ParVagues SuperDirt (sclang start_and_midi.scd, headless)
 After=pipewire.service
 Wants=pipewire.service
+# Bound the restart loop (see Restart= below): three tries in five minutes, then stay
+# down and stay loud. This is what makes recovery safe without flapping.
+StartLimitIntervalSec=300
+StartLimitBurst=3
 
 [Service]
 Type=simple
@@ -36,7 +40,17 @@ ExecStart=/usr/bin/sclang start_and_midi.scd
 # Deliberately NOT Restart=always: if the boot script fails, a restart loop
 # would spam the MIDI graph and hide the failure. Fail loudly, restart on
 # purpose. Audio gear should never flap.
-Restart=no
+#
+# 2026-08-01 — that reasoning still holds, so this is on-failure (not always) and
+# bounded by StartLimitBurst above: it recovers from a CRASH, and a boot script that
+# is genuinely broken still fails loudly after three tries instead of flapping.
+#
+# Note what this does NOT fix. On 2026-08-01 suspend killed *scsynth* while sclang
+# stayed perfectly alive, so the unit never failed and no Restart= setting of any
+# kind would have fired. Recovery from that has to watch the audio server itself:
+# parvagues-sc-watchdog.service, tools/sc-watchdog.sh.
+Restart=on-failure
+RestartSec=3
 # Give the post window to the journal, tagged so it is easy to filter.
 StandardOutput=journal
 StandardError=journal

tools/parvagues-sc-watchdog.service

+[Unit]
+# The recovery half of parvagues-sc.service.
+#
+# parvagues-sc's MainPID is sclang; scsynth is its child. systemd therefore reports
+# the unit `active (running)` while the audio server is dead — which is exactly what
+# happened on 2026-08-01 (suspend killed scsynth at 10:48, sclang logged it, nothing
+# acted, 63 Bridge warnings went to a journal nobody reads).
+#
+# This unit is deliberately SEPARATE from parvagues-sc rather than folded into its
+# ExecStart: at J-7 from a gig, a bug in the boot path is fatal and a bug out here is
+# merely noisy. Post-gig this should become an sclang-side ServerQuit handler (#115).
+Description=ParVagues SuperDirt watchdog (restarts SC when scsynth dies)
+After=parvagues-sc.service
+#
+# NO PartOf=/BindsTo= HERE, AND THAT IS LOAD-BEARING. The first version had
+# `PartOf=parvagues-sc.service`, reasoning that stopping SuperDirt should stop the
+# watchdog. But PartOf propagates RESTARTS too — so the instant the watchdog issued
+# `systemctl restart parvagues-sc`, systemd killed the watchdog, Restart=always
+# revived it, and its in-memory rate-limit counter reset to zero. Measured
+# 2026-08-01, three lines 90ms apart:
+#
+#   11:35:46.265  sc-watchdog[1102323]: scsynth GONE — restarting
+#   11:35:46.321  systemd: parvagues-sc.service: Consumed 1min CPU time
+#   11:35:46.354  sc-watchdog[1123545]: watching scsynth        <- NEW PID
+#
+# MAX_RESTARTS was therefore unreachable and the loop was unbounded. A supervisor
+# must OUTLIVE the thing it supervises. Respecting a deliberate stop is handled in
+# the script instead, by checking `systemctl is-active` before ever acting.
+
+[Service]
+Type=simple
+ExecStart=/home/pln/Work/Sound/Tidal/tools/sc-watchdog.sh
+# The watchdog itself may restart freely — it makes no sound and holds no ports.
+Restart=always
+RestartSec=5
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=sc-watchdog
+
+[Install]
+WantedBy=default.target

tools/sc-watchdog.sh

+#!/usr/bin/env bash
+# sc-watchdog — restart SuperDirt when the AUDIO SERVER dies, not when sclang does.
+#
+#     tools/sc-watchdog.sh              # supervise (what the unit runs)
+#     tools/sc-watchdog.sh --dry-run    # observe + log + notify, NEVER restart
+#
+# Why this exists (2026-08-01, OPAL J-7). The rig went silent overnight and nothing
+# recovered it. The chain, from the journal:
+#
+#     10:37:36  PM: suspend entry (s2idle)
+#     10:48:38  systemd-coredump: scsynth terminated abnormally
+#     10:48:38  sclang: "Server 'localhost' exited with exit code 0."
+#     11:18:47  Bridge: "scsynth process not found"   x63, journal only
+#
+# scsynth is a CHILD of sclang, and systemd's MainPID is sclang. So the audio server
+# can die while the unit reports `active (running)` — a green unit is not sound. The
+# d834667 fix (QT_QPA_PLATFORM=offscreen) successfully stopped suspend from killing
+# *sclang*; suspend then killed *scsynth* instead. The mitigation was one layer too
+# shallow, and `Restart=on-failure` cannot help because the unit never fails.
+#
+# This watchdog closes that gap from OUTSIDE the boot path. It deliberately does not
+# touch start_and_midi.scd or the unit's ExecStart: a bug in those is gig-fatal, a bug
+# in here is merely annoying. Folding it into an sclang-side ServerQuit handler is the
+# correct long-term shape — POST-GIG (#115).
+#
+# ---------------------------------------------------------------------------
+# THE FIRST VERSION FLAPPED. What it got wrong, so it is not repeated:
+#
+#  1. The unit had `PartOf=parvagues-sc.service`. PartOf propagates RESTARTS, so
+#     issuing the restart killed the watchdog; Restart=always revived it with a
+#     ZEROED rate-limit counter. The limiter could never trigger — the state that
+#     enforced it was destroyed by the action it was limiting. A supervisor must
+#     outlive its supervisee. Fixed in the unit (no PartOf) AND here, by persisting
+#     the restart log to a file so it survives being killed anyway.
+#  2. It slept a fixed 25s after restarting and then judged the outcome. But
+#     TimeoutStopSec is 90s, and a SuperDirt boot recompiles the class library and
+#     re-registers sample banks. Judging too early meant "still absent" -> restart
+#     again, killing the boot in flight. Now it WAITS for scsynth to appear, up to
+#     BOOT_WAIT_SECS, and only then decides.
+#  3. It was never run in a mode that could be tested without silencing the rig.
+#     Hence --dry-run. Exercise that first, always.
+#
+# It also honours the reason parvagues-sc.service carries its "audio gear should never
+# flap" comment: MAX_RESTARTS in a rolling window, then give up LOUDLY and stay down,
+# because a server that cannot boot needs eyes, not another try.
+
+set -uo pipefail
+
+# Injectable so the logic can be exercised against a FAKE unit + FAKE server, with
+# zero risk to the rig. tools/tests/test-sc-watchdog.sh does exactly that — the first
+# version of this script was only ever testable by killing the real audio server,
+# which is why its flap was discovered on PLN's desk instead of in a test.
+UNIT="${SCWD_UNIT:-parvagues-sc.service}"
+PROC="${SCWD_PROC:-scsynth}"
+POLL_SECS=2            # how often we look
+MISSES_TO_ACT=3        # ~6s absent before we believe it
+BOOT_WAIT_SECS=100     # > TimeoutStopSec (90s); a restart may legitimately take this
+MAX_RESTARTS=3         # per WINDOW_SECS, then stop trying and stay loud
+WINDOW_SECS=600
+
+# Rate-limit state OUTSIDE the process, so a watchdog restart cannot launder it.
+# XDG_RUNTIME_DIR is cleared at logout, which is the right lifetime.
+STATE="${SCWD_STATE:-${XDG_RUNTIME_DIR:-/tmp}/parvagues-sc-watchdog.restarts}"
+
+DRY_RUN=0
+[ "${1:-}" = "--dry-run" ] && DRY_RUN=1
+
+# stderr: unbuffered, so a crash cannot swallow the last thing we said.
+log() { printf '%s %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
+
+# The mouth this rig was missing. Never fatal: a headless session has no notify
+# daemon, and that must not stop the recovery.
+#
+# notify <urgency> <timeout-ms> <title> <body>
+#
+# STICKINESS IS THE SIGNAL, and it is chosen per message:
+#   normal   + timeout  -> "this happened and is already handled", fades on its own
+#   critical + 0        -> "this needs your hands", stays until dismissed
+# Plasma (and most daemons) IGNORE the timeout for urgency=critical — that is the
+# documented behaviour, not a bug. The first version shouted `critical` for every
+# routine event, so PLN got a wall of notifications that would not go away while the
+# rig was busy recovering itself. If everything is sticky, nothing is urgent.
+notify() {
+  local urgency="$1" timeout="$2" title="$3" body="$4"
+  command -v notify-send >/dev/null 2>&1 || return 0
+  notify-send -a "ParVagues" -u "$urgency" -t "$timeout" "$title" "$body" 2>/dev/null || true
+}
+
+# Restart timestamps within the rolling window, persisted across our own death.
+recent_restarts() {
+  local now t keep=()
+  now=$(date +%s)
+  if [ -f "$STATE" ]; then
+    while read -r t; do
+      [ -n "$t" ] && (( now - t < WINDOW_SECS )) && keep+=("$t")
+    done < "$STATE"
+  fi
+  printf '%s\n' ${keep[@]+"${keep[@]}"} > "$STATE"
+  echo "${#keep[@]}"
+}
+
+record_restart() { date +%s >> "$STATE"; }
+
+# Wait for scsynth to come back, however long the unit legitimately takes.
+# Returns 0 as soon as it appears.
+await_scsynth() {
+  local waited=0
+  while (( waited < BOOT_WAIT_SECS )); do
+    sleep "$POLL_SECS"; waited=$(( waited + POLL_SECS ))
+    if pgrep -x "$PROC" >/dev/null 2>&1; then
+      log "$PROC up after ${waited}s"; return 0
+    fi
+  done
+  return 1
+}
+
+log "sc-watchdog: watching $PROC (poll ${POLL_SECS}s, act after ${MISSES_TO_ACT} misses)$( ((DRY_RUN)) && echo '  [DRY RUN — will not restart]')"
+
+misses=0
+gave_up=0
+
+while true; do
+  sleep "$POLL_SECS"
+
+  # Only our business while the unit is meant to be up. If PLN stopped SuperDirt on
+  # purpose, silence is the correct state and we must not resurrect it.
+  if ! systemctl --user is-active --quiet "$UNIT"; then
+    misses=0; gave_up=0; continue
+  fi
+
+  if pgrep -x "$PROC" >/dev/null 2>&1; then
+    (( misses > 0 )) && log "$PROC back after ${misses} miss(es)"
+    misses=0; gave_up=0; continue
+  fi
+
+  misses=$(( misses + 1 ))
+  (( misses < MISSES_TO_ACT )) && continue
+  (( gave_up )) && continue
+
+  if (( DRY_RUN )); then
+    log "[dry-run] $PROC GONE (${misses} polls) — WOULD restart ${UNIT}"
+    notify normal 6000 "Audio server died (dry run)" "scsynth is gone. Not restarting: --dry-run."
+    gave_up=1     # say it once, then stay quiet
+    continue
+  fi
+
+  n=$(recent_restarts)
+  if (( n >= MAX_RESTARTS )); then
+    gave_up=1
+    log "GIVING UP: ${n} restarts in ${WINDOW_SECS}s and $PROC will not stay up."
+    notify critical 0 "SuperDirt will not start" \
+      "${n} restarts in $((WINDOW_SECS/60)) min. Giving up — this needs your eyes.
+journalctl --user -u ${UNIT} -n 50"
+    continue
+  fi
+
+  log "$PROC GONE (${misses} polls) while ${UNIT} is active — restarting (${n} prior in window)"
+  notify normal 6000 "Audio server died" "scsynth is gone. Restarting SuperDirt…"
+  record_restart
+
+  # Restart, then WAIT for the result rather than guessing how long it takes.
+  systemctl --user restart "$UNIT" || log "systemctl restart returned non-zero"
+  if await_scsynth; then
+    log "recovered"
+    notify normal 5000 "Audio server recovered" "SuperDirt is back. Samples reload on demand."
+  else
+    log "$PROC did not appear within ${BOOT_WAIT_SECS}s"
+    notify critical 0 "SuperDirt did not come back" \
+      "No scsynth ${BOOT_WAIT_SECS}s after restart. journalctl --user -u ${UNIT} -n 50"
+  fi
+  misses=0
+done

tools/tests/test-sc-watchdog.sh

+#!/usr/bin/env bash
+# test-sc-watchdog — exercise tools/sc-watchdog.sh against a FAKE unit and a FAKE
+# audio server. Touches nothing real: no scsynth, no sclang, no sound, no MIDI.
+#
+# It exists because the first version of the watchdog was only testable by killing
+# the real audio server on PLN's desk, five days before a gig. It flapped, and the
+# flap was discovered by a human hearing four notifications rather than by a test.
+#
+# The fake reproduces the exact topology that made the bug possible:
+#   fake-lang  (the unit's MainPID, stays alive)
+#     └── fake-server  (the child that actually dies)
+# — which is why systemd reports the unit healthy while the "audio" is gone.
+#
+#   tools/tests/test-sc-watchdog.sh        # ~90s
+
+set -uo pipefail
+cd "$(dirname "$0")/../.." || exit 1
+
+TMP=$(mktemp -d); trap 'cleanup' EXIT
+UNIT="scwd-test-$$.service"
+PROC="scwdfake$$"
+STATE="$TMP/restarts"
+WD_PID=""
+
+PASS=0; FAIL=0
+ok()   { printf '  \033[32mok\033[0m   %s\n' "$1"; PASS=$((PASS+1)); }
+bad()  { printf '  \033[31mFAIL\033[0m %s\n' "$1"; FAIL=$((FAIL+1)); }
+
+cleanup() {
+  [ -n "$WD_PID" ] && kill "$WD_PID" 2>/dev/null
+  systemctl --user stop "$UNIT" 2>/dev/null
+  systemctl --user reset-failed "$UNIT" 2>/dev/null
+  pkill -x "$PROC" 2>/dev/null
+  rm -rf "$TMP"
+}
+
+# --- the fake server: a distinctly-named binary so pgrep -x can find it ----------
+cp "$(command -v sleep)" "$TMP/$PROC"
+
+# --- the fake sclang: spawns the server as a CHILD and outlives it ---------------
+cat > "$TMP/fake-lang.sh" <<EOF
+#!/usr/bin/env bash
+"$TMP/$PROC" infinity &
+# Deliberately does NOT exit when the child dies — this is the whole point.
+while true; do sleep 1; done
+EOF
+chmod +x "$TMP/fake-lang.sh"
+
+echo "test-sc-watchdog: fake unit=$UNIT fake server=$PROC"
+
+systemd-run --user --unit="$UNIT" --quiet \
+  --property=Restart=no --property=Type=simple \
+  "$TMP/fake-lang.sh" || { echo "could not create transient unit"; exit 1; }
+sleep 2
+
+pgrep -x "$PROC" >/dev/null && ok "fake server running under fake unit" \
+                            || bad "fake server did not start"
+
+# --- 1. THE CORE CLAIM: unit stays 'active' while its audio child is dead --------
+pkill -x "$PROC"; sleep 1
+if systemctl --user is-active --quiet "$UNIT" && ! pgrep -x "$PROC" >/dev/null; then
+  ok "unit reports active while the server is DEAD (the bug this all exists for)"
+else
+  bad "could not reproduce the green-unit-no-sound state"
+fi
+
+# --- 2. the watchdog notices and restarts ---------------------------------------
+SCWD_UNIT="$UNIT" SCWD_PROC="$PROC" SCWD_STATE="$STATE" \
+  ./tools/sc-watchdog.sh >"$TMP/wd.log" 2>&1 &
+WD_PID=$!
+
+for i in $(seq 1 20); do sleep 2; pgrep -x "$PROC" >/dev/null && break; done
+if pgrep -x "$PROC" >/dev/null; then ok "watchdog restarted the unit; server is back"
+else bad "watchdog did not recover the server"; fi
+
+# --- 3. THE REGRESSION: the watchdog must SURVIVE the restart it issues ----------
+# The first version had PartOf= in its unit, so restarting the supervisee killed the
+# supervisor and zeroed its rate-limit counter, making MAX_RESTARTS unreachable.
+if kill -0 "$WD_PID" 2>/dev/null; then
+  ok "watchdog survived issuing the restart (no PartOf self-kill)"
+else
+  bad "watchdog DIED issuing the restart — the PartOf flap bug is back"
+fi
+
+# --- 4. the rate limiter actually bites -----------------------------------------
+# Kill repeatedly; after MAX_RESTARTS (3) in the window it must give up and STOP
+# restarting, rather than looping forever.
+for i in 1 2 3 4; do pkill -x "$PROC" 2>/dev/null; sleep 12; done
+sleep 5
+n=$(grep -c 'restarting' "$TMP/wd.log" 2>/dev/null || echo 0)
+if [ "$n" -le 3 ]; then ok "rate limiter held: $n restart(s), max 3"
+else bad "rate limiter did NOT hold: $n restarts (unbounded flap)"; fi
+
+if grep -q 'GIVING UP' "$TMP/wd.log" 2>/dev/null; then
+  ok "gave up loudly instead of flapping"
+else
+  bad "never logged GIVING UP (expected after exceeding the limit)"
+fi
+
+# --- 5. a deliberate stop must NOT be resurrected --------------------------------
+systemctl --user stop "$UNIT" 2>/dev/null; sleep 8
+if ! systemctl --user is-active --quiet "$UNIT"; then
+  ok "respected a deliberate stop (did not resurrect it)"
+else
+  bad "resurrected a unit PLN stopped on purpose"
+fi
+
+echo
+echo "--- watchdog log ---"; sed 's/^/  /' "$TMP/wd.log"
+echo
+printf 'test-sc-watchdog: %d passed, %d failed\n' "$PASS" "$FAIL"
+[ "$FAIL" -eq 0 ]