The rig went silent overnight and NOTHING recovered it. Seven layers watched it
happen. From the journal:
10:37:36 PM: suspend entry (s2idle)
10:48:38 systemd-coredump: scsynth terminated abnormally
10:48:38 sclang: "Server 'localhost' exited with exit code 0."
11:18:47 Bridge: "scsynth process not found" x63, to the journal only
scsynth is a CHILD of sclang and systemd's MainPID is sclang, so the unit reported
`active (running)` for two days while there was no audio server at all. A green unit
is not sound.
Two findings behind it:
* `parvagues-sc.service` was DISABLED. Six user units start with the session —
gig-log, bridge, perf-tray, tidal-ardour-autoroute, lcxl-leds-watch,
midi-autoconnect — and the only one that produces audio was not among them. It had
been alive since Jul 28 purely as a manual leftover, which is why its death was
permanent. Now enabled, and symlinked to the repo like the others so it cannot
drift from the tracked copy.
* The d8346672 fix (QT_QPA_PLATFORM=offscreen) worked exactly as designed: sclang
survived the suspend perfectly. Suspend killed scsynth instead. The mitigation had
been applied one layer too shallow — and because sclang stayed up, the unit never
failed, so no Restart= setting of any kind would have fired.
So: Restart=on-failure (bounded by StartLimitBurst=3/5min, honouring the unit's own
"audio gear should never flap" comment) for the sclang case, plus an external
watchdog for the scsynth case. Deliberately outside the boot path — a bug in
start_and_midi.scd is gig-fatal, a bug in a watchdog is merely annoying. Folding
this into an sclang ServerQuit handler is the right long-term shape; not seven days
before a gig.
THE FIRST VERSION FLAPPED ON PLN'S DESK, and the bug is the interesting part.
The unit had `PartOf=parvagues-sc.service`, so that stopping SuperDirt would stop
the watchdog. But PartOf propagates RESTARTS. Three lines, 90ms apart:
11:35:46.265 sc-watchdog[1102323]: scsynth GONE — restarting
11:35:46.321 systemd: parvagues-sc.service: Consumed 1min CPU time
11:35:46.354 sc-watchdog[1123545]: watching scsynth <- NEW PID
The watchdog killed itself issuing the restart, Restart=always revived it, and its
in-memory rate-limit counter reset to zero. MAX_RESTARTS was unreachable BY
CONSTRUCTION: the state enforcing the limit was destroyed by the action it was
limiting. It would not have stopped at four notifications; it would have gone
forever. A supervisor must outlive its supervisee.
Fixed three ways, defence in depth:
- no PartOf on the watchdog unit (the actual fix)
- restart log persisted to $XDG_RUNTIME_DIR, so being killed cannot launder it
- await the server's return (up to 100s > TimeoutStopSec=90s) instead of sleeping
a guessed 25s and judging a boot that was still in flight
Also: every notification was urgency=critical, which Plasma treats as never-expire,
so recovery buried the screen in sticky popups. Stickiness is now the signal —
normal+timeout for "handled itself", critical+sticky only for "this needs your
hands".
Validation. tools/tests/test-sc-watchdog.sh runs the real script against a FAKE unit
holding a FAKE child server, reproducing the green-unit-no-sound topology exactly.
7/7 pass in ~90s and scsynth's PID is unchanged across the run — it touches no real
audio. It pins the flap directly: "watchdog survived issuing the restart", and the
log shows (0 prior) -> (1 prior) -> (2 prior) -> GIVING UP. Plus a 25s --dry-run
against the live unit: zero false positives on a healthy rig.
That testability is the real lesson. v1 could only be exercised by killing the
actual audio server five days before a gig, so its flap was found by a human hearing
four notifications instead of by a test. UNIT and PROC are now injectable.
The watchdog is committed but NOT enabled: it needs one real-rig recovery test
before it earns a place in the boot set.
