-
feat(rig): the audio server can die alone — supervise scsynth, not sclang (#115) · b978a2fa
The rig went silent overnight and NOTHING recovered it. Seven layers watched it happen. From the journal: 10:37:36 PM: suspend entry (s2idle) 10:48:38 systemd-coredump: scsynth terminated abnormally 10:48:38 sclang: "Server 'localhost' exited with exit code 0." 11:18:47 Bridge: "scsynth process not found" x63, to the journal only scsynth is a CHILD of sclang and systemd's MainPID is sclang, so the unit reported `active (running)` for two days while there was no audio server at all. A green unit is not sound. Two findings behind it: * `parvagues-sc.service` was DISABLED. Six user units start with the session — gig-log, bridge, perf-tray, tidal-ardour-autoroute, lcxl-leds-watch, midi-autoconnect — and the only one that produces audio was not among them. It had been alive since Jul 28 purely as a manual leftover, which is why its death was permanent. Now enabled, and symlinked to the repo like the others so it cannot drift from the tracked copy. * The d8346672 fix (QT_QPA_PLATFORM=offscreen) worked exactly as designed: sclang survived the suspend perfectly. Suspend killed scsynth instead. The mitigation had been applied one layer too shallow — and because sclang stayed up, the unit never failed, so no Restart= setting of any kind would have fired. So: Restart=on-failure (bounded by StartLimitBurst=3/5min, honouring the unit's own "audio gear should never flap" comment) for the sclang case, plus an external watchdog for the scsynth case. Deliberately outside the boot path — a bug in start_and_midi.scd is gig-fatal, a bug in a watchdog is merely annoying. Folding this into an sclang ServerQuit handler is the right long-term shape; not seven days before a gig. THE FIRST VERSION FLAPPED ON PLN'S DESK, and the bug is the interesting part. The unit had `PartOf=parvagues-sc.service`, so that stopping SuperDirt would stop the watchdog. But PartOf propagates RESTARTS. Three lines, 90ms apart: 11:35:46.265 sc-watchdog[1102323]: scsynth GONE — restarting 11:35:46.321 systemd: parvagues-sc.service: Consumed 1min CPU time 11:35:46.354 sc-watchdog[1123545]: watching scsynth <- NEW PID The watchdog killed itself issuing the restart, Restart=always revived it, and its in-memory rate-limit counter reset to zero. MAX_RESTARTS was unreachable BY CONSTRUCTION: the state enforcing the limit was destroyed by the action it was limiting. It would not have stopped at four notifications; it would have gone forever. A supervisor must outlive its supervisee. Fixed three ways, defence in depth: - no PartOf on the watchdog unit (the actual fix) - restart log persisted to $XDG_RUNTIME_DIR, so being killed cannot launder it - await the server's return (up to 100s > TimeoutStopSec=90s) instead of sleeping a guessed 25s and judging a boot that was still in flight Also: every notification was urgency=critical, which Plasma treats as never-expire, so recovery buried the screen in sticky popups. Stickiness is now the signal — normal+timeout for "handled itself", critical+sticky only for "this needs your hands". Validation. tools/tests/test-sc-watchdog.sh runs the real script against a FAKE unit holding a FAKE child server, reproducing the green-unit-no-sound topology exactly. 7/7 pass in ~90s and scsynth's PID is unchanged across the run — it touches no real audio. It pins the flap directly: "watchdog survived issuing the restart", and the log shows (0 prior) -> (1 prior) -> (2 prior) -> GIVING UP. Plus a 25s --dry-run against the live unit: zero false positives on a healthy rig. That testability is the real lesson. v1 could only be exercised by killing the actual audio server five days before a gig, so its flap was found by a human hearing four notifications instead of by a test. UNIT and PROC are now injectable. The watchdog is committed but NOT enabled: it needs one real-rig recovery test before it earns a place in the boot set.PLN (Algolia) authoredb978a2fa
×