The rig went silent overnight and NOTHING recovered it. Seven layers watched it
happen. From the journal:
10:37:36 PM: suspend entry (s2idle)
10:48:38 systemd-coredump: scsynth terminated abnormally
10:48:38 sclang: "Server 'localhost' exited with exit code 0."
11:18:47 Bridge: "scsynth process not found" x63, to the journal only
scsynth is a CHILD of sclang and systemd's MainPID is sclang, so the unit reported
`active (running)` for two days while there was no audio server at all. A green unit
is not sound.
Two findings behind it:
* `parvagues-sc.service` was DISABLED. Six user units start with the session —
gig-log, bridge, perf-tray, tidal-ardour-autoroute, lcxl-leds-watch,
midi-autoconnect — and the only one that produces audio was not among them. It had
been alive since Jul 28 purely as a manual leftover, which is why its death was
permanent. Now enabled, and symlinked to the repo like the others so it cannot
drift from the tracked copy.
* The d8346672 fix (QT_QPA_PLATFORM=offscreen) worked exactly as designed: sclang
survived the suspend perfectly. Suspend killed scsynth instead. The mitigation had
been applied one layer too shallow — and because sclang stayed up, the unit never
failed, so no Restart= setting of any kind would have fired.
So: Restart=on-failure (bounded by StartLimitBurst=3/5min, honouring the unit's own
"audio gear should never flap" comment) for the sclang case, plus an external
watchdog for the scsynth case. Deliberately outside the boot path — a bug in
start_and_midi.scd is gig-fatal, a bug in a watchdog is merely annoying. Folding
this into an sclang ServerQuit handler is the right long-term shape; not seven days
before a gig.
THE FIRST VERSION FLAPPED ON PLN'S DESK, and the bug is the interesting part.
The unit had `PartOf=parvagues-sc.service`, so that stopping SuperDirt would stop
the watchdog. But PartOf propagates RESTARTS. Three lines, 90ms apart:
11:35:46.265 sc-watchdog[1102323]: scsynth GONE — restarting
11:35:46.321 systemd: parvagues-sc.service: Consumed 1min CPU time
11:35:46.354 sc-watchdog[1123545]: watching scsynth <- NEW PID
The watchdog killed itself issuing the restart, Restart=always revived it, and its
in-memory rate-limit counter reset to zero. MAX_RESTARTS was unreachable BY
CONSTRUCTION: the state enforcing the limit was destroyed by the action it was
limiting. It would not have stopped at four notifications; it would have gone
forever. A supervisor must outlive its supervisee.
Fixed three ways, defence in depth:
- no PartOf on the watchdog unit (the actual fix)
- restart log persisted to $XDG_RUNTIME_DIR, so being killed cannot launder it
- await the server's return (up to 100s > TimeoutStopSec=90s) instead of sleeping
a guessed 25s and judging a boot that was still in flight
Also: every notification was urgency=critical, which Plasma treats as never-expire,
so recovery buried the screen in sticky popups. Stickiness is now the signal —
normal+timeout for "handled itself", critical+sticky only for "this needs your
hands".
Validation. tools/tests/test-sc-watchdog.sh runs the real script against a FAKE unit
holding a FAKE child server, reproducing the green-unit-no-sound topology exactly.
7/7 pass in ~90s and scsynth's PID is unchanged across the run — it touches no real
audio. It pins the flap directly: "watchdog survived issuing the restart", and the
log shows (0 prior) -> (1 prior) -> (2 prior) -> GIVING UP. Plus a 25s --dry-run
against the live unit: zero false positives on a healthy rig.
That testability is the real lesson. v1 could only be exercised by killing the
actual audio server five days before a gig, so its flap was found by a human hearing
four notifications instead of by a test. UNIT and PROC are now injectable.
The watchdog is committed but NOT enabled: it needs one real-rig recovery test
before it earns a place in the boot set.
| Name |
Last commit
|
Last update |
|---|---|---|
| .gitea/workflows | Loading commit data... | |
| armada | Loading commit data... | |
| blocks | Loading commit data... | |
| blog | Loading commit data... | |
| copycat | Loading commit data... | |
| docs | Loading commit data... | |
| gifts | Loading commit data... | |
| lib-parvagues | Loading commit data... | |
| live | Loading commit data... | |
| output | Loading commit data... | |
| releases | Loading commit data... | |
| sandbox | Loading commit data... | |
| study | Loading commit data... | |
| test | Loading commit data... | |
| tools | Loading commit data... | |
| utils | Loading commit data... | |
| visuals/slop | Loading commit data... | |
| viz | Loading commit data... | |
| .gitignore | Loading commit data... | |
| BootTidal.hs | Loading commit data... | |
| BootTidal.hs.broken | Loading commit data... | |
| BootTidal.visuals.broken.hs | Loading commit data... | |
| CLAUDE.md | Loading commit data... | |
| FOH_MASTER.md | Loading commit data... | |
| LICENSE | Loading commit data... | |
| MORNING.md | Loading commit data... | |
| PatchBay_MultiChannel.xml | Loading commit data... | |
| PatchBay_Perf.xml | Loading commit data... | |
| PatchBay_Perf5.1.xml | Loading commit data... | |
| README.md | Loading commit data... | |
| RUNBOOK.md | Loading commit data... | |
| SHOPPING_LIST.md | Loading commit data... | |
| backlog.md | Loading commit data... | |
| bullet_train.tidal | Loading commit data... | |
| collide.sh | Loading commit data... | |
| debug.tidal | Loading commit data... | |
| didacticpatternvisualizer.pde | Loading commit data... | |
| gig-up.sh | Loading commit data... | |
| gpu-mode.sh | Loading commit data... | |
| hello.sc | Loading commit data... | |
| init_midi.sh | Loading commit data... | |
| intro.md | Loading commit data... | |
| library.txt | Loading commit data... | |
| link_jack.sh | Loading commit data... | |
| link_samples.sh | Loading commit data... | |
| link_samples_all.txt | Loading commit data... | |
| main_fairbanks.scd | Loading commit data... | |
| notes.md | Loading commit data... | |
| parvagues-sc.service | Loading commit data... | |
| perf-audio.sudoers | Loading commit data... | |
| perf-tray.py | Loading commit data... | |
| perf.sh | Loading commit data... | |
| samples.txt | Loading commit data... | |
| setlist_opal2026.txt | Loading commit data... | |
| start_and_midi.scd | Loading commit data... | |
| test.tidal | Loading commit data... |