Skip to content

  • Projects
  • Groups
  • Snippets
  • Help
  • This project
    • Loading...
  • Sign in / Register
T
Tidal
  • Overview
    • Overview
    • Details
    • Activity
    • Cycle Analytics
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Charts
  • Issues 0
    • Issues 0
    • List
    • Board
    • Labels
    • Milestones
  • Merge Requests 0
    • Merge Requests 0
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Charts
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Charts
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • PLN
  • Tidal
  • Repository

Switch branch/tag
  • Tidal
  • parvagues-sc.service
Find file
BlameHistoryPermalink
  • PLN (Algolia)'s avatar
    feat(rig): the audio server can die alone — supervise scsynth, not sclang (#115) · b978a2fa
    The rig went silent overnight and NOTHING recovered it. Seven layers watched it
    happen. From the journal:
    
        10:37:36  PM: suspend entry (s2idle)
        10:48:38  systemd-coredump: scsynth terminated abnormally
        10:48:38  sclang: "Server 'localhost' exited with exit code 0."
        11:18:47  Bridge: "scsynth process not found"   x63, to the journal only
    
    scsynth is a CHILD of sclang and systemd's MainPID is sclang, so the unit reported
    `active (running)` for two days while there was no audio server at all. A green unit
    is not sound.
    
    Two findings behind it:
    
    * `parvagues-sc.service` was DISABLED. Six user units start with the session —
      gig-log, bridge, perf-tray, tidal-ardour-autoroute, lcxl-leds-watch,
      midi-autoconnect — and the only one that produces audio was not among them. It had
      been alive since Jul 28 purely as a manual leftover, which is why its death was
      permanent. Now enabled, and symlinked to the repo like the others so it cannot
      drift from the tracked copy.
    * The d8346672 fix (QT_QPA_PLATFORM=offscreen) worked exactly as designed: sclang
      survived the suspend perfectly. Suspend killed scsynth instead. The mitigation had
      been applied one layer too shallow — and because sclang stayed up, the unit never
      failed, so no Restart= setting of any kind would have fired.
    
    So: Restart=on-failure (bounded by StartLimitBurst=3/5min, honouring the unit's own
    "audio gear should never flap" comment) for the sclang case, plus an external
    watchdog for the scsynth case. Deliberately outside the boot path — a bug in
    start_and_midi.scd is gig-fatal, a bug in a watchdog is merely annoying. Folding
    this into an sclang ServerQuit handler is the right long-term shape; not seven days
    before a gig.
    
    THE FIRST VERSION FLAPPED ON PLN'S DESK, and the bug is the interesting part.
    The unit had `PartOf=parvagues-sc.service`, so that stopping SuperDirt would stop
    the watchdog. But PartOf propagates RESTARTS. Three lines, 90ms apart:
    
        11:35:46.265  sc-watchdog[1102323]: scsynth GONE — restarting
        11:35:46.321  systemd: parvagues-sc.service: Consumed 1min CPU time
        11:35:46.354  sc-watchdog[1123545]: watching scsynth        <- NEW PID
    
    The watchdog killed itself issuing the restart, Restart=always revived it, and its
    in-memory rate-limit counter reset to zero. MAX_RESTARTS was unreachable BY
    CONSTRUCTION: the state enforcing the limit was destroyed by the action it was
    limiting. It would not have stopped at four notifications; it would have gone
    forever. A supervisor must outlive its supervisee.
    
    Fixed three ways, defence in depth:
      - no PartOf on the watchdog unit (the actual fix)
      - restart log persisted to $XDG_RUNTIME_DIR, so being killed cannot launder it
      - await the server's return (up to 100s > TimeoutStopSec=90s) instead of sleeping
        a guessed 25s and judging a boot that was still in flight
    
    Also: every notification was urgency=critical, which Plasma treats as never-expire,
    so recovery buried the screen in sticky popups. Stickiness is now the signal —
    normal+timeout for "handled itself", critical+sticky only for "this needs your
    hands".
    
    Validation. tools/tests/test-sc-watchdog.sh runs the real script against a FAKE unit
    holding a FAKE child server, reproducing the green-unit-no-sound topology exactly.
    7/7 pass in ~90s and scsynth's PID is unchanged across the run — it touches no real
    audio. It pins the flap directly: "watchdog survived issuing the restart", and the
    log shows (0 prior) -> (1 prior) -> (2 prior) -> GIVING UP. Plus a 25s --dry-run
    against the live unit: zero false positives on a healthy rig.
    
    That testability is the real lesson. v1 could only be exercised by killing the
    actual audio server five days before a gig, so its flap was found by a human hearing
    four notifications instead of by a test. UNIT and PROC are now injectable.
    
    The watchdog is committed but NOT enabled: it needs one real-rig recovery test
    before it earns a place in the boot set.
    PLN (Algolia) authored Aug 01, 2026
    b978a2fa
parvagues-sc.service 2.7 KB
EditWeb IDE
×

Replace parvagues-sc.service

Attach a file by drag & drop or click to upload


Cancel
A new branch will be created in your fork and a new merge request will be started.