Commit 8d307401 by PLN (Algolia)

feat(gig): gig-up --converge, and the gate finally checks its own hands (#121, #122)

--converge is the CLI big red button: bring the rig to ready, then let the existing
checks prove it. Scope is SERVICES, not apps — it starts systemd units it owns and
REPORTS Pulsar/Ardour with the command instead of launching them, because a script
that opens windows under someone is a script they stop trusting (and Ardour has a
session + crash-recovery dialog, #20). Launching apps belongs to the Bridge (#116).

Safe at any moment: every action is conditional on the thing being broken, so a
healthy rig converges to a no-op — verified, scsynth PID unchanged. It handles the
exact 2026-08-01 failure explicitly: unit ACTIVE but audio server DEAD needs a
RESTART, since `start` is a no-op on an active unit. Distinguishing those two is the
whole lesson. Preload is fixed BEFORE SuperDirt boots, or the warm would not happen
until the next restart. And it waits for scsynth to appear rather than guessing —
guessing is what made the first watchdog flap.

=== THE THIRD TIME, AND THIS TIME IT WAS US ===

gig-up's first run (1219c704) found check-boot.sh and check-tracks.sh at mode 644 —
never runnable as `tools/check-tracks.sh`. That fix turned out to be FAKE: the chmod
only ever touched the working tree. Git has recorded 100644 for both from that day to
this, so every fresh clone still got a broken pre-gig gate.

And gig-up.sh had the same bug, from birth. Committed 100644 by 1219c704 and again by
6ab09eaa. It ran only because the local working tree happened to carry the bit; a
`git checkout master` materialised it at 644 and it died with "permission denied".

That failure was nearly invisible, which is the real story: the caller filtered its
output through sed, so a run that never executed LOOKED like a clean no-op, and I
reported "no-op confirmed" about a script that had not run. A false green produced by
the absence of a program.

So the gate now checks its own hands — "tools executable", HARD, verifying BOTH the
on-disk bit and what git records. It found four real problems on its first run:
    disk=x  git=100644  check-boot.sh          <- day-one "fix" never committed
    disk=x  git=100644  check-tracks.sh        <- same
    disk=-  git=100644  sc-watchdog.sh         <- not executable AT ALL
    disk=-  git=100644  tests/test-sc-watchdog.sh
The third is the one that matters: parvagues-sc-watchdog.service had been ENABLED AT
BOOT with an ExecStart it could not execute. It was stuck `activating`, restart-
looping silently. The supervisor I added to catch silent failure was itself failing
silently, and nothing would have said so until an audio server died at a venue and
did not come back. All six now 100755 on disk and in the index; watchdog restarted
and confirmed logging.

Same family as the day-one finding: a thing verified once, in a context that no
longer holds. The new twist is that "chmod +x" is not a fix — recording it in git is.

Also honest-ing the recovery toast (#122). PLN reported needing a ctrl+enter in
Pulsar to resume after a recovery. Likely ICMP port-unreachable poisoning Tidal's
connected UDP socket (ECONNREFUSED on next send, stream stops until re-evaluated).
Until #122 tests it, the toast says "if the music does not resume on its own,
ctrl+enter in Pulsar" rather than claiming a full recovery — a rig that is alive and
silent is indistinguishable on stage from the original fault.

Validation: 16/16 green on the live rig (9 cold + 7 live). Converge no-op verified on
a healthy rig; converge ACTION verified by stopping the watchdog and watching it come
back, with scsynth untouched throughout.
parent 6ab09eaa
File mode changed from 100644 to 100755
File mode changed from 100644 to 100755
File mode changed from 100644 to 100755
...@@ -47,6 +47,7 @@ ...@@ -47,6 +47,7 @@
# tools/gig-up.sh # the cold gate — safe anywhere, ~1 min # tools/gig-up.sh # the cold gate — safe anywhere, ~1 min
# tools/gig-up.sh --audio # + check-tracks.sh (makes sound, ~10 min) # tools/gig-up.sh --audio # + check-tracks.sh (makes sound, ~10 min)
# tools/gig-up.sh --live # + assert a RIG exists, not just a set # tools/gig-up.sh --live # + assert a RIG exists, not just a set
# tools/gig-up.sh --converge # + START what is missing, then assert (implies --live)
# tools/gig-up.sh --quiet # only the verdict and any failures # tools/gig-up.sh --quiet # only the verdict and any failures
# #
# Exit 0 = GO. Non-zero = do not start; the reasons are printed first. # Exit 0 = GO. Non-zero = do not start; the reasons are printed first.
...@@ -56,10 +57,12 @@ cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" || exit 2 ...@@ -56,10 +57,12 @@ cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" || exit 2
AUDIO=0 AUDIO=0
QUIET=0 QUIET=0
LIVE=0 LIVE=0
CONVERGE=0
for a in "$@"; do for a in "$@"; do
case "$a" in case "$a" in
--audio) AUDIO=1 ;; --audio) AUDIO=1 ;;
--live) LIVE=1 ;; --live) LIVE=1 ;;
--converge) CONVERGE=1; LIVE=1 ;;
--quiet|-q) QUIET=1 ;; --quiet|-q) QUIET=1 ;;
-h|--help) sed -n '2,58p' "${BASH_SOURCE[0]}" | sed 's/^# \?//'; exit 0 ;; -h|--help) sed -n '2,58p' "${BASH_SOURCE[0]}" | sed 's/^# \?//'; exit 0 ;;
*) echo "gig-up: unknown option $a (try --help)" >&2; exit 2 ;; *) echo "gig-up: unknown option $a (try --help)" >&2; exit 2 ;;
...@@ -119,6 +122,67 @@ read -r SC ARD <<<"$(rig_state)" ...@@ -119,6 +122,67 @@ read -r SC ARD <<<"$(rig_state)"
echo echo
} }
# --- 0. converge, opt-in (--converge) ----------------------------------------
# THE BIG RED BUTTON, CLI half: bring the rig to ready, then let the checks below
# prove it. Everything after this point only OBSERVES; this is the only section that
# changes the world, which is why it is opt-in and why it is first.
#
# Scope is SERVICES, not apps. It starts systemd units it owns; it does NOT launch
# Pulsar or Ardour. Those are PLN's GUI apps with sessions and crash-recovery dialogs
# (#20), and a script that opens windows under someone is a script they stop trusting.
# They are REPORTED instead, with the command. Launching them belongs to the Bridge
# gear panel (#116).
#
# SAFE TO RUN AT ANY MOMENT, because every action is conditional on the thing being
# broken. A healthy rig converges to a no-op: nothing restarts, nothing makes a sound,
# no MIDI is sent, and CC 77-84 / CC 93 are never touched by anything here.
if (( CONVERGE )); then
(( QUIET )) || { echo " ${D}converge — starting what is missing${Z}"; }
did=0
# Preload FIRST: fixing it after SuperDirt has booted would warm nothing until the
# next restart, and the whole point is that the boot reads the set's banks off disk.
if ! tools/check-preload.sh >/dev/null 2>&1; then
tools/check-preload.sh --fix >>"$LOG" 2>&1 && { echo " ${G}+${Z} preload plan regenerated"; did=1; }
fi
# The exact failure of 2026-08-01: unit ACTIVE, audio server DEAD. `start` is a no-op
# on an active unit, so this case needs a RESTART — and distinguishing the two is the
# entire lesson (a green unit is not sound).
if ! pgrep -x scsynth >/dev/null 2>&1; then
if systemctl --user is-active --quiet parvagues-sc.service; then
echo " ${Y}+${Z} unit is active but scsynth is DEAD — restarting SuperDirt"
systemctl --user restart parvagues-sc.service >>"$LOG" 2>&1
else
echo " ${G}+${Z} starting SuperDirt"
systemctl --user start parvagues-sc.service >>"$LOG" 2>&1
fi
did=1
# Wait for the server, do not guess: a cold boot recompiles the class library and
# warms 51 banks. Judging too early is what made the first watchdog flap.
for _ in $(seq 1 50); do sleep 2; pgrep -x scsynth >/dev/null 2>&1 && break; done
fi
for u in parvagues-sc-watchdog tidal-ardour-autoroute; do
if ! systemctl --user is-active --quiet "$u.service"; then
systemctl --user start "$u.service" >>"$LOG" 2>&1 && { echo " ${G}+${Z} started $u"; did=1; }
fi
done
if command -v powerprofilesctl >/dev/null 2>&1 && ! powerprofilesctl get 2>/dev/null | grep -q performance; then
powerprofilesctl set performance >>"$LOG" 2>&1 && { echo " ${G}+${Z} perf mode on"; did=1; }
fi
# Apps: report, never launch.
pgrep -x pulsar >/dev/null 2>&1 || echo " ${Y}!${Z} Pulsar is not running — open it and boot Tidal"
pgrep -f 'ardour[0-9]*' >/dev/null 2>&1 || echo " ${Y}!${Z} Ardour is not running — open the \"Tidal Live\" session"
(( did )) || { (( QUIET )) || echo " ${D}nothing to do — already converged${Z}"; }
echo
# Re-read: converge just changed the world the banner described.
read -r SC ARD <<<"$(rig_state)"
fi
# --- 1. the helpers typecheck ------------------------------------------------ # --- 1. the helpers typecheck ------------------------------------------------
# First because one parse error in BootTidal.hs silences EVERY track at once — # First because one parse error in BootTidal.hs silences EVERY track at once —
# the worst failure this rig has, and invisible while a stale ghci still holds # the worst failure this rig has, and invisible while a stale ghci still holds
...@@ -127,6 +191,30 @@ run "boot helpers" \ ...@@ -127,6 +191,30 @@ run "boot helpers" \
"read the ghc error: tools/check-boot.sh" \ "read the ghc error: tools/check-boot.sh" \
tools/check-boot.sh tools/check-boot.sh
# --- 1a. the gate's own tools are runnable -----------------------------------
# THE THIRD TIME. gig-up's very first run (1219c70) found check-boot.sh and
# check-tracks.sh at mode 644 — never runnable as `tools/check-tracks.sh`, only ever
# working because a human typed `zsh tools/...` out of habit. On 2026-08-01 the same
# bug turned up in THIS FILE: gig-up.sh was committed 100644 by that very commit and
# by 6ab09ea, so it had never been executable in git. It ran only because the local
# working tree happened to carry the bit; a `git checkout` materialised it at 644 and
# it died with "permission denied" — and because the caller filtered the output, the
# run LOOKED like a clean no-op. A false green from a script that never executed.
#
# So the gate now checks its own hands. Cheap, and it has earned its place three times.
run "tools executable" \
"chmod +x the file(s) listed, then: git update-index --chmod=+x <file>" \
bash -c '
bad=0
for f in tools/gig-up.sh tools/check-boot.sh tools/check-tracks.sh \
tools/check-preload.sh tools/sc-watchdog.sh; do
[ -e "$f" ] || continue
[ -x "$f" ] || { echo "not executable on disk: $f"; bad=1; }
m=$(git ls-files -s "$f" 2>/dev/null | awk "{print \$1}")
[ -n "$m" ] && [ "$m" != "100755" ] && { echo "git records $m (not 100755): $f"; bad=1; }
done
exit $bad'
# --- 2. the saved Ardour faders ---------------------------------------------- # --- 2. the saved Ardour faders ----------------------------------------------
run "ardour faders" \ run "ardour faders" \
"raise it on the desk, then Ctrl+S in Ardour, then re-run. Still -inf after a save = genuinely down." \ "raise it on the desk, then Ctrl+S in Ardour, then re-run. Still -inf after a save = genuinely down." \
......
...@@ -165,10 +165,19 @@ journalctl --user -u ${UNIT} -n 50" ...@@ -165,10 +165,19 @@ journalctl --user -u ${UNIT} -n 50"
record_restart record_restart
# Restart, then WAIT for the result rather than guessing how long it takes. # Restart, then WAIT for the result rather than guessing how long it takes.
#
# UNVERIFIED (#122): PLN reported needing a ctrl+enter in Pulsar to resume playing
# after a recovery. Likely cause: while SuperDirt was down the kernel answered Tidal's
# OSC datagrams with ICMP port-unreachable, which a CONNECTED UDP socket surfaces as
# ECONNREFUSED on the next send — so the stream may stop permanently until
# re-evaluated. Until that is tested, the toast SAYS SO rather than claiming a full
# recovery: a rig that is alive and silent is indistinguishable on stage from the
# original fault.
systemctl --user restart "$UNIT" || log "systemctl restart returned non-zero" systemctl --user restart "$UNIT" || log "systemctl restart returned non-zero"
if await_scsynth; then if await_scsynth; then
log "recovered" log "recovered"
notify normal 5000 "Audio server recovered" "SuperDirt is back. Samples reload on demand." notify normal 9000 "Audio server recovered" \
"SuperDirt is back. If the music does not resume on its own, ctrl+enter in Pulsar."
else else
log "$PROC did not appear within ${BOOT_WAIT_SECS}s" log "$PROC did not appear within ${BOOT_WAIT_SECS}s"
notify critical 0 "SuperDirt did not come back" \ notify critical 0 "SuperDirt did not come back" \
......
File mode changed from 100644 to 100755
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment