Commit db42ffce by PLN (Algolia)

fix(audio): prune the graph — 2.1M xruns came from nodes nobody used

THE J-5 DROPOUT. Sound went glitchy-then-silent mid-rehearsal, five days before
OPAL, and every obvious suspect was healthy: sclang up, scsynth up and using
6.7% CPU, the UMC sink present and RUNNING, every link in place. Diagnosing the
audio engine would have found nothing wrong with it, because nothing was.

THE BEHRINGER IS THE CANARY, NOT THE FAULT. The internal speaker played fine the
whole time and only the UMC202HD died, which looks exactly like a broken
interface. It is not: period_size 128 on the UMC against 1024 on the SOF speaker
is an 8x tighter deadline, so the UMC is the only device in the box with a
real-time deadline tight enough to miss. USB devnum stayed at 14 across samples
(not the 2026-07-26 re-enumeration bug), mixer 127/127, links intact. Rule for
next time: when only the UMC is silent, suspect the SCHEDULER.

WHAT pw-top FOUND. Xruns per node:

    sof_sdw.HiFi__Headset__source   1,421,369   nothing used it
    sof_sdw.HiFi__HDMI1__sink         704,081   2.8ms BUSY/cycle, no client
    UMC202HD Line sink                 32,338   the victim
    alsa_input.hw_U192k_0              14,338   W/Q 1.24 = waits > a full quantum
    SuperCollider                       1,119
    ardour                                422

Two nodes NOBODY USES produced 2.1 million of them. HDMI1 appeared when the
screen was mirrored for the WYSIWYG show.

THE ROOT CAUSE WAS A FIX. 52-no-suspend.conf matched ~alsa_output.* and
~alsa_input.* — every node in the graph. It was written for a real reason (a
woken device eats the first note's attack) but applied graph-wide it held six SOF
sinks and a broken mic source permanently open INSIDE the real-time graph.
Never-suspend is precisely what it asked for. The generalisable lesson, and it is
the one worth keeping: a rule scoped to one device but applied to the whole graph
stops being a fix and becomes a load. Suspension is not the enemy — suspension of
the device you are playing through is. Now narrowed to the UMC plus the internal
Speaker, the latter kept deliberately because it is the fallback that proved the
machine could still make sound during the incident.

AND THE LAPTOP MICROPHONE WAS IN THE d1 RECORDING. pw-link showed
Headset__source:capture_FL -> ardour:Tidal 01/audio_in 1. Ardour auto-connects
physical inputs, so the internal mic was summed into the input of the track that
records d1, alongside SuperCollider:out_1 — invisible in Ardour's GUI and in the
saved session file. 53-rig-graph.conf disables the source outright rather than
trusting nobody to re-arm that track. Take95 was recorded with it connected;
auditing that is #132.

RESULT, measured. Load 7.17 -> 4.80. UMC hw_ptr went from restarting once a
second — never past ~3000 frames, deltas going NEGATIVE — to a steady 48,145
frames/sec across six consecutive samples with zero restarts.

VERSIONED, NOT COPIED. The three wireplumber files now live in
config/wireplumber/ and are SYMLINKED from ~/.config. One source of truth: a copy
would have become the fourth thing in this project to drift from its generated
twin. check-audio-graph.sh also fails on a dangling symlink, because a broken one
is silent until the next wireplumber restart — the worst possible moment to find
out.

NEW GATE: tools/check-audio-graph.sh, wired into gig-up as a HARD check. It
asserts what was actually wrong, in the order it bites: no non-SuperCollider
source on any Tidal input (recording integrity first — a dropout you hear and
redo, a stem with room noise in it you ship), Master really reaching the UMC,
the interface actually clocking, and the prune still in effect. check-mix reads
the SAVED session; this reads the LIVE graph, and the saved file has been wrong
before (#62).

TWO METHODOLOGY RULES BAKED INTO IT, both learned that day:

  - NEVER SCAN THE JOURNAL. `journalctl --since` on a long-uptime box reads the
    whole journal and burned a full core twice WHILE PLN WAS LOSING AUDIO — the
    diagnostic making the fault worse. Everything is /proc, pw-link, bounded
    pw-top.
  - `state: RUNNING` PROVES NOTHING. It read RUNNING while the device was dying.
    Only the hardware pointer, sampled repeatedly, is honest — and take four
    samples, not two: my first two-sample read gave a confidently wrong "still
    stalling" verdict because a ring-buffer wrap looks identical to a stall.

Mutation-checked: the recording-integrity awk was run against the real 2026-08-03
pw-link dump WITH the mic present (it fires) and against a clean one (silent, no
false positive). A check that has never seen the bug it was written for is a
guess.
parent 05df7fee
# Disable the XPS internal digital-mic capture node on the SOF/SoundWire codec.
#
# WHY (2026-07-26): opening this node hard-hung the SOF DSP. The kernel logged
# sof-audio-pci-intel-cnl 0000:00:1f.3: ipc tx timed out for 0x30100000
# ASoC error (-110) ... Failed to setup widget PIPELINE.5.ALH770.IN
# on card3 pcm4c ("Microphone"). WirePlumber then blocked forever in an
# uninterruptible kernel wait (D state, ipc3_tx_msg_unlocked) — unkillable even
# by SIGKILL, which took the whole audio stack down and killed scsynth mid-set.
# Only a reboot cleared it. That is an unacceptable failure mode during a gig.
#
# SCOPE: capture only, and only the internal DMIC array. SOF *playback*
# (speakers, 3.5mm jack, HDMI) is untouched, so Firefox/system sounds still
# work if the default sink is moved back off the USB interface. The headset-jack
# mic source is also left alone — it uses a different pipeline and was not
# implicated.
#
# The real monitoring path is now the Behringer UMC202HD (separate USB driver,
# unaffected throughout the hang). See task #39 and memory
# reference_umc202hd_interface.
#
# To re-enable: delete this file and `systemctl --user restart wireplumber`.
monitor.alsa.rules = [
{
matches = [
{ node.name = "alsa_input.pci-0000_00_1f.3-platform-sof_sdw.HiFi__Mic__source" }
]
actions = {
update-props = {
node.disabled = true
}
}
}
]
# Never park an audio device in SUSPENDED.
#
# WHY (2026-07-26): WirePlumber suspends idle ALSA nodes to save power. Waking
# the device takes long enough that the attack of the first sound after silence
# is eaten — PLN heard the volume-feedback beep arrive faint and clipped
# ("muffled, like at 0.2"), while the on-screen volume UI reacted instantly.
#
# Harmless on a laptop; unacceptable on stage. The same mechanism would swallow
# the first hit of a track coming out of a quiet passage or a break — exactly
# the moments that have to land. Livecoding has long gaps where a stream is
# silent but must be ready to fire on the next cycle.
#
# session.suspend-timeout-seconds = 0 disables suspension entirely, so the
# device holds the stream open. Costs a little idle power; buys deterministic
# first-note timing.
#
# NARROWED 2026-08-03 — it used to match `~alsa_output.*` and `~alsa_input.*`,
# i.e. EVERY node in the graph.
#
# The reasoning at the time was "so it still holds if the default sink moves".
# The cost, measured during the J-5 dropout: six SOF sinks and a broken headset
# mic source were all held permanently open and inside the real-time graph,
# because never suspending is exactly what this file asks for. `pw-top` found
# HDMI1 burning 2.8ms of processing per cycle with no client attached, and the
# headset source at 1.4 MILLION xruns. A fix written to protect the UMC's first
# note was keeping every unused device awake and competing with it.
#
# THE LESSON, and it generalises: a rule written for one device and applied to
# the whole graph stops being a fix and becomes a load. Suspension is not the
# enemy — suspension OF THE DEVICE WE ARE PLAYING THROUGH is.
#
# Now scoped to the devices the rig actually performs on:
# - the UMC202HD (the real output, and the one whose first-note attack was
# being eaten in the first place)
# - the internal Speaker, kept because it is the fallback that proves the
# machine can still make sound when the UMC is in trouble
# Everything else may park, which is what we want: a parked device costs
# nothing and is not in the graph.
#
# Matched by NAME, not by card index — the UMC has been card 0 and card 1
# across replugs (see memory reference_umc202hd_interface).
#
# To revert: delete this file and `systemctl --user restart wireplumber`.
monitor.alsa.rules = [
{
matches = [
{ node.name = "~alsa_output.usb-BEHRINGER_UMC202HD.*" }
{ node.name = "~alsa_input.*U192k.*" }
{ node.name = "~alsa_input.usb-BEHRINGER_UMC202HD.*" }
{ node.name = "alsa_output.pci-0000_00_1f.3-platform-sof_sdw.HiFi__Speaker__sink" }
]
actions = {
update-props = {
session.suspend-timeout-seconds = 0
}
}
}
]
# Prune the audio graph down to what the RIG actually uses.
#
# WHY (2026-08-03): sound went glitchy-then-silent mid-rehearsal, J-5 before
# OPAL. Nothing had crashed — sclang, scsynth, the UMC sink and every link were
# healthy, and scsynth was using 6.7% CPU. The machine simply could not meet the
# audio deadline, and the UMC202HD (period_size 128) is the only device in the
# box with a deadline tight enough to notice. The internal speaker
# (period_size 1024) played fine throughout, which is what made it look like a
# Behringer fault. It was not: USB devnum stayed at 14, mixer at 127/127, links
# intact.
#
# `pw-top` named the actual cost. Xruns per node:
#
# sof_sdw.HiFi__Headset__source 1,421,369 <- nothing uses it
# sof_sdw.HiFi__HDMI1__sink 704,081 <- 2.8ms BUSY per cycle
# UMC202HD Line sink 32,338 <- the victim
# alsa_input.hw_U192k_0 14,338 <- W/Q 1.24: waits > a full quantum
# SuperCollider 1,119
# ardour 422
#
# Two nodes NOBODY USES accounted for 2.1 million of them. HDMI1 appeared when
# PLN set up screen mirroring for the WYSIWYG show, and it burns 2.8ms of
# processing every cycle for an output with no client attached. Earlier the same
# device threw a `snd_pcm_start: File descriptor in bad state` retry storm.
#
# AND THE HEADSET MIC WAS IN THE RECORDING PATH. `pw-link` showed:
# Headset__source:capture_FL -> ardour:Tidal 01/audio_in 1
# Ardour auto-connects physical inputs, so the laptop's microphone was summed
# into the input of the track that records d1, alongside SuperCollider:out_1.
# Disabling the source removes it structurally rather than relying on nobody
# ever re-arming that track.
#
# WHY THIS GOT BAD NOW: 52-no-suspend.conf disables suspension for ALL alsa
# nodes. That was written for a real reason — a woken device eats the attack of
# the first note — but applied graph-wide it holds six SOF sinks and a broken
# mic source permanently open and in the real-time graph. The fix that helped
# the UMC was amplifying everything else. That file is narrowed alongside this
# one.
#
# SCOPE / WHAT IS DELIBERATELY KEPT:
# - Speaker and Headphones sinks stay ENABLED. They are the fallback that
# proved the machine could still play audio at all during this incident,
# and that diagnostic value is worth their (idle, suspendable) cost.
# - The UMC's own capture stays enabled — the mic inputs are real I/O.
# - Nothing here touches the UMC playback path.
#
# AUDIO OVER HDMI IS NOT IN THE SIGNAL CHAIN. The show goes UMC -> FOH; HDMI
# carries the mirrored SCREEN only. If a venue ever needs sound over HDMI,
# delete this file.
#
# To revert everything: delete this file and
# systemctl --user restart wireplumber
# Then re-check TWO things that a wireplumber restart is known to break here
# (see memory reference_umc202hd_interface): the UMC sink volume resets to the
# device default (~87%), and Ardour's Master -> UMC link does NOT come back on
# its own.
monitor.alsa.rules = [
{
# The SOF headset-jack microphone. 1.4M xruns, no legitimate consumer, and
# it had wired itself into the d1 stem recording.
matches = [
{ node.name = "alsa_input.pci-0000_00_1f.3-platform-sof_sdw.HiFi__Headset__source" }
]
actions = {
update-props = {
node.disabled = true
}
}
}
{
# HDMI audio outputs. Screen mirroring brings these up; the rig never sends
# audio through them. HDMI1 alone was 704k xruns and 2.8ms BUSY per cycle.
matches = [
{ node.name = "alsa_output.pci-0000_00_1f.3-platform-sof_sdw.HiFi__HDMI1__sink" }
{ node.name = "alsa_output.pci-0000_00_1f.3-platform-sof_sdw.HiFi__HDMI2__sink" }
{ node.name = "alsa_output.pci-0000_00_1f.3-platform-sof_sdw.HiFi__HDMI3__sink" }
]
actions = {
update-props = {
node.disabled = true
}
}
}
]
#!/usr/bin/env bash
# check-audio-graph — assert the PipeWire graph is the one the RIG wants.
#
# WHY THIS EXISTS (2026-08-03, J-5 before OPAL)
#
# Sound went glitchy-then-silent mid-rehearsal and every obvious suspect was
# healthy: sclang up, scsynth up and using 6.7% CPU, the UMC sink present and
# RUNNING, every link in place. Nothing that existed at the time could have
# caught it, because the fault was not in any component — it was in the SHAPE OF
# THE GRAPH. Two nodes nobody used (the SOF headset mic and an HDMI sink brought
# up by screen mirroring) were held permanently open by a graph-wide no-suspend
# rule and had accumulated 2.1 MILLION xruns between them, starving the only
# device in the box with a real-time deadline tight enough to care.
#
# Worse, the same audit found the laptop MICROPHONE wired into `ardour:Tidal 01`
# — the track that records d1 — because Ardour auto-connects physical inputs.
# That is invisible in Ardour's GUI and invisible in the saved session file. It
# had been silently summed into stem recordings.
#
# So this script asserts the things that were wrong, in the order that they bite.
#
# TWO METHODOLOGY RULES IT OBEYS, both learned the hard way that day:
#
# 1. NEVER SCAN THE JOURNAL. `journalctl --since ...` on a long-uptime box reads
# the whole journal and burned a full core TWICE while PLN was losing audio —
# the diagnostic making the fault worse. Everything here is /proc, pw-link
# and a bounded pw-top.
#
# 2. `state: RUNNING` PROVES NOTHING. It read RUNNING while the device was dying
# and restarting once a second. The only honest reading is the hardware
# pointer, sampled repeatedly: ~48000 frames/sec is realtime, and a NEGATIVE
# delta means the stream restarted. Take several samples — a two-sample read
# gave a confidently wrong verdict because a ring-buffer wrap looks exactly
# like a stall.
#
# Usage:
# tools/check-audio-graph.sh # assert; non-zero on any hard failure
# tools/check-audio-graph.sh -v # show every measurement
set -uo pipefail
VERBOSE=0
[ "${1:-}" = "-v" ] && VERBOSE=1
FAIL=0
WARN=0
ok() { printf ' \033[32mok\033[0m %s\n' "$*"; }
bad() { printf ' \033[31mFAIL\033[0m %s\n' "$*"; FAIL=$((FAIL+1)); }
warn() { printf ' \033[33mwarn\033[0m %s\n' "$*"; WARN=$((WARN+1)); }
note() { [ "$VERBOSE" = 1 ] && printf ' %s\n' "$*"; return 0; }
echo "check-audio-graph — is the graph the one the rig wants?"
echo
# ---------------------------------------------------------------------------
# 1. NOTHING BUT SUPERCOLLIDER MAY FEED A TIDAL TRACK
#
# This is the #132 bug and it goes first because it is the only one here that
# corrupts a RECORDING rather than merely risking a dropout. A dropout you hear
# and can redo; a stem with room noise baked in you ship.
# ---------------------------------------------------------------------------
echo "1. recording integrity — only SuperCollider feeds the Tidal tracks"
if ! command -v pw-link >/dev/null 2>&1; then
warn "pw-link not found — cannot audit the graph"
else
LINKS=$(pw-link -l 2>/dev/null)
# pw-link prints a port, then its peers indented. Walk it and remember the
# last non-indented line so an indented `ardour:Tidal NN/audio_in` peer can
# be attributed to the SOURCE that feeds it.
STRAY=$(printf '%s\n' "$LINKS" | awk '
/^[^[:space:]]/ { src = $0; next }
/\|-> *ardour:Tidal [0-9]+\/audio_in/ {
if (src !~ /^SuperCollider:out_/) print src
}' | sort -u)
if [ -n "$STRAY" ]; then
bad "a NON-SuperCollider source feeds a Tidal track — stems will contain it:"
printf '%s\n' "$STRAY" | sed 's/^/ /'
else
N=$(printf '%s\n' "$LINKS" | grep -c 'ardour:Tidal [0-9]*/audio_in' || true)
ok "no stray source on any Tidal input ($N Tidal inputs linked)"
fi
fi
echo
# ---------------------------------------------------------------------------
# 2. THE MONITOR PATH ACTUALLY REACHES THE INTERFACE
#
# Ardour restores its own saved port connections and does NOT follow the
# PipeWire default sink. Switching the default sink once left the whole rig
# playing out of the laptop jack — the very codec the UMC exists to escape —
# with no warning at all. Assert the last hop explicitly, every time.
# ---------------------------------------------------------------------------
echo "2. monitor path — ardour Master reaches the UMC"
MASTER=$(pw-link -l 2>/dev/null | awk '/^ardour:Master\/audio_out/ {f=1; next} /^[^[:space:]]/ {f=0} f')
UMC_CH=$(printf '%s\n' "$MASTER" | grep -c 'UMC202HD' || true)
SOF_CH=$(printf '%s\n' "$MASTER" | grep -c 'sof_sdw' || true)
if [ "$UMC_CH" -ge 2 ]; then
ok "both Master channels on the UMC202HD"
elif [ "$SOF_CH" -gt 0 ]; then
bad "Master is on the INTERNAL codec (sof_sdw), not the UMC — the thing the UMC exists to avoid"
else
bad "Master reaches the UMC on $UMC_CH/2 channels"
fi
echo
# ---------------------------------------------------------------------------
# 3. IS THE INTERFACE ACTUALLY CLOCKING?
#
# The measurement that cannot lie. See rule 2 in the header.
# ---------------------------------------------------------------------------
echo "3. the interface is clocking (hw_ptr, not state)"
PCM=$(ls -d /proc/asound/U192k/pcm0p/sub0 2>/dev/null || true)
if [ -z "$PCM" ]; then
warn "no UMC playback substream (device absent, or card renamed — it has been card 0 and card 1)"
elif ! grep -q '^state' "$PCM/status" 2>/dev/null; then
note "stream closed (nothing playing) — cannot judge clocking"
ok "no stream open; nothing to measure"
else
prev=""; restarts=0; slow=0; samples=0
for _ in 1 2 3 4; do
read -r st hp < <(awk '/^state/{s=$2} /^hw_ptr/{h=$3} END{print s, h}' "$PCM/status" 2>/dev/null)
if [ -n "$prev" ] && [ -n "${hp:-}" ]; then
d=$((hp - prev)); samples=$((samples+1))
note "state=$st delta=$d frames/s"
# Negative => the stream was torn down and restarted. That is the
# signature of the J-5 failure: it never got past ~3000 frames.
[ "$d" -lt 0 ] && restarts=$((restarts+1))
# Allow generous slack: this is a "is it moving at all" test, not a
# drift measurement, and the sleep itself is not precise.
[ "$d" -ge 0 ] && [ "$d" -lt 40000 ] && slow=$((slow+1))
fi
prev=${hp:-}
sleep 1
done
if [ "$restarts" -gt 0 ]; then
bad "playback stream RESTARTED $restarts time(s) in ${samples}s (negative hw_ptr delta) — the J-5 dropout signature"
elif [ "$slow" -gt 0 ]; then
bad "hw_ptr advanced under 40k frames/s on $slow of $samples samples — not keeping realtime"
elif [ "$samples" -gt 0 ]; then
ok "clocking at realtime across $samples samples, zero restarts"
else
warn "could not read hw_ptr"
fi
fi
echo
# ---------------------------------------------------------------------------
# 4. THE PRUNE IS STILL IN EFFECT
#
# These nodes are disabled in config/wireplumber/53-rig-graph.conf. If they are
# back, either the symlink broke or someone deleted the file — and the 2.1M-xrun
# condition is back with them.
# ---------------------------------------------------------------------------
echo "4. the graph prune is still in effect"
if command -v pactl >/dev/null 2>&1; then
SINKS=$(pactl list short sinks 2>/dev/null)
SRCS=$(pactl list short sources 2>/dev/null)
HDMI=$(printf '%s\n' "$SINKS" | grep -ci 'HDMI' || true)
HEAD=$(printf '%s\n' "$SRCS" | grep -ci 'Headset__source' || true)
[ "$HDMI" -eq 0 ] && ok "no HDMI audio sinks (704k xruns, 2.8ms BUSY/cycle when present)" \
|| bad "$HDMI HDMI sink(s) are back — 53-rig-graph.conf is not being applied"
[ "$HEAD" -eq 0 ] && ok "no SOF headset mic source (1.4M xruns, and it wired itself into d1)" \
|| bad "the SOF headset source is back — this is the node that fed d1"
note "$(printf '%s\n' "$SINKS" | wc -l) sink(s) total"
else
warn "pactl not found"
fi
# The symlinks are the reason the config survives a fresh checkout; a broken one
# is silent until the next wireplumber restart, which is the worst time to learn.
WPD="$HOME/.config/wireplumber/wireplumber.conf.d"
brk=0
for f in "$WPD"/*.conf; do
[ -e "$f" ] || continue
if [ -L "$f" ] && [ ! -e "$(readlink -f "$f")" ]; then
bad "dangling symlink: $(basename "$f") -> $(readlink "$f")"; brk=1
fi
done
[ "$brk" = 0 ] && ok "no dangling wireplumber symlinks"
echo
# ---------------------------------------------------------------------------
# 5. HEADROOM — a WARNING, never a failure
#
# The rig has never had a headroom budget; that is the whole lesson of the J-5
# dropout. But load is spiky and this must not block a gate over a transient, so
# it warns. `pulsar` gets named explicitly because it is the known offender (#7,
# measured at 82% IDLE, and 93% during the incident).
# ---------------------------------------------------------------------------
echo "5. headroom (advisory)"
LOAD=$(awk '{print $1}' /proc/loadavg)
CORES=$(nproc 2>/dev/null || echo 4)
note "load1=$LOAD over $CORES cores"
if awk -v l="$LOAD" -v c="$CORES" 'BEGIN{exit !(l > c*0.75)}'; then
warn "load1 $LOAD is high for $CORES cores — the UMC's deadline is the first thing to miss"
else
ok "load1 $LOAD on $CORES cores"
fi
PULSAR=$(ps -eo pcpu,comm --no-headers 2>/dev/null | awk '$2=="pulsar"{s+=$1} END{printf "%.0f", s+0}')
if [ "${PULSAR:-0}" -ge 60 ]; then
warn "pulsar at ${PULSAR}% — #7 (highlighter marker pool). This is the JITTER SOURCE; a Window:Reload is the stopgap"
else
ok "pulsar at ${PULSAR:-0}%"
fi
echo
if [ "$FAIL" -gt 0 ]; then
echo "check-audio-graph: $FAIL failure(s), $WARN warning(s)"
exit 1
fi
echo "check-audio-graph: OK${WARN:+ ($WARN warning(s))}"
......@@ -244,7 +244,8 @@ run "tools executable" \
bad=0
for f in tools/gig-up.sh tools/check-boot.sh tools/check-tracks.sh \
tools/check-preload.sh tools/sc-watchdog.sh tools/setlist.py \
tools/check-drift.sh tools/take-segments.py tools/fader-baseline.py; do
tools/check-drift.sh tools/take-segments.py tools/fader-baseline.py \
tools/check-audio-graph.sh; do
[ -e "$f" ] || continue
[ -x "$f" ] || { echo "not executable on disk: $f"; bad=1; }
m=$(git ls-files -s "$f" 2>/dev/null | awk "{print \$1}")
......@@ -300,6 +301,19 @@ run "ardour faders" \
"raise it on the desk, then Ctrl+S in Ardour, then re-run. Still -inf after a save = genuinely down." \
python3 tools/check-mix.py --quiet
# --- 2b. the shape of the audio graph ----------------------------------------
# Added after the 2026-08-03 J-5 dropout, which NOTHING in this gate could have
# caught: every component was healthy and the fault was in the graph around them.
# It also catches the laptop microphone that had wired itself into the d1 stem
# (#132) — invisible in Ardour's GUI and in the saved session, so no amount of
# inspecting Ardour would ever have found it.
#
# check-mix above reads the SAVED session; this reads the LIVE graph. They are
# different questions and the saved file has been wrong before (#62).
run "audio graph" \
"tools/check-audio-graph.sh -v # recording integrity, Master->UMC, is the interface clocking, is the prune live" \
bash tools/check-audio-graph.sh
# --- 3. every setlist track compiles -----------------------------------------
# Cold, seeded, no rig required. Baseline 13/13 as of 2026-07-31.
run "setlist compiles" \
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment