Commit 0c553fd6 by PLN (Algolia)

fix(rig): the test paged a human about a unit that did not exist (#115)

Minutes after test-sc-watchdog exited, PLN got a STICKY desktop notification:

    "3 restarts in 10 min. Giving up — this needs your eyes.
     journalctl --user -u scwd-test-1269280.service -n 50"

scwd-test-1269280.service was the test's own FAKE transient unit. By the time the
notification was read, it had been torn down — so the suggested command returns
nothing, about a failure that never happened, while the real rig was playing fine.

Two defects in one message. The test could reach a real notification daemon at all;
and the give-up path is deliberately urgency=critical/sticky, which is correct for a
real outage and actively harmful for a fake one — it cannot be dismissed by waiting.

Fix: SCWD_NO_NOTIFY=1, honoured by notify() and set by the harness. A test must never
be able to page a human about something that is not real.

Re-run green: 7 passed, 0 failed, and silent. The log still shows the rate limiter
walking (0 prior) -> (1 prior) -> (2 prior) -> GIVING UP, so suppressing the toast
did not suppress the evidence — the assertion reads the log, not the notification.

Same family as the flap it was written to catch: a supervisory tool whose side
effects escape the scope it was reasoning about.
parent b978a2fa
...@@ -82,6 +82,12 @@ log() { printf '%s %s\n' "$(date +%H:%M:%S)" "$*" >&2; } ...@@ -82,6 +82,12 @@ log() { printf '%s %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
# rig was busy recovering itself. If everything is sticky, nothing is urgent. # rig was busy recovering itself. If everything is sticky, nothing is urgent.
notify() { notify() {
local urgency="$1" timeout="$2" title="$3" body="$4" local urgency="$1" timeout="$2" title="$3" body="$4"
# SCWD_NO_NOTIFY: for the test harness. The first test run fired REAL sticky desktop
# notifications about a FAKE unit ("3 restarts in 10 min. Giving up — this needs your
# eyes. journalctl --user -u scwd-test-1269280.service"), which reached PLN minutes
# after the test had finished and named a unit that no longer existed. A test must
# never be able to page a human about something that is not real.
[ "${SCWD_NO_NOTIFY:-0}" = "1" ] && return 0
command -v notify-send >/dev/null 2>&1 || return 0 command -v notify-send >/dev/null 2>&1 || return 0
notify-send -a "ParVagues" -u "$urgency" -t "$timeout" "$title" "$body" 2>/dev/null || true notify-send -a "ParVagues" -u "$urgency" -t "$timeout" "$title" "$body" 2>/dev/null || true
} }
......
...@@ -65,7 +65,10 @@ else ...@@ -65,7 +65,10 @@ else
fi fi
# --- 2. the watchdog notices and restarts --------------------------------------- # --- 2. the watchdog notices and restarts ---------------------------------------
SCWD_UNIT="$UNIT" SCWD_PROC="$PROC" SCWD_STATE="$STATE" \ # SCWD_NO_NOTIFY=1: a test must never page a human about a fake unit. The first run
# fired real STICKY notifications naming scwd-test-<pid>.service, which reached PLN
# after the test had exited and after that unit had ceased to exist.
SCWD_UNIT="$UNIT" SCWD_PROC="$PROC" SCWD_STATE="$STATE" SCWD_NO_NOTIFY=1 \
./tools/sc-watchdog.sh >"$TMP/wd.log" 2>&1 & ./tools/sc-watchdog.sh >"$TMP/wd.log" 2>&1 &
WD_PID=$! WD_PID=$!
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment