Commit d280acf0 by PLN (Algolia)

fix(silent-eval): the harness called 128 working tracks broken, and a crash "silence" (#112)

#112 said "14 corpus tracks do not compile". Running the harness over the whole
corpus said 128. Neither number was about the music.

BUG 1 — column-0 `$` continuations reported as parse errors
  PLN writes a lot of the corpus like this:

      d1
      $ whenmod 128 129 (…)
      $ s "<k k*2 <k*2 k> k>"

  Pulsar sends a block wrapped in GHCi's `:{ … :}`, which suspends the layout
  rule, so that `$` at column 0 is a continuation and the track plays fine.
  `orbit_bindings` rewrites `dN` into `name = idcp` and emits the block into a
  real MODULE, where a column-0 `$` opens a NEW top-level declaration — a hard
  parse error. The harness then printed, in these words:

       THE TRACK DOES NOT COMPILE — it will fail live too.

  about code that has never once failed live. The intent to indent was already
  written in the comment above the line ("becomes `name = idcp` + `  $ x`");
  the indent itself was never applied. Two spaces on lines[1:] restores it and
  preserves every relative indent below.

  Why it stayed hidden: ZERO of the 13 setlist tracks use that style, and the
  setlist is what anyone actually runs. The harness was green on everything it
  was ever pointed at. Same shape as the chmod bug in the previous commit — a
  thing verified once, in a context that no longer covers the corpus.

BUG 2 — a crashed probe reported as "FAIL — 0 orbit(s) SILENT"
  The Haskell probe sets its failure flag ONLY when it prints a SILENT line, so
  a non-zero exit with no SILENT line never measured silence at all — it threw
  while QUERYING. `check_track` mapped any non-zero return to "SILENT" and
  discarded stderr, so the one string that said what went wrong was thrown away
  and replaced by a self-contradicting verdict. New CRASH verdict keeps stderr.

  It immediately paid for itself. `live/chill/dub.tidal` compiles, plays three
  orbits, then throws:

      Syntax error in sequence:
        "<0.75 .. 0.8 0.8 .. 0.65>"
                     ^

  A real mini-notation bug, precisely located — and one that WOULD fail live,
  since the scheduler queries every cycle. It had been sitting behind a message
  that said zero orbits were silent.

BUG 3 (latent, found while fixing 2) — a false green in the exit code
  Splitting the summary so crashes stop being counted as silence revealed that
  the return was keyed to the silence count alone. A corpus where every single
  track failed to compile would have printed "OK — every declared orbit emits
  events" and exited 0. The exit code now answers "is anything wrong", which is
  the only question a gate may answer — this is the one tool whose entire job
  is catching the failure that makes no noise.

Verified: setlist still 13/13 green and gig-up still GO (so no regression on
the path that matters); dub now reports THROWS WHEN QUERIED and exits 1;
previously-"broken" tracks compile. The corpus-wide count is deliberately NOT
restated here — it needs a clean re-run, and quoting a number I have not
re-measured is how #112 got a wrong one in the first place.
parent 1219c704
...@@ -171,6 +171,16 @@ def orbit_bindings(track: Track) -> list[tuple[str, int, int, str]]: ...@@ -171,6 +171,16 @@ def orbit_bindings(track: Track) -> list[tuple[str, int, int, str]]:
# `name = idcp $ x`, and a bare `dN` with the `$` on the next line # `name = idcp $ x`, and a bare `dN` with the `$` on the next line
# becomes `name = idcp` + ` $ x`, which still parses as application. # becomes `name = idcp` + ` $ x`, which still parses as application.
lines[0] = f"{name} = idcp" + lines[0][len(m.group(1)):] lines[0] = f"{name} = idcp" + lines[0][len(m.group(1)):]
# Continuation lines MUST be indented. Pulsar sends a block wrapped in
# GHCi's `:{ … :}`, which suspends layout, so PLN can and does write
# d1
# $ whenmod 128 129 (…)
# with the `$` at column 0. In a real module that column-0 `$` opens a
# NEW top-level declaration and is a hard parse error — so the harness
# was reporting "THE TRACK DOES NOT COMPILE — it will fail live too"
# about code that plays perfectly. Two spaces restores the layout
# continuation and preserves every relative indent below it.
lines[1:] = [" " + l for l in lines[1:]]
out.append((name, orb.number, orb.start, "\n".join(lines))) out.append((name, orb.number, orb.start, "\n".join(lines)))
return out return out
...@@ -380,7 +390,17 @@ def check_track(path: Path, seeded: bool, keep: bool) -> tuple[str, str]: ...@@ -380,7 +390,17 @@ def check_track(path: Path, seeded: bool, keep: bool) -> tuple[str, str]:
return ("BROKEN" if blame(err, ranges) == "TRACK" else "BUILD"), \ return ("BROKEN" if blame(err, ranges) == "TRACK" else "BUILD"), \
relocate(err.strip(), src, path) relocate(err.strip(), src, path)
r = subprocess.run([str(work / "run")], capture_output=True, text=True) r = subprocess.run([str(work / "run")], capture_output=True, text=True)
return ("SILENT" if r.returncode else "ok"), r.stdout.rstrip() if not r.returncode:
return "ok", r.stdout.rstrip()
# The probe sets its failure flag ONLY when it prints a SILENT line, so
# a non-zero exit with no SILENT line did not measure silence — it threw
# while QUERYING the pattern. Reporting that as "FAIL — 0 orbit(s)
# SILENT" is a verdict about the music that the run never reached, and
# it hid the stderr that says what actually went wrong.
if "SILENT" not in r.stdout:
return "CRASH", ((r.stdout.rstrip() + "\n") if r.stdout.strip() else "") \
+ r.stderr.strip()
return "SILENT", r.stdout.rstrip()
def setlist_tracks() -> list[Path]: def setlist_tracks() -> list[Path]:
...@@ -409,7 +429,7 @@ def main(argv: list[str] | None = None) -> int: ...@@ -409,7 +429,7 @@ def main(argv: list[str] | None = None) -> int:
mode = "WITH the #55 boot seed" if a.seeded else "with an EMPTY control map" mode = "WITH the #55 boot seed" if a.seeded else "with an EMPTY control map"
print(f"silent-eval: {len(tracks)} track(s), {mode}\n") print(f"silent-eval: {len(tracks)} track(s), {mode}\n")
bad = built = broken = 0 bad = built = broken = crashed = 0
for t in tracks: for t in tracks:
if not t.exists(): if not t.exists():
print(f" {t.name}: no such file") print(f" {t.name}: no such file")
...@@ -431,6 +451,16 @@ def main(argv: list[str] | None = None) -> int: ...@@ -431,6 +451,16 @@ def main(argv: list[str] | None = None) -> int:
print("\n".join(" " + l for l in detail.splitlines()[:6])) print("\n".join(" " + l for l in detail.splitlines()[:6]))
built += 1 built += 1
continue continue
if verdict == "CRASH":
# It compiled and then threw while being queried. Live, the
# scheduler queries every cycle, so this is a real track fault —
# just not the one the SILENT wording would have claimed.
print(f" {t.stem}: ✖ THROWS WHEN QUERIED — it compiles, then blows "
f"up. Live, the scheduler queries every cycle.")
print("\n".join(" " + l for l in detail.splitlines()[:6]))
crashed += 1
bad += 1
continue
n_silent = detail.count("SILENT") n_silent = detail.count("SILENT")
head = "ok" if verdict == "ok" else f"FAIL — {n_silent} orbit(s) SILENT" head = "ok" if verdict == "ok" else f"FAIL — {n_silent} orbit(s) SILENT"
print(f" {t.stem}: {head}") print(f" {t.stem}: {head}")
...@@ -447,9 +477,20 @@ def main(argv: list[str] | None = None) -> int: ...@@ -447,9 +477,20 @@ def main(argv: list[str] | None = None) -> int:
if built: if built:
print(f"silent-eval: {built} track(s) could not be built by the harness " print(f"silent-eval: {built} track(s) could not be built by the harness "
f"— fix the harness, do not read these as passes") f"— fix the harness, do not read these as passes")
if bad: if crashed:
print(f"silent-eval: FAIL — {bad} track(s) have an orbit that emits " print(f"silent-eval: ✖ {crashed} track(s) COMPILE BUT THROW when queried "
f"— read the exception, it is not a silence problem.")
# Only the tracks that actually MEASURED silence get the silence wording;
# broken/crashed ones never got far enough for that to be a claim about them.
silent_only = bad - broken - crashed
if silent_only > 0:
print(f"silent-eval: FAIL — {silent_only} track(s) have an orbit that emits "
f"NOTHING on a fresh boot.") f"NOTHING on a fresh boot.")
# The exit code answers "is anything wrong", NOT "is anything silent". Keying
# it to silence alone would print OK and exit 0 for a corpus where every
# track fails to compile — a false green from the one tool whose whole job is
# catching the failure that makes no noise.
if bad:
return 1 return 1
print("silent-eval: OK — every declared orbit emits events.") print("silent-eval: OK — every declared orbit emits events.")
return 0 return 0
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment