# L'Armada — completed-task archive (structured)

Long-term learning + blog/video source material. Each entry = a finished board task
captured in full (description · done · learnings · deps), appended by the `/pre-compact`
ritual before the task is cleared off the active board. Richer than the terse tracker
snapshot in `board-archive.md`. Newest first.

---

## #85 — Pack-level sample grouping (the grouping above the folder)
- **Description:** The Dirt-Samples FOLDER is a loose convention, not the true grouping —
  `rample*` is one bank sliced A0..S57, `808bd/cy/hc/mc` is one drum machine per voice,
  `*_commodore` is one chip sliced by role. PLN wanted the PACK granularity above the
  folder, so we can browse/aggregate at three grains (pack / folder / sample).
- **Done:** `sample_packs.py` + `sample_packs.json` (commit 96bc7bc), captain's log 016.
  A curated REGISTRY (types: drum_machine / bank / chip / source / genre) + an auto-grouper
  (alpha-prefix-before-digit + underscore head/tail tokens, ≥2 folders) that PROPOSES
  packs the registry missed — it caught `trance_*` and `voices_*`, folded in. Reads the
  already-grounded `sample_families.json` (no audio → fast, CPU-light). 22 packs over
  83/150 grounded folders; rolls per-sample family counts up to each pack.
- **Learnings:** Each grain answers a different question — folder = "what did the pack
  author call this slice?", pack = "where did it come from?", sample = "what does it
  actually sound like?". The signal isn't agreement, it's the MISMATCH: 15 of 22 packs
  span multiple audio families BY DESIGN (rample → 8 families, jungle → 11, one Commodore
  chip → bass/lead/pad/fx/synth). A pack defined by ORIGIN is *expected* to scatter across
  the timbre wheel; that scatter is a story (for Unwrapped viz / vibe-search scoping), not
  an error. A 1-folder "pack" is just a folder within the grounded subset — require ≥2
  (may gain siblings when #84 grounds all 718). Same discipline as the folder: a hint to
  compare against, never a law.
- **Deps:** Builds on the grounded `sample_families.json` (#69 resolver); feeds the
  Unwrapped viz (#64 lineage) and vibe-search. Re-run after #84.

## #65 — Fix serve.py Range broken-pipe crash on audio seek
- **Description:** `armada/serve.py copyfile()` threw `ConnectionResetError`/`BrokenPipeError`
  (Errno 104) when a browser aborts/seeks a Range request mid-stream (e.g. `proxy_take89.mp3`
  in the triangle drawer) — the 206 responses worked but the unhandled exception spammed the
  console.
- **Done:** Wrapped both copy branches of `copyfile()` and guaranteed the file handle closes
  in a `finally`; swallow `BrokenPipeError`/`ConnectionResetError`. Verified: a mid-stream
  abort (`curl --max-time` cut) leaves NO traceback and the server keeps serving (206
  follow-up OK). Commit **d70c3ee**.
- **Learnings:** A client disconnect on `<audio>` scrubbing is *normal*, not a failure —
  swallow it, don't log it. This was the keystone QOL fix: it de-frictions every audition
  task (#35 ear-verify, #43/#44 frag-audition, #20 best-take), so cheap-but-high-leverage.
- **Deps:** Blocked nothing formally; unblocks the ear-gating workflow ergonomically.

## #64 — "Information is Beautiful" dataviz of the generated corpus (EPIC)
- **Description:** Phase-2 infodesign of the tide-table corpus (after data-scientist
  exploration) — explorable insights served to a music aficionado.
- **Done:** `corpus.html` narrative + the **Unwrapped** explorable (vibe-search via CLAP,
  dominant-colour density field, zoom/pan) + landing `index.html` + honest style cloud +
  serve.py `/vibe` warm endpoint. Shipped across the prior session's commits; vibe-search,
  field-tuning, label-completion (1490/1501 samples), and zoom all landed.
- **Learnings:** Dots are the stars — the field stays subtle; **don't average hues** (→ mud),
  use the per-cell DOMINANT family colour + purity-fade + desaturate + lift. Cache the field
  offscreen, rebuild on colour/axis/filter change, never on hover. Per-one-shot CLAP
  genre/mood is unreliable (0.3s hits) but CLAP **embeddings are gold for relative
  similarity** (vibe-search / find-similar). Seed free-text search with the user's OWN words.
  DarkReader can fight a dark theme — suspect the inverter before re-tuning to a phantom.
- **Deps:** Spawned #63 (cross-app SPA landing — the tide-table index shipped; tying in the
  React surfaces remains). Set the foundation the emotion-engine work (#89–#92) built on.

---

# The Foundry — phase-2 loops chain (overnight, 2026-06-23)

One session built the entire url→stems→**loops** chain end to end: grader, tierlist,
correlation, finder, naming, GUI, plus engine2 (merge) and audio hardening. Every claim
below is measured; honest negatives kept honest.

## #8 — engine2 Roformer VRAM spike
- **Done:** `research/engine2_spike/spike.sh`; commit 1a201df.
- **Learnings:** Roformer FITS the 6GB RTX 2060 (peak 3051 MiB, ~0.55× realtime) — the
  wall was never VRAM, it was Python: py3.14 breaks (beartype rejects audio-separator's
  `Callable|None` hints); a py3.12 venv works (`~/.virtualenvs/audio-separator`).
- **Deps:** fed #9.

## #10 — Loop grader (the katana), engine/grade.py
- **Done:** pure `grade(wav)->LoopGrade` (S/A/B/C/D + per-rule subscores + flags); commit edb6225.
- **Learnings:** (1) librosa 0.11 tempo is `librosa.feature.tempo`, NOT `.rhythm.tempo` — a
  swallowed AttributeError flat-lined bar-consistency at 0.50 for every file. (2) seam =
  CONTINUITY = the 2nd difference `|x0-2x[-1]+x[-2]|` at the wrap, not the single-sample step
  (the step metric punishes a clean wrap at a zero crossing — a perfect sine scored 0.40).
- **Deps:** reused by #5, #7, #11, #18, #19.

## #11 — Hall of Fame tierlist (data + impeccable viz)
- **Done:** `build_tierlist.py` (1690 loops/97 kits) + `tierlist.html`; commits f88273b, 4c9a831.
- **Learnings:** grade↔.tidal-usage Spearman ρ=+0.34 (rubric points right). The CONFOUND is
  the insight: FX kits (risers 16×, weird_dialogs) tier LOW correctly — sweeps aren't loops,
  so the grader tells loops from transitions; custom `f*` palette validates from the other
  side (fsynth/fguitar A). Only 34/97 kits carry a usage token (folder-name vs `s`-token).
  Showed the confound in the scatter rather than hiding it (trust the instrument).

## #12 — Correlation engine (provenance recovery), engine/correlate.py
- **Done:** NCC sample↔stem matcher + `build_provenance.py`; 482/1690 matched (29%),
  cross-stem alignment 0.34; commit 7a79dc8.
- **Learnings:** a hand-cut loop is a literal slice → NCC peaks ~1.0 (foreign ~0.03). TWO
  load-bearing bugs: (a) naive all-pairs full-res took 56 MIN → fixed with coarse-to-fine
  (rank stems on a frame-RMS ENVELOPE, precise NCC on top-3, ~40× cheaper, minutes); (b)
  SILENT-WINDOW BLOWUP — catastrophic cancellation in the sliding-variance cumsum drove the
  normaliser to ~0 in quiet regions, so silence scored ∞ and faked perfect matches (a run
  reported a bogus 55%/0.69, ~440 false positives). Floor the window norm at 5% of global
  energy + clamp NCC to [-1,1]. Also: py3.14 mp defaults to FORKSERVER → COW-global trick
  fails (empty globals per worker); pin `mp.get_context("fork")`.

## #5 + #18 — Loop finder (engine/loops.py) + v2
- **Done:** §1 pipeline (PLP→beat-sync SSM→Foote→scored windows→Takes→export); grid-from-drums;
  chop mode; commits c17dd0d, de0cb11, cdf2f13, 0a2921a, 5ac5cd0.
- **Learnings:** bare PLP latches the eighth-note pulse → ~2× tempo (Loituma read 250 not 125);
  pin tempo_min/max to one octave + min peak distance. Per-stem PLP grids are WEAK on
  sparse-onset vocals → grid-from-drums (one grid from the rhythmic stem, reused for all)
  lifted recall 0.27→0.34 (vocals 0.27→0.50). Chop mode (onset sub-bar) WORKS but recall
  0.13 < loops 0.34 — PLN picks chops by TASTE from many onsets (recall is a harsh metric;
  onsets over-segment, floored chops at 0.25s). Loops=default, chops=opt-in.

## #7 — Finder recall harness, build_finder_eval.py
- **Done:** recall vs provenance GT (distinct-window, uncapped N + precision); commit 09bace6.
- **Learnings:** honest 0.27 recall / 0.24 precision on the 8 most-cut stems — BELOW the 0.70
  target, and that's diagnostic not failure: those are chop-heavy sources (14-42 sub-bar cuts
  /stem); the bar-finder can't reproduce sub-bar chops, and the finder is a SUGGESTER anyway.

## #9 — Merge-of-stems multimodel, engine/separate.py
- **Done:** MultiModelBackend cascade (Roformer vocals → demucs on the instrumental for
  drums/bass/other → merged 4-stem) + wired RoformerBackend + demucs model selector; commits
  853eeaf, 14e69fe.
- **Learnings:** there is NO 4-stem Roformer (all are 2-stem) — the win is an ENSEMBLE.
  htdemucs_ft is a free drop-in 4-stem A/B (vocals SDR 10.8 vs 9.9). Refactored backends to
  declare `produces` + `run_composite`. Verified on Loituma: 4 clean PCM_16 stems.

## #13/#16/#17 + audio fix — supporting wins
- **#13 naming** (engine/naming.py, 146070b): NN_role[_character] derived from the corpus
  (46% indexed, 74% underscore, kit-name not repeated).
- **#16 live MIDI panel** (c0fd5d4): SSE fan-out into the Bridge, colour-by-event.
- **#17 self-updating corpus essay** (24606d4): templated Story 02 from EDA + freshness badge;
  honesty fix — 4→14 gigs is "grew 3.5×", not the hand-copied "tripled".
- **AUDIO STREAMING fix** (7c3154a): stdlib http.server defaults to HTTP/1.0 → `<audio>`
  seeking (many small Range requests) is flaky in real browsers. Set
  `protocol_version="HTTP/1.1"` on the Foundry Handler AND armada RangeHandler (safe: every
  response sets Content-Length). Also preload="metadata" so durations show (0:00/0:00 read as
  "broken"), and offer the ORIGINAL alongside stems. A fleet-wide gotcha for any audio server.

- **Deps:** the chain feeds #19 (auto-tune settings against the grader) and #20 (batch-explore
  many sources). Engine registry now seeds the product/API/service direction PLN wants.

---

# Douanier — birth of the Audio-Intelligence API (2026-06-25)

Designed live with PLN via a long AskUserQuestion co-design (5 rounds), then built lean. The
product: a self-hosted, token-authed, async API in `armada/api/` exposing the fleet's engines
(emotion, samples, features, separation, loops) to hexa (Shipow) + future seats. Full design in
memory `project_douanier_api`. Named after Le Douanier Rousseau (the customs gate).

## #22 / #36 — Scaffold + emotion walking skeleton
- **Description:** stand up the service shell + ONE real endpoint to de-risk the three seams
  (engines import · service runs · client can auth) before building auth/jobs/cache.
- **Done:** FastAPI app (`app.py`) versioned `/v1`; `/v1/healthz` + synchronous
  `POST /v1/analyze/emotion` (upload → valence/arousal via the tide-table CLAP seam
  sample_semantics.embed_audios → emotion_ontology.score); `engines/` lazy-import adapter;
  runs in `~/.virtualenvs/douanier` (--system-site-packages). Commit 7f558a7.
- **Learnings:** lean rewire — the cheap "ears" don't need the job queue/quota for one trusted
  customer, so a walking skeleton (sync + stub token) reaches value in ~1 session vs 6 tasks.
  Arch PEP-668 blocks system pip → venv that INHERITS the multi-GB ML stack, adds only FastAPI.
  CLAP cold-load is ~34 s (→ later #28 warms it once in the worker). Real pipeline proven on a
  synthetic clip end-to-end.
- **Deps:** root of the whole Douanier tree.

## #23 — Bearer-token auth + scopes + the `douanier` CLI
- **Description:** replace the dev-token stub with a real self-hosted token store.
- **Done:** `db.py` (stdlib sqlite3, WAL for API+worker concurrency); `auth.py` — mint()
  returns plaintext ONCE, stores only sha256 (DB leak unreplayable), verify() → Principal with
  scope/expiry checks, `require_scope()` dependency (401/403); `douanier.py` admin CLI
  (token mint/list/revoke). `/v1/me` echoes identity. Commit fe9ca02.
- **Learnings:** scope='*' wildcard for hexa; dev-token kept as a documented local-only
  bootstrap (prod never sets it) so curl works without minting. 15/15 tests; mint expiry landed
  exactly 31 days out.
- **Deps:** unblocks #24 quota, #33 hexa token; gates every future engine route.

## #26 — Content-addressed cache (the margin lever)
- **Description:** identical audio analyzed once, served forever — the economic core.
- **Done:** `cache.py` — content_id = sha256 of DECODED PCM (re-encodes dedupe) + raw-bytes
  fallback; params_hash folds engine settings; get/put (JSON inline or result_ref path);
  yt-id/url→content_id aliases; LRU/size-cap gc() that unlinks evicted artifacts. Wired into
  `/v1/analyze/emotion` (`X-Douanier-Cache: hit|miss`). Commit 394bec0.
- **Learnings:** **measured 5663× on a real CLAP run** (10.6 s miss → 0.002 s hit). Cache keys
  on the engine so clap/light don't collide. 22/22 tests. This is the margin story vs cloud
  egress that justified self-hosting.
- **Deps:** underpins every engine endpoint + job idempotency; unblocks #27.

## (no task) — Sampling-classics demo sources (Foundry)
- **Done:** `tools/foundry/demos.json` — 10 classics (Amen/Funky Drummer/Apache breaks, Chic
  bass, Loituma vocal, Mozart no-drums, Rickroll, GM Flash, Humpty, Think) + loader + `foundry
  demos` CLI + `/api/demos` + UI quick-picks; `/api/fetch` accepts `ytsearch1:`. Commit fd1fae8.
- **Learnings:** only verified-id URLs ship as links (Loituma from the repo fixture, Rickroll);
  the rest are `ytsearch1:` queries so we never ship a guessed/broken video id. Pedagogically
  diverse on purpose — fuel for #19 autotune + #20 batch-explore.
- **Deps:** feeds #19 / #20.

## #37 — Refit for erable: CPU image + nech.pl platform contract
- **Description:** the SRE replied (SRE.md) that erable has NO GPU / ~2 GB RAM and CLAP/torch
  won't fit; ship a CPU-deployable image. Same letter reframed api.nech.pl as a multi-API
  PLATFORM (we're the `audio` sub-API).
- **Done:** Dockerfile (python:3.12-slim + ffmpeg/libsndfile + torch-free requirements-deploy.txt)
  → **~1 GB image**, <2 GB RAM, default engine=light. Conformed the app: public path `/audio/v1`
  (gateway strips it; routes at root + `root_path`), **central auth** (trust gateway-injected
  X-Tenant/X-Scopes, loopback-only; dropped our SQLite bearer/quota = #23/#24 superseded),
  `/metrics` Prometheus, `/separate` → 503 `compute_unavailable` seam. DEPLOY.md = erable
  contract. Commit 722dade.
- **Learnings:** the SRE EDGE WENT LIVE first (503 warming_up at the public URL) — deploy is
  now a one-command unblock. The container holds ONLY armada/api/ → every engine must be
  self-contained (no tide-table/Foundry import at runtime). 32/32 tests; validated in-container.
- **Deps:** unblocks #42 deploy + #33 onboard; superseded #23/#24.

## #38 — Modular ears building blocks: /features + /analyze/samples
- **Done:** two CPU/torch-free cached endpoints. `/features` (~35-dim spectral/MFCC/chroma/key/
  envelope + rhythm); `/analyze/samples` (per-sample EDA + role percs|bass|melodic|tops|atmos by
  MEASURED spectrum, never the name). `engines/feats.py` self-contained (vendored from
  sample_features/audio_lens). Shared `_cached()` helper (DRY across the 3 analyze routes). Commit 9eb2623.
- **Learnings:** vendoring fixed the librosa-0.11 `feature.rhythm.tempo` bug the tide-table
  original silently swallowed (→ `feature.tempo`). Role-by-measurement upholds feedback_mastering_eda.
  40/40 tests; validated in the torch-free container.
- **Deps:** established the vendoring pattern reused by #39/#41.

## #39 — /grade: the Foundry loop grader as a building block
- **Done:** `/grade` exposes the Foundry mechanical quality grader (composite 0..1 + S/A/B/C/D
  tier + seam/zc/bar/level/dc/bass sub-scores + flags). `engines/grade.py` is a BYTE-FOR-BYTE
  vendored copy; tests/test_grade_endpoint.py is a DRIFT GUARD (imports the Foundry canonical
  standalone, asserts identical grade/tier/sub + WEIGHTS/THRESH). Commit cd3caa7.
- **Learnings:** vendoring + a drift-guard test = DRY-safe deployment when the container can't
  hold the source repo (parsers-over-copy applied to a vendored copy). 43/43 tests.
- **Deps:** the CPU slice of #31 (/loops + /correlate still need a corpus/finder).

## #40 — Low-hanging endpoints: /onsets, /waveform, /analyze
- **Done:** `/onsets` (hit times + tempo + rate), `/waveform` (per-bin [min,max] + 0..1 RMS
  envelope — render-ready), `/analyze` (composite emotion+features+sample in one cached call).
  `engines/signal.py` self-contained librosa. Commit 8c2be60.
- **Learnings:** /onsets+/waveform are aimed at hexa's hydra VISUALS (rhythmic sync + reactive
  envelope) — the API's value isn't just analysis, it's render-ready signal. 50/50 tests.
- **Deps:** —

## #41 — /loudness, /spectrum (FFT-as-a-service), /naming
- **Done:** `/loudness` (BS.1770 integrated LUFS via pyloudnorm + 4×-oversample true-peak +
  crest + gain-to-target -14/-9); `/spectrum` (downsampled mel/linear spectrogram bands×frames,
  0..1, frames=1=avg spectrum — PLN's "FFT-as-a-service" idea); `/naming` (vendored Foundry
  namer, drift-guarded, + feats.character_of). pyloudnorm added to requirements-deploy. Commit f11bef9.
- **Learnings:** /spectrum is the canonical cache-as-margin building block — compute the FFT
  once, serve the bounded matrix forever. **11 engines total.** 58/58 tests.
- **Deps:** —

## #32 — Make it callable: OpenAPI snapshot + zero-dep TS client
- **Done:** `clients/openapi.json` (3.1 snapshot, servers pinned to api.nech.pl/audio/v1, +
  drift-guard test vs the live app), `clients/douanier.ts` (typed, zero-dep, one method/engine,
  `_cache` surfaced, DouanierError; tsc --strict clean), `clients/README.md` (Vercel example +
  scope table + codegen one-liner). Commit 11c8f9d.
- **Learnings:** a hand-written zero-dep client beats generated bloat for ~11 simple multipart
  routes; the snapshot-drift test keeps hexa's spec honest as routes evolve. 60/60 tests.
- **Deps:** unblocks #33 (onboard hexa); needs #42 deploy to be live-callable.

## #42 — Deploy douanier:latest to erable → green /audio/v1/healthz
- **Done:** First real deploy of the CPU image to erable (pln@nech.pl). Image transferred via
  `docker save | ssh erable docker load`, run loopback-only on 127.0.0.1:9780. Registered
  /openapi.json + /docs as public gateway routes (nechapi 89586eb). DEPLOY.md gotchas captured
  (Tidal fbe1742). Green end-to-end: healthz/openapi/docs 200, authed 401 without token.
- **Learnings:** TWO erable traps — (1) `--security-opt seccomp=unconfined` is MANDATORY:
  Docker 19.03/kernel 4.9 default seccomp blocks the `clone3` syscall that python:3.12-slim's
  glibc 2.36 uses for pthread_create → container boots uvicorn then SIGSEGVs (exit 139) /
  "OpenBLAS pthread_create … Operation not permitted". (2) pin OPENBLAS/OMP/NUMEXPR/MKL
  _NUM_THREADS=1. The _platform service sidesteps #1 by being Alpine/musl on purpose. Data
  volume = ~/srv/douanier/data (no sudo for /srv; pure transient cache).
- **Deps:** unblocked #33; the keystone for the whole live API.

## #43 — Hierarchical scope convention (realm:domain:path) + central enforcement + CLI/docs
- **Done:** Convention `realm:domain:path` (realm=api|admin|future mail/data); required scope
  DERIVED from the path (api:audio:analyze:emotion), subtree wildcards (*, api:*, api:audio:*),
  non-wildcard=exact, api:* never grants admin:*. Canonical matcher `_platform/scopes.py` vendored
  byte-identical into `armada/api/scopes.py` (drift-guarded). Audio app: per-route scope strings
  replaced by ONE path-derived `require` dependency + footgun fix (missing X-Scopes → "" not "*").
  CENTRAL enforcement deployed in `_auth` (derive + 403). CLI validates scopes + `check` cmd.
  nechapi 104748f, Tidal 4197696. Verified: hexa api:*→200, admin admin:*→403 on audio, no-id→401.
- **Learnings:** PLN chose a top-level REALM (vs flat domain root) to future-proof for non-API
  capabilities (mail, personal data). Path-derived scopes = maintenance-free access control
  (add a route, its scope exists). The realm split is what lets a customer hold api:* (everything
  API) while admin stays a separate sealed plane. Tenant migration caveat: the active `maps`
  token has OLD-style geo:read/route — must become api:geo:* BEFORE geo joins the gateway.
- **Deps:** built on #37 platform reframe; enables clean multi-API + selling seats.

## #33 — Onboard Shipow / hexa (the MVP delivery)
- **Done:** hexa minted as tenant 3, plan pro, scope api:* (full platform access). Verified
  end-to-end through https://api.nech.pl/audio/v1 with his real token: /me, /features (cache
  miss→hit), /analyze/emotion. Branded onboarding PDF (clients/onboarding.html template →
  chromium --print-to-pdf; rendered PDF gitignored as it carries the token). Tidal b225ba7.
- **Learnings:** platform tokens DON'T expire (the schema has no expiry — revoke-by-prefix
  instead), so the old "31-day" idea didn't apply. Token plaintext is shown ONCE at mint (only
  the sha256 is stored). Onboarding artifact is transitional — PLN's stance: Scalar /docs +
  landing will replace the PDF; "only token distribution matters." Client SDK renamed
  Douanier→NechAPI (umbrella + .audio namespace) + header X-Douanier-Cache→X-Nech-Cache so the
  internal codename never reaches a consumer.
- **Deps:** was blocked by #42 (deploy) + #43 (scopes); closes the EPIC's MVP.

## #45 — Platform observability: usage analytics, security monitoring, uptime status landing
- **Done:** (1) CAPTURE — `_auth` logs EVERY attempt to usage_log (status 200/401/403 + reason +
  client ip; tenant_id nullable; quota counts 200 only); migration +reason +ip +health_checks.
  (2) ADMIN ANALYTICS (admin:*-gated, self-authed): /admin/usage (totals, by-tenant, top routes,
  14d series) + /admin/security (rejects by reason, offender IPs, recent rejects). (3) PUBLIC
  STATUS — api.nech.pl/ now renders per-API up/down + uptime% (24h/7d/30d) from a 60s background
  probe → health_checks; ca-certificates added for the https probe. nechapi e489d26. Verified:
  landing renders, admin endpoints return real data, hexa api:*→403 on /admin (plane sealed),
  rejects logged with source IP.
- **Learnings:** before this, _auth returned early on reject and logged NOTHING — a total
  security blind spot; logging the decision (not just authorized calls) is the foundation.
  Probing the PUBLIC healthz URL (not loopback) measures the real edge a caller sees — and a
  container can't reach the host's 127.0.0.1 anyway. nginx reload briefly fails one probe → a
  small transient uptime dip that self-heals. The admin/api realm split makes /admin trivially
  safe (an api:* token is auto-403'd).
- **Deps:** built on #43 (admin realm) + #45 capture; Grafana visual layer scheduled as #46.
