Commit cafce446 by PLN (Algolia)

deploy(fourier): idempotent launcher that persists the memory cap

A hand-typed docker run omitted --memory, so a runaway job could OOM its
neighbours on the shared box; docker update only survives a restart, not a
rm+run redeploy. deploy/fourier.sh is the durable home for the cap: it
(re)launches both the API and worker containers with every flag baked in
(seccomp, env-file, /data volume, loopback port, restart policy, --memory,
default 3g each, overridable). Preflight refuses to launch under disk pressure.

DEPLOY.md §4 now points at the launcher. The host-venv worker.service unit is
marked superseded (the worker runs containerized on erable).
parent 94711a72
...@@ -67,18 +67,29 @@ MKL_NUM_THREADS=1 ...@@ -67,18 +67,29 @@ MKL_NUM_THREADS=1
## 4. Run ## 4. Run
Use the launcher — it bakes in every flag idempotently (rm + run), including the
**memory cap** that a hand-typed `docker run` used to omit:
```bash ```bash
docker run -d --name fourier --restart unless-stopped \ deploy/fourier.sh up # (re)launches BOTH the API and the worker container
--security-opt seccomp=unconfined \ deploy/fourier.sh ps # status + effective mem caps
--env-file ~/.config/fourier/fourier.env \
-v /home/pln/srv/fourier/data:/data \
-p 127.0.0.1:9780:9780 \
fourier:latest
``` ```
`-p 127.0.0.1:9780:9780` is the whole security model: loopback-only means the It launches two containers off the same image: **`fourier`** (uvicorn, the HTTP
gateway is the *only* way in, so the injected `X-Tenant`/`X-Scopes` headers can be API) and **`fourier-worker`** (`python -m worker`, the heavy-engine job runner).
trusted. SRE wraps this in a keep-alive systemd unit. Both get `--restart unless-stopped`, `--security-opt seccomp=unconfined`, the
`--env-file`, the `/data` volume, and `--memory` (default `3g` each — override
via `FOURIER_MEM` / `FOURIER_API_MEM` / `FOURIER_WORKER_MEM`).
**Why the cap matters:** erable is a shared ~7.7 GB box (Postgres + the platform
live there too). Uncapped, a runaway separate/sources job could OOM its
neighbours. `docker update --memory` only survives a *restart*, not a rm+run
redeploy — the script is the durable home for the cap, so redeploys keep it.
The API publishes `-p 127.0.0.1:9780:9780`; loopback-only is the whole security
model — the gateway is the *only* way in, so the injected `X-Tenant`/`X-Scopes`
headers can be trusted. uvicorn binds `0.0.0.0` *inside* the container (the
loopback restriction is the host-side `-p`). The worker publishes no ports.
### Two erable gotchas (verified at first deploy, 2026-06-28) ### Two erable gotchas (verified at first deploy, 2026-06-28)
......
# Fourier job worker (#25) — systemd --user unit. # Fourier job worker (#25) — systemd --user unit.
# #
# SUPERSEDED on erable (2026-07): the worker now runs CONTAINERIZED off the same
# fourier:latest image, launched by `deploy/fourier.sh up` alongside the API (see
# DEPLOY.md §4). This host-venv unit remains as the paved-road alternative for a
# deploy host that runs the worker outside Docker; on erable it is NOT installed.
#
# The worker is a SEPARATE long-lived process from the API container: it claims # The worker is a SEPARATE long-lived process from the API container: it claims
# pending jobs and runs the heavy engines (separate/sources/loops). Runs as a # pending jobs and runs the heavy engines (separate/sources/loops). Runs as a
# systemd --user service with lingering on, so it survives logout and starts at # systemd --user service with lingering on, so it survives logout and starts at
......
#!/usr/bin/env bash
# Idempotent launcher for the Fourier (audio sub-API) containers on erable.
#
# Replaces the hand-typed `docker run` from DEPLOY.md §4 so that redeploys
# reproduce ALL the flags — most importantly the memory cap, which a bare
# `docker run` used to omit (a runaway job could then OOM the DB/platform on
# the shared 7.7 GB box). `docker update` only survives a restart, NOT a
# rm+run redeploy; this script is the durable home for the cap.
#
# Usage:
# deploy/fourier.sh up # (re)launch both containers (default)
# deploy/fourier.sh restart # same as up (rm + run is our restart)
# deploy/fourier.sh down # stop + remove both
# deploy/fourier.sh ps # show status + effective mem caps
#
# Everything is overridable by env; defaults match the live 2026-07 config.
set -euo pipefail
IMAGE="${FOURIER_IMAGE:-fourier:latest}"
DATA_DIR="${FOURIER_DATA_DIR:-$HOME/srv/fourier/data}"
ENV_FILE="${FOURIER_ENV_FILE:-$HOME/.config/fourier/fourier.env}"
BIND="${FOURIER_BIND:-127.0.0.1}" # loopback-only: the gateway is the only way in
PORT="${FOURIER_PORT:-9780}"
# Memory caps — the whole point of this script existing.
# The box has ~7.7 GB total shared with Postgres + the platform; keep the two
# audio containers bounded so a heavy job can't OOM its neighbours. Worker runs
# the heavy engines (separate/sources), API is CPU-light — but we cap both at
# 3g to match the live config. Override per-container if you retune.
MEM="${FOURIER_MEM:-3g}"
API_MEM="${FOURIER_API_MEM:-$MEM}"
WORKER_MEM="${FOURIER_WORKER_MEM:-$MEM}"
# Flags shared by both containers (see DEPLOY.md §4 for the why of each).
common_flags=(
--restart unless-stopped
--security-opt seccomp=unconfined # kernel 4.9 / clone3 — REQUIRED, see DEPLOY.md gotcha #1
--env-file "$ENV_FILE"
-v "$DATA_DIR:/data"
)
preflight() {
[[ -f "$ENV_FILE" ]] || { echo "!! env file missing: $ENV_FILE (see DEPLOY.md §3)" >&2; exit 1; }
mkdir -p "$DATA_DIR"
# Guard the shared box: refuse to launch if the disk is already tight.
local pct; pct=$(df --output=pcent / | tail -1 | tr -dc '0-9')
if (( pct >= 90 )); then
echo "!! root fs at ${pct}% — refusing to launch (free space first; see disk-monitor)" >&2
exit 1
fi
}
down() {
docker rm -f fourier fourier-worker 2>/dev/null || true
}
up() {
preflight
down
# API: serves HTTP on loopback; uvicorn binds 0.0.0.0 INSIDE (loopback is host-side -p).
docker run -d --name fourier "${common_flags[@]}" \
--memory "$API_MEM" \
-p "$BIND:$PORT:$PORT" \
"$IMAGE" \
uvicorn app:app --host 0.0.0.0 --port "$PORT"
# Worker: no ports; claims jobs and runs the heavy engines.
docker run -d --name fourier-worker "${common_flags[@]}" \
--memory "$WORKER_MEM" \
"$IMAGE" \
python -m worker
echo "launched fourier (mem=$API_MEM) + fourier-worker (mem=$WORKER_MEM)"
ps
}
ps() {
docker ps --filter name=fourier --format 'table {{.Names}}\t{{.Status}}'
for c in fourier fourier-worker; do
printf '%s mem-cap: ' "$c"
docker inspect -f '{{.HostConfig.Memory}} bytes' "$c" 2>/dev/null || echo '(not running)'
done
}
case "${1:-up}" in
up|restart) up ;;
down) down; echo "removed fourier + fourier-worker" ;;
ps|status) ps ;;
*) echo "usage: $0 {up|restart|down|ps}" >&2; exit 2 ;;
esac
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment