Ops & Security

Start with the mental rule that never fails: anything that compiles into a static site is public. MOONKEY LAB is Astro 4 in SSG mode — the build generates plain HTML+JS served from Cloudflare Pages. There is no server holding secrets at runtime. Any string you import into a .astro component that reaches the client ends up, literally, in a file anyone can download with `view-source` or by opening DevTools → Sources. That is why `src/lib/supabase.ts` contains ONLY the anon key. The anon key is DESIGNED to be public: it is a project identifier plus a token carrying the `anon` role's permissions, whose real authority is bounded by Postgres RLS. It is not a secret; it is an address with a doorman.

Now the contrast you have to internalize. There are two Supabase keys with the same appearance (a long JWT) but opposite universes of power. The `anon` key operates under the `anon`/`authenticated` role and is SUBJECT to RLS: it sees only what the policies allow. The `service_role` key operates with `BYPASSRLS`: it ignores every policy, reads and writes any row of any table — including those of XHUB IRON (`iron_*`, `world_*`) that share the same `wuchsslgbqlhyxljsmxi` instance. If the `service_role` appears in the client, an attacker not only compromises MOONKEY: they pivot to every project living in that DB. Hard rule: the `service_role` NEVER enters a frontend repo, nor a `.astro`, nor an `import.meta.env.PUBLIC_*`. Its place is a backend with a session (an Edge Function, a server process) or your local keychain — never the bundle.

The most common leak vector is not pasting the key into code: it is the environment-variable prefix. In Astro, `import.meta.env.PUBLIC_X` is INLINED into the client bundle at build time; `import.meta.env.X` (without `PUBLIC_`) exists only in the server/build context and is NOT exposed. Vite/Astro do this by design. The classic mistake: naming a variable `PUBLIC_SUPABASE_SERVICE_KEY` 'so the component can see it.' You succeed — and you hand it to the world. Your decision tree for naming an env var: can this key be public? Yes → `PUBLIC_`. No → no prefix, and on top of that ask yourself what it is doing in an SSG project (it probably shouldn't be there at all).

Build the inventory. Open your project and classify every credential in a three-column table: name · class (public/private) · where it lives today. In MOONKEY the real entries are: `PUBLIC_SUPABASE_URL` (public, identifier), `PUBLIC_SUPABASE_ANON_KEY` (public, bounded by RLS), and the `service_role` (private — it must NOT exist in this repo at all; if you need it for a migration script, it lives in your local shell or in a CI secret, never in `src/`). Add any third-party token you touch: transactional-email keys, webhooks, server-side analytics tokens. Every private row sitting on the wrong side is an incident waiting to happen.

The control that closes the loop is automatic detection, because human discipline fails. Before every commit you want something to scream if a private key slips through. The minimal pattern: a pre-commit grep that looks for the signature of a `service_role` (Supabase service JWTs carry `"role":"service_role"` in their base64 payload) and for words like `service_role`, `BEGIN PRIVATE KEY`, `sk-`. For real repos use `gitleaks` or `trufflehog` as a hook. And if a key has ALREADY leaked in a past commit: deleting it from the code is not enough — it is still in the git history. You have to ROTATE it in the Supabase panel (Settings → API → roll key) and rewrite history if it was a hard secret. Rotation is mandatory; deleting the file revokes nothing.

Close with the Espejo case, which adds an ethical axis, not just a technical one. Espejo is multi-tenant: each influencer-tenant has its own data. There the 'public vs private' question doubles: on top of infrastructure keys, you handle consented personal data of end users. The service key that could read ACROSS tenants is the product's most sensitive secret — its leak is not merely technical, it violates the data boundary the project memory marks as non-negotiable. Cross-cutting lesson: a private credential is not defined by its format, it is defined by what it unlocks. Always measure it by its blast radius.

Adopt the correct mental frame: in an SSG, the client is not your application, it is enemy territory. The attacker has your complete bundle (it's public), your `PUBLIC_SUPABASE_URL`, your anon key, and the knowledge that Postgres sits behind it. They don't need to 'hack' your JavaScript: they read it. They can instantiate their own Supabase client with YOUR anon key from a Node console and talk directly to your DB, never passing through your HTML. Therefore, any security logic living in `.astro` or in client JS — an `if (user.role === 'admin')`, a component that hides itself — is UX, not security. MOONKEY's CLAUDE.md says it literally: `admin.astro` is UX; security does NOT depend on that gate.

Enumerate the concrete attack surface. For MOONKEY there are five tables reachable with the anon key: `profiles, progress, feedback, leads, proofs`. The attacker, with a client authenticated via magic-link (anyone can sign up), will try the obvious: `supabase.from('profiles').select('*')` to dump ALL profiles; `supabase.from('leads').select('*')` to steal the lead list; an `update` on their own `profiles` row setting `role = 'admin'` or `founder_badge = true` to self-escalate; an `insert` into `proofs` with someone else's `user_id` to fake another user's progress. Each of these is an attack hypothesis that YOU must refute with a policy. If you haven't explicitly refuted it, assume it works.

Understand why RLS is the authority and how MOONKEY's real defense is composed. SELECT on `profiles/progress/feedback/proofs` is self-or-admin: the policy permits the row only if `user_id = auth.uid()` OR `is_admin()`. `is_admin()` is SECURITY DEFINER — it runs with the owner's permissions, checking whether the caller is an admin without the caller being able to tamper with that decision. Result: a non-admin who asks for `select('*')` doesn't get an error, they get THEIR rows and nothing more — the mass dump returns a single row, their own. Privilege escalation is cut off by a trigger (`guard_privileged_profile_columns`): `role` and `founder_badge` are immutable for non-admins, so the `update role='admin'` is rejected at the DB level. The foreign insert is cut off by `proofs_insert_self`: it requires a session and ties `user_id = auth.uid()`.

The operational consequence of the model: verify by impersonation, not by reading code. A policy that 'looks' correct can have a hole (a misplaced OR, a USING without WITH CHECK that lets updates through). The only proof that counts is putting yourself in the attacker's shoes: authenticate as a normal user and EXECUTE the attacks enumerated above against the real DB, confirming that each one returns empty or an error. CLAUDE.md mentions that read isolation is 'verified by impersonation' — that is the standard. 'I wrote the policy' is not evidence; 'I tried the attack as user X and it did not read user Y's row' is.

Don't forget the non-RLS vectors. The threat model of an SSG also includes: (a) the USING vs WITH CHECK distinction — USING filters which rows you see/affect, WITH CHECK validates the resulting state of an INSERT/UPDATE; forgetting WITH CHECK on an UPDATE lets you move a row to someone else's user_id. (b) RPCs: a badly written SECURITY DEFINER function is a hole that ignores RLS by design — MOONKEY restricts them to `authenticated` and revokes dangerous operations. (c) TRUNCATE: revoked from anon/authenticated, because RLS does not protect against TRUNCATE (it wipes the whole table without evaluating per-row policies). (d) Leakage through errors and through columns: a SELECT that returns a sensitive column you forgot to exclude.

Land it with the project contrast to fix the principle. XNLAB is a brand site with almost no backend: its surface is minimal — contact forms, headers, maybe a capture endpoint. There the threat model is 'spam and XSS in the little input it accepts,' not DB escalation. XHUB IRON lives in the SAME DB as MOONKEY but is the founder's operational panel: its `iron_*` tables must not be readable by a MOONKEY user — the same RLS engine has to keep TWO products sharing Postgres without one reading the other (that's OS-05). The threat model is not generic: it is derived from WHAT data exists, WHO can authenticate, and WHAT the infrastructure shares. Always start by inventorying those three axes.

The governing principle is Popperian: a security finding is not true because you assert it, it is true because you TRIED to refute it and couldn't. Invert the natural flow. When you think you've found a vulnerability ('anyone can read `leads`'), your next step is not to report it — it is to try to prove yourself wrong: is there a policy that prevents it? did I test it as the correct role? did the `select` return real data or empty? Only when your refutation attempt fails does the finding rise to confirmed. And in reverse: when you believe something is safe ('the self-or-admin policy protects `profiles`'), your job is to attack it until it breaks or survives.

Define the two errors you are hunting with clear names. False positive: you report a vulnerability that doesn't exist — e.g. you scream 'the anon key is exposed in the bundle' when the anon key is public by design and the DB is protected by RLS (see OS-01). You burn trust; next time nobody believes you. False negative: you assume something is safe when it isn't — e.g. 'the admin panel is hidden, we're fine' without testing direct DB access. You leave the breach open. Adversarial auditing exists to minimize BOTH, and the tool for both is the same: the executable proof.

The concrete method: for each finding, produce a reproducible proof, not an assertion. 'Reproducible' means: a command or snippet anyone can run and see the same result. To confirm that `profiles` CANNOT be dumped: a script that authenticates as user_A, runs `.from('profiles').select('*')`, and prints the row count — expected 1 (only their own). To confirm that privilege escalation is closed: as user_A, `.from('profiles').update({ role: 'admin' }).eq('id', myId)` and print the error the guard trigger returns. If the attack returns what you expected (empty/error), the control is verified. If it returns data, you have a REAL finding — and it already comes with its PoC attached.

Work with Supabase's own tool but without surrendering to it. `get_advisors` (security lints) points you to tables without RLS, permissive policies, SECURITY DEFINER functions without a fixed search_path. Treat it as a generator of HYPOTHESES, not verdicts. An advisor that says 'function X is SECURITY DEFINER with mutable search_path' is a risk hypothesis: go to the function, see whether an attacker can plant an object in a schema it resolves first. Sometimes it's exploitable, sometimes it isn't given the context. The advisor saves you the initial sweep; the refutation is yours to do. CLAUDE.md already institutionalizes it: RLS changes = versioned migration + `get_advisors` afterward. That second part is adversarial auditing turned into routine.

Watch out for the false negatives of the method itself: the attack that 'passes' may pass for the wrong reason. If your `select('*')` on `profiles` returns empty, is it because RLS blocked it, or because your test session expired / you weren't authenticated / the table was empty? A control is never accepted as good without a CONTRAST case: prove that the same query DOES return data when it should (as admin, or as the row's owner). A security test without its positive control is a false negative disguised as success. Always verify that your weapon fires before concluding the target is immune.

Land the rigor in the projects where getting it wrong hurts most. In XCAP, adversarial auditing is central doctrine: the project memory describes it as 'anti-fabrication by construction' — a system that records forecasts and calibrates them against real outcomes, refuting its own predictions instead of believing them. You apply that same muscle to security: don't sign off on 'the ledger is read-only' until a write attempt with the ingest credential fails before your eyes. In Espejo, where the personal-data boundary is ethical as well as technical, an audit that says 'there is no leak between tenants' without a PoC of one tenant trying to read another is NOT an audit: it's a wish. Refute first, always, especially when the result you expect is the comfortable one.

Internalize the DCRP sequence: Detect → Contain → Recover (rollback) → Post-mortem. Most people's mistake is jumping straight to 'fix,' touching production blindly. Order matters because each phase protects the next: if you don't detect well, you contain the wrong thing; if you don't contain, the damage grows while you recover; if you recover without understanding, you repeat the incident. And the zeroth rule, written in any serious runbook: during an incident you do NOT improvise on production. You execute predefined steps. Creativity is for the post-mortem, not for 3 a.m. with the site down.

Detecting means knowing WHAT is broken and SINCE WHEN. In MOONKEY (Cloudflare Pages + Supabase) your signals are: the Cloudflare deploy (which commit is live? did the last build pass?), the Supabase logs (`get_logs` per service: api, postgres, auth — there you see policy errors, failing queries, spikes of 4xx/5xx), and the user complaint (often the first signal). The first thing you establish is the timeline: what changed just before? The incident almost always correlates with a recent deploy or migration. 'It started after the 16:20 merge' narrows the search space from hours to minutes.

Containing means stopping the bleeding before treating the wound. Key question: is this corrupting data RIGHT NOW? If a bug is writing bad rows into `proofs` or `progress`, every minute you take multiplies the cleanup. Containment options by severity: revert the Cloudflare deploy to the previous one (one click — Cloudflare Pages keeps previous deploys and allows instant rollback, this is your fastest and safest lever), disable the broken feature, or in the extreme case of destructive writes, cut access at the policy level. Containing is NOT fixing: it is freezing the damage in its current state to buy yourself time to think.

Recovering requires distinguishing TWO rollbacks people confuse and that carry opposite risks. (1) CODE rollback: returning to a previous deploy/commit. It's cheap, reversible, and almost always safe — in Cloudflare Pages it's immediate. This is your first option. (2) SCHEMA/DATA rollback: reverting a migration or restoring data. It's DANGEROUS and irreversible: a `down` migration that runs `DROP COLUMN` deletes data that was perhaps only corrupted, not lost. Rules: never run a data rollback without a confirmed backup/PITR first; prefer a FORWARD migration that corrects (a new `UPDATE` that repairs the bad rows) over a destructive `down`; every Supabase migration is versioned (`apply_migration`) precisely so the state is reconstructible. If the doubt is 'code or data rollback,' ALWAYS start with code: it is often enough and doesn't touch the DB.

The post-mortem is where the incident pays its debt by becoming learning. It is BLAMELESS: the goal is not who broke it, but what about the SYSTEM allowed a normal human to break it. Structure: timeline (what happened and when, minute by minute), impact (how many users, what data, how long), root cause (the '5 whys' down to the system failure, not the person's failure), and corrective actions with owner and date. Every incident must produce at least one control that would have PREVENTED or DETECTED it sooner — a test, a CI advisor, an alert. If the post-mortem changes nothing about the system, it wasn't a post-mortem, it was a confession.

Anchor the runbook in each project's reality, because severity and levers change. In XCAP there is a sacred capital invariant — capital accumulates across sessions and only moves on realized closes, never resets. An incident touching that invariant is NOT fixed with a blind data rollback: you would erase real accounting history. There the runbook forces a forward correction and verification of the invariant before touching anything. In Espejo, multi-tenant, an incident has an extra question in the detection phase: did the failure cross the boundary between tenants? If a bug exposed one tenant's data to another, beyond the rollback there is an obligation to contain the leak. The generic runbook gives you the backbone; each project's invariants give you the rules you NEVER violate during the panic.

Start from the concrete fact: the `wuchsslgbqlhyxljsmxi` instance hosts TWO products. MOONKEY uses `profiles, progress, feedback, leads, proofs`. XHUB IRON uses `iron_*, world_*, focus_*, daily_focus_history`. They share the same Postgres, the same `auth.users`, and — crucially — the SAME anon key is used to talk to the DB. This means MOONKEY's public client technically CAN attempt `from('iron_command').select('*')`. The only reason it doesn't work has to be a hard barrier in the DB, not the CLAUDE.md note that says 'don't touch them.' That note protects against YOU editing them by mistake while writing code; it does not protect against an attacker who already has your anon key.

Understand the isolation layers from weakest to strongest. (1) Convention (`iron_*` names vs no prefix): it organizes, it doesn't isolate — an attacker ignores conventions. (2) Per-table RLS: each XHUB table must have policies that grant access only to its own role/user; an `iron_*` table without RLS or with a permissive policy is a direct leak. (3) Separate schemas: putting XHUB in its own Postgres schema and NOT exposing it in the PostREST API (the `db.schema` Supabase publishes) is a stronger barrier — what isn't in the exposed schema, the client can't even see. (4) Roles/grants: REVOKE from `anon`/`authenticated` on foreign tables/objects. A robust defense STACKS these layers; don't trust a single one.

For the MOONKEY↔XHUB case, the audit question is direct: can an authenticated MOONKEY user read a row of `iron_*`? Refute it (OS-03): authenticate as a normal MOONKEY user and run `supabase.from('iron_command').select('*')`. The correct outcome is a permission error or empty by RLS. If it returns data, you have a critical isolation leak. The project memory says health is computed and the model is 'open RLS, computed health' for the HUB — that means you must verify EXPLICITLY that 'open' for the HUB does not mean 'open to MOONKEY clients.' Cross-project isolation = each table responds only to its legitimate owner, proven by cross-impersonation.

The second axis is multi-tenancy WITHIN a project, which is Espejo. Here it's not two products, it's N tenants (each influencer) in the same tables. Isolation becomes per-row: each row carries a `tenant_id`, and the RLS policy requires `tenant_id` to match the caller's tenant. The attack to refute: tenant A tries to read/write rows with B's `tenant_id`. The classic trap is the UPDATE without WITH CHECK: USING limits which rows A sees, but without WITH CHECK, A could reassign a row to B or create one with a foreign `tenant_id`. Another trap: a SECURITY DEFINER RPC that receives `tenant_id` as a parameter and doesn't validate that it matches `auth.uid()` — a hole that crosses every tenant. Multi-tenant isolation is only real if EVERY policy of every operation (SELECT/INSERT/UPDATE/DELETE) anchors the tenant to the caller's identity, not to a parameter the client controls.

There are crossing vectors that aren't SELECT and that people forget. `auth.users` is shared: a user is the same in MOONKEY and in XHUB — make sure that having an account in one grants no roles in the other (in MOONKEY, `role` and `founder_badge` are immutable for non-admins precisely so that signing up doesn't escalate privileges across projects). RPCs and SECURITY DEFINER functions are global objects: an XHUB function executable by `authenticated` is invokable from the MOONKEY client — restrict it by role or validate the context inside. Shared triggers and sequences, views that JOIN tables from both products, and `get_advisors` flagging missing RLS on any project table: all of that is crossing surface. Isolation is not a policy, it is a property of the whole system.

Close with the operational discipline that keeps isolation alive over time. Every schema change enters as a versioned migration (`apply_migration`), never as a manual edit in the panel — so the security state is reconstructible and auditable. After every migration that touches tables or policies, run `get_advisors` to catch disabled RLS or permissive policies introduced unintentionally. And keep an isolation test as part of the runbook: a script that, as a MOONKEY user, tries to read each foreign prefix (`iron_*`, `world_*`) and fails the build if any returns rows. Cross-project isolation is not 'configured once'; it is re-verified on every change, because an innocent migration can open a door that had been closed for months.

The three streams of a live system. Logs = the diary of what happens (requests, queries, auth). Errors = what breaks silently on the client side, where your server logs don't reach. Cost = the counter that runs even when nobody uses the system. They are three distinct pipes with three distinct tools; confusing them is why people think they're monitoring and they aren't.

Observability in an SSG is not the same as in a server. MOONKEY is static: there is NO server-side handler to put a logger in. Your telemetry lives in three places outside your HTML: (1) Cloudflare Pages Analytics (requests, bandwidth, edge errors), (2) Supabase logs (Dashboard → Logs: API, Postgres, Auth — every query passing through PostgREST and every login leaves a trace), and (3) the user's browser (JS errors that only you don't see). Accept it: in an SSG, much of what fails happens on a machine you don't control.

What to actually watch, not everything. The temptation is to log everything and read nothing. Define a handful of signals that matter: the 4xx/5xx error rate at the edge (a spike of 401/403 = RLS rejecting or an attacker probing), Auth errors in Supabase (failed magic-link attempts), the latency of slow queries (Supabase flags the ones that take long), and the volume of rows read (an unfiltered query that suddenly returns 10,000 rows is a cost bug). A metric that triggers no decision is noise; delete it.

Client-side error tracking: the blind spot. In a static site, a TypeError in your JS breaks the user's experience and NOTHING reaches you — no server log, no alert. You need to capture errors in the browser and send them out. The honest minimum: a window.addEventListener('error', ...) and ('unhandledrejection', ...) that does a fetch to your own endpoint (a Supabase Edge Function that inserts into a logs table with insert-only RLS). The robust version: Sentry or similar with its free SDK up to a certain volume. The rule: if you don't capture client errors, you're operating blind in half your system.

The real cost of operating, broken down. The Supabase free tier has concrete ceilings that, when crossed, either cut you off or charge you: DB rows, egress bandwidth, storage, Auth Monthly Active Users, and Edge Function invocations. Cloudflare Pages is generous but has limits on builds/month and on requests in functions. And the spend that scales most and is seen least: AI tokens. Every call to the Claude API is billed by input and output tokens, at different rates; a prompt with a giant CLAUDE.md or an agent loop with no cap can multiply the cost without changing 'what the app does.' The operator knows their three bills: DB, edge and tokens.

Why surprise bills arrive, and how not to receive them. Three classic causes: (1) an endpoint or Edge Function with no rate-limit that someone discovers and hammers — usage = money; (2) a query with no LIMIT and no index that scans the whole table on every load; (3) an AI agent loop with no token cap nor iteration cap (the XCAP capital invariant exists in part because of this: loops that accumulate learning, not cost). The defense is not watching the bill at month's end, it's putting the brakes on BEFORE: budgets with alerts at 50/80/100%, hard limits where the platform offers them, and explicit ceilings on max_tokens and iterations in every AI call.

Alert on what matters, without alert fatigue. An alert that fires every hour is ignored within a week. The criterion: alert only on the actionable and the urgent. Good: 'the month's spend exceeded 80% of the budget,' 'the 5xx rate went above 1%,' 'client errors > X in 10 min.' Bad: 'there was a request.' Configure the budget alerts that Supabase, Cloudflare and the Anthropic console already give you — they're free and they're your first line. Every alert must answer 'what do I do when it fires'; if there's no action, it's not an alert, it's anxiety.

Close the loop with incident response (OS-04). Observing without a plan to react is just watching it burn. The panel and alerts of this module are the sensors; the rollback and incident response of OS-04 are the actuators. A well-operated system connects the two: the alert wakes you, the logs tell you what happened, and the response plan tells you what to do. Observability + cost + response = operating for real, not praying.