From 91daabc45c89ff11113b57892dfc906a38729f41 Mon Sep 17 00:00:00 2001 From: feruzm Date: Thu, 20 Aug 2026 19:07:32 +0000 Subject: [PATCH 1/3] Bring the docs in line with the tracked origin config README and docs/cache both still filed the origin nginx as living somewhere else. It has been in this repo since #1574, with a public-repo contract and a CI guard, and neither was mentioned anywhere a reader would look. docs/cache/nginx.md carried a hand-copied server block that had drifted into contradicting the tracked file: it showed X-Cache-Tier with always, which the vhost explicitly forbids because the upstream already sets that header and adding it emitted the header twice on every response, and it showed always on X-Cache-Status, which the real config deliberately omits. Replaced with the stable directives and a pointer to the file, so there is one copy rather than two that disagree. CLAUDE.md listed only apps and packages, and described pnpm test/lint/ typecheck as the checks, while three script audits gate the typecheck workflow and pnpm test runs none of them. The origin README now records why HEAD must not be answered in location /, and why the always flags are the other half of that fix rather than a tidy-up: those statuses were unreachable while every HEAD was a fabricated 200, so the deletion alone would have dropped HSTS from error responses. Also: the sitekey is inlined at build time, which the env template did not say, and the tenant nginx comment predates the Turnstile widget the form now carries. --- .github/workflows/typecheck.yml | 5 ++ CLAUDE.md | 22 +++++++ README.md | 5 +- .../hosting/nginx-multi-tenant.conf | 14 ++-- apps/web/.env.template | 4 ++ docs/cache/README.md | 7 +- docs/cache/nginx.md | 64 ++++++++----------- infra/origin/README.md | 24 ++++++- 8 files changed, 98 insertions(+), 47 deletions(-) diff --git a/.github/workflows/typecheck.yml b/.github/workflows/typecheck.yml index b3493d0e94..8095955e18 100644 --- a/.github/workflows/typecheck.yml +++ b/.github/workflows/typecheck.yml @@ -51,6 +51,11 @@ jobs: - name: Slim-entry convention audit run: node scripts/slim-entries-audit.mjs --fail + # This repo is PUBLIC and infra/origin holds production vhosts, so the audit keeps + # thresholds, source addresses, inline allowlists and secret markers out of them, and + # rejects a wildcard include on a file carrying a protective directive (nginx accepts + # one matching no files and reloads clean, i.e. it fails OPEN). --self-test runs first + # so a rule edited into uselessness fails here rather than passing silently. - name: Audit committed origin nginx config run: | node scripts/origin-config-audit.mjs --self-test diff --git a/CLAUDE.md b/CLAUDE.md index b209115d76..57fef5999e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -65,6 +65,22 @@ pnpm lint pnpm typecheck ``` +**Three script audits also gate `.github/workflows/typecheck.yml`**, and `pnpm test` runs none +of them: + +```bash +node scripts/icon-tsx-audit.mjs --fail # icon sizing, see docs/icons.md +node scripts/icon-scss-audit.mjs # also fails if a retired SCSS rule reappears +node scripts/slim-entries-audit.mjs # feed payload invariants +node scripts/origin-config-audit.mjs --self-test && node scripts/origin-config-audit.mjs --fail +``` + +`origin-config-audit` keeps thresholds, source addresses, inline allowlists, secret markers and +fail-open wildcard includes out of `infra/`. It reads **comments** too — an earlier version +stripped them and reported a clean run while every rate sat in prose two lines above. Run +`--self-test` first: it proves each rule still fires, so a regex edited into uselessness is +caught in the same step that relies on it. + ### Running Single Tests ```bash @@ -99,6 +115,12 @@ pnpm publish:ui - **packages/wallets** - Multi-chain wallet management (`@ecency/wallets`) - **packages/render-helper** - Markdown rendering utilities (`@ecency/render-helper`) - **packages/ui** - Shared UI component library (`@ecency/ui`) +- **infra/origin** - The web origin nginx vhosts (`eu`/`us.ecency.com.conf`), tracked since + 2026-08-20. ⛔ **This repo is public**: structure is committed, thresholds and addresses are + not — they live in host-only includes. Enforced by `scripts/origin-config-audit.mjs` in CI. + ⛔ CI does **not** deploy these; they are applied by hand to both boxes, and the rule is + apply-and-reload BEFORE committing so the tracked copy matches what is running. See + `infra/origin/README.md`. All packages use `workspace:*` protocol for local dependencies. The main app transpiles workspace packages during build (configured in `next.config.js`). diff --git a/README.md b/README.md index 2d035b4509..cf82634d47 100644 --- a/README.md +++ b/README.md @@ -141,8 +141,9 @@ CF-Cache-Status: HIT ### Infra configuration -Nginx and CF worker configs live in the infra repo; their behaviour is -documented here in `docs/cache/`. The rules are simple: **respect origin +The web origin nginx vhosts live in this repo at `infra/origin/` (tracked since +2026-08-20), guarded by `scripts/origin-config-audit.mjs` in CI; the CF worker config +lives in the infra repo. Behaviour is documented here in `docs/cache/`. The rules are simple: **respect origin `Cache-Control`**, **bypass on `active_user` cookie** and **preserve `x-cache-tier`** in the response headers. diff --git a/apps/self-hosted/hosting/nginx-multi-tenant.conf b/apps/self-hosted/hosting/nginx-multi-tenant.conf index 21120b88cc..8815193a80 100644 --- a/apps/self-hosted/hosting/nginx-multi-tenant.conf +++ b/apps/self-hosted/hosting/nginx-multi-tenant.conf @@ -128,8 +128,11 @@ server { # THIS BOX (host.docker.internal, an extra_hosts entry in the compose file), # deliberately not through public ecency.com: a Cloudflare hop would make # every tenant's signup look like one datacenter client (one IP for consent - # records, one bucket for rate limits, and a bot challenge would break the - # form outright). The reader's address travels as CF-Connecting-IP, set from + # records, one bucket for rate limits, and an EDGE bot challenge would break + # the form outright). Note the form DOES carry a Turnstile widget of its own + # since 2026-08-20, solved in-page before submit and verified by the relay; + # that is a different thing from an edge interstitial, which the reader could + # not clear. The reader's address travels as CF-Connecting-IP, set from # X-Real-IP, which the HOST's edge vhost asserted from its own remote_addr; # the origin's /api/ location forwards exactly that header pair to the app. # Managed instances only, by construction: only they sit behind this nginx. @@ -247,8 +250,11 @@ server { # THIS BOX (host.docker.internal, an extra_hosts entry in the compose file), # deliberately not through public ecency.com: a Cloudflare hop would make # every tenant's signup look like one datacenter client (one IP for consent - # records, one bucket for rate limits, and a bot challenge would break the - # form outright). The reader's address travels as CF-Connecting-IP, set from + # records, one bucket for rate limits, and an EDGE bot challenge would break + # the form outright). Note the form DOES carry a Turnstile widget of its own + # since 2026-08-20, solved in-page before submit and verified by the relay; + # that is a different thing from an edge interstitial, which the reader could + # not clear. The reader's address travels as CF-Connecting-IP, set from # X-Real-IP, which the HOST's edge vhost asserted from its own remote_addr; # the origin's /api/ location forwards exactly that header pair to the app. # Managed instances only, by construction: only they sit behind this nginx. diff --git a/apps/web/.env.template b/apps/web/.env.template index 4c43543747..65fd6617ce 100644 --- a/apps/web/.env.template +++ b/apps/web/.env.template @@ -56,5 +56,9 @@ NEWSLETTER_SERVICE_TOKEN= # deployment that has not configured the newsletter at all never reaches that code, since # the route already 503s without NEWSLETTER_API_URL and NEWSLETTER_SERVICE_TOKEN. # specs/deploy/newsletter-wiring pins it on the web service in both compose files. +# ⛔ The SITEKEY is inlined at BUILD time, not read at runtime, so it must reach the image +# build: apps/web/Dockerfile takes it as an ARG and both deploy workflows pass it as a build +# arg. Setting it only in the runtime environment has no effect, and the client silently falls +# back to the literal in features/shared/turnstile.tsx. NEXT_PUBLIC_TURNSTILE_SITEKEY= TURNSTILE_SECRET= diff --git a/docs/cache/README.md b/docs/cache/README.md index 0c7f39dfe7..b99e666daa 100644 --- a/docs/cache/README.md +++ b/docs/cache/README.md @@ -150,9 +150,14 @@ Implemented in this repo: - Entry-page TTL refined by post age via L1 Map + per-host Redis L2 - `scripts/purge-cache.sh` supports manual DMCA / moderation invalidation +Tracked here, applied by hand (CI does NOT deploy them): + +- Nginx (`ssrcache` zone) on each origin host — the vhosts are in `infra/origin/` + since 2026-08-20; see [nginx.md](./nginx.md) and `infra/origin/README.md` for the + public-repo contract that keeps thresholds and addresses off the record + Operated outside this repo: -- Nginx (`ssrcache` zone) on each origin host — see [nginx.md](./nginx.md) - Cloudflare worker `ecency-geo-router` — see [cloudflare-worker.md](./cloudflare-worker.md) Before production rollout of changes here, verify `x-cache-tier` values diff --git a/docs/cache/nginx.md b/docs/cache/nginx.md index 16de9ead8b..a53600915a 100644 --- a/docs/cache/nginx.md +++ b/docs/cache/nginx.md @@ -1,7 +1,7 @@ # Nginx Cache Alignment -Nginx (`ssrcache` zone) sits between the CF worker and vision_web. It runs -on each origin server (eu/us/asia.ecency.com). Because the worker has +Nginx (`ssrcache` zone) sits between the CF worker and vision_web. It runs on each +origin server (EU and US; the asia origin was decommissioned). Because the worker has already keyed on auth-class and only forwards cacheable requests, **nginx does NOT need to gate on `active_user` cookie itself** — origin's `Cache-Control` is the source of truth. @@ -25,45 +25,33 @@ proxy_cache_path /var/cache/nginx/ssr levels=1:2 ## Per-host config +⛔ **The vhosts are tracked: read `infra/origin/eu.ecency.com.conf` and +`us.ecency.com.conf` rather than a snippet here.** This section used to carry a +hand-copied `server { … }` block, which drifted and began contradicting the real config +— it showed `add_header X-Cache-Tier $upstream_http_x_cache_tier always;`, which the +tracked file explicitly forbids because the upstream already sets that header and nginx +proxies it through, so adding it emitted the header **twice on every response**. It also +showed `always` on `X-Cache-Status`, which the real config deliberately omits: that value +is a cache diagnostic, not something we owe an error response. + +What matters conceptually, and is stable: + ```nginx -# See "Why the bot UA class is in the cache key" below. -map $http_user_agent $html_limited_bot { - default ""; - "~*(Googlebot|[\w-]+-Google|Google-[\w-]+|googleweblight|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|TelegramBot|WhatsApp|SkypeUriPreview|Yeti)" "|htmlbot"; -} - -server { - listen 80; - server_name eu.ecency.com; # or us per host - - location / { - proxy_cache ssrcache; - proxy_cache_key "$request_uri$html_limited_bot"; - # Defer to origin Cache-Control. Fallback for responses without it. - proxy_cache_valid 200 0; - proxy_cache_valid any 30s; - # Stale-while-revalidate semantics - proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504; - proxy_cache_background_update on; - proxy_cache_lock on; - proxy_cache_lock_timeout 5s; - - # Observability - add_header X-Cache-Status $upstream_cache_status always; - add_header X-Cache-Tier $upstream_http_x_cache_tier always; - - proxy_pass http://127.0.0.1:3000; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_connect_timeout 5s; - proxy_send_timeout 20s; - proxy_read_timeout 20s; - } -} +proxy_cache ssrcache; +proxy_cache_key "$request_uri$html_limited_bot"; # see the next section +proxy_cache_valid 200 0; # defer to origin Cache-Control +proxy_cache_valid any 30s; +proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504; +proxy_cache_background_update on; +proxy_cache_lock on; ``` +The `$html_limited_bot` map lives at `http` level in `/etc/nginx/nginx.conf`, which is +**not** tracked (see `infra/origin/README.md`, "What these files depend on"). Its pattern +list must stay in step with `htmlLimitedBots` in `next.config.js`: if the app serves a +blocking render to an agent the map does not classify, that response and a browser's +share one cache entry. + ## Why the bot UA class is in the cache key `htmlLimitedBots` in `apps/web/next.config.js` makes Next.js render metadata diff --git a/infra/origin/README.md b/infra/origin/README.md index 7593bbf9b0..dcf930cc33 100644 --- a/infra/origin/README.md +++ b/infra/origin/README.md @@ -42,8 +42,28 @@ What IS committed and deliberately so: **Comments count as published.** A threshold quoted in a comment is as disclosed as one in a directive, so the vhost comments name the include rather than the number. The audit -reads comments for exactly this reason: an earlier version stripped them and reported a -clean run while every rate sat in prose two lines above. +(`scripts/origin-config-audit.mjs`, run from `.github/workflows/typecheck.yml` as +`--self-test` then `--fail`) reads comments for exactly this reason: an earlier version +stripped them and reported a clean run while every rate sat in prose two lines above. + +## Why `location /` does not answer HEAD itself + +It used to: `if ($request_method = HEAD) { add_header Cache-Control no-store; return 200; }`. +That made every HEAD report a live page whatever the truth was — `GET /@good-karma/points` +answered 307 while `HEAD` answered 200 — so link checkers, uptime probes and crawlers were all +told the wrong thing. Removed 2026-08-20 (#1575). + +The `always` on the CORS and security `add_header`s is the **other half of the same fix**, not +a tidy-up. While every HEAD was a fabricated 200 those headers applied, because 200 is in +`add_header`'s default status list; deleting the block alone would have made HEAD reach real +404/429/5xx responses with the headers gone. `always` alone fixes nothing either, since an +`add_header` nested inside the `if` suppresses inheritance regardless. `X-Cache-Status` +deliberately keeps its default list: it is a cache diagnostic, not something we owe an error +response. + +⛔ Verify this against the origin, never the public hostname. Cloudflare normalises the request +and already answered 307 with headers, so `curl -I https://ecency.com/...` reports the bug as +already fixed. ## What these files depend on From fa62b353850e6207d807bb8bf6380b58fade0aa4 Mon Sep 17 00:00:00 2001 From: feruzm Date: Thu, 20 Aug 2026 19:42:39 +0000 Subject: [PATCH 2/3] Put back the bot map: it is canonical, not a stale copy Removing it broke the parity test that reads it. social-bot-metadata parses the $html_limited_bot map out of docs/cache/nginx.md and asserts term-for-term agreement with htmlLimitedBots in next.config.js, because drift there is invisible in dev and in production serves a browser-primed page to a crawler from the wrong cache namespace (#1257). I judged the whole block a stale hand-copy. Half of it was: the server stanza had drifted into contradicting the tracked vhost. The map had not, and is deliberately duplicated BECAUSE it is enforced. I even wrote the invariant into the replacement text while deleting the thing that enforces it, so the section now says plainly that the block is machine read and names the spec that reads it. Also: the audit list said three and listed four, and omitted --fail on slim-entries, so following it locally would report and exit 0 while CI failed. And the cache verification recipe claimed logged-in /discover always bypasses, contradicting cache-policy.ts (tier list, s-maxage 300) and the README's own note eighty lines above that only the mute-filtered tiers go private. Replaced with a tier that genuinely bypasses. --- CLAUDE.md | 14 ++++++++------ README.md | 12 +++++++++++- docs/cache/nginx.md | 30 +++++++++++++++++++++++------- 3 files changed, 42 insertions(+), 14 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 57fef5999e..3cb72019ea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -65,14 +65,16 @@ pnpm lint pnpm typecheck ``` -**Three script audits also gate `.github/workflows/typecheck.yml`**, and `pnpm test` runs none -of them: +**Four script audits also gate `.github/workflows/typecheck.yml`**, and `pnpm test` runs none +of them. ⛔ Copy the flags exactly: without `--fail` these REPORT and exit 0, so a violation +passes locally and fails in CI. ```bash -node scripts/icon-tsx-audit.mjs --fail # icon sizing, see docs/icons.md -node scripts/icon-scss-audit.mjs # also fails if a retired SCSS rule reappears -node scripts/slim-entries-audit.mjs # feed payload invariants -node scripts/origin-config-audit.mjs --self-test && node scripts/origin-config-audit.mjs --fail +node scripts/icon-scss-audit.mjs # also fails if a retired SCSS rule reappears +node scripts/icon-tsx-audit.mjs --fail # icon sizing, see docs/icons.md +node scripts/slim-entries-audit.mjs --fail # feed payload invariants +node scripts/origin-config-audit.mjs --self-test # prove the rules still fire, THEN enforce +node scripts/origin-config-audit.mjs --fail ``` `origin-config-audit` keeps thresholds, source addresses, inline allowlists, secret markers and diff --git a/README.md b/README.md index cf82634d47..67bfb24fe7 100644 --- a/README.md +++ b/README.md @@ -126,10 +126,20 @@ expires. # Anonymous — should HIT after the first request curl -sI https://ecency.com/discover | grep -iE 'cache|tier' -# Logged-in — should always BYPASS +# Logged-in on the SAME path — still cached. /discover is the `list` tier, and most +# tiers are auth-class-equivalent, so logged-in users share one edge entry. curl -sI --cookie "active_user=alice" https://ecency.com/discover | grep -iE 'cache|tier' + +# Logged-in on a MUTE-FILTERED tier — this is what actually bypasses: +# `feed`, `feed-created` and `profile-feed` emit `private, no-store` when the +# active_user cookie is present, because each user's mutes filter it differently. +curl -sI --cookie "active_user=alice" https://ecency.com/@ecency/feed | grep -iE 'cache|tier' ``` +`curl -sI` sends HEAD, which reaches the app and returns the real status only since +#1578 — before that `location /` short-circuited every HEAD with a fabricated 200, so +these recipes reported nothing useful. + Expected headers on an anonymous hit: ```http diff --git a/docs/cache/nginx.md b/docs/cache/nginx.md index a53600915a..bc3474f6e4 100644 --- a/docs/cache/nginx.md +++ b/docs/cache/nginx.md @@ -25,8 +25,29 @@ proxy_cache_path /var/cache/nginx/ssr levels=1:2 ## Per-host config -⛔ **The vhosts are tracked: read `infra/origin/eu.ecency.com.conf` and -`us.ecency.com.conf` rather than a snippet here.** This section used to carry a +### The bot map is CANONICAL here — do not remove it + +⛔ This block is **not** documentation of the config, it IS a copy the test suite reads. +`apps/web/src/specs/features/next-middleware/social-bot-metadata.spec.ts` parses it out of +this file and asserts term-for-term parity with `htmlLimitedBots` in `next.config.js`. The +two drifting is invisible in dev, where nothing is cached, and in production means a page +primed by a browser is served to a crawler from the wrong cache namespace with streamed +metadata (#1257). Keep the fenced `nginx` block and the `"~*( … )"` shape — the test matches +on them. + +The live definition sits at `http` level in `/etc/nginx/nginx.conf`, which is not tracked. + +```nginx +map $http_user_agent $html_limited_bot { + default ""; + "~*(Googlebot|[\w-]+-Google|Google-[\w-]+|googleweblight|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|TelegramBot|WhatsApp|SkypeUriPreview|Yeti)" "|htmlbot"; +} +``` + +### The rest of the vhost is tracked, so read it there + +⛔ **Read `infra/origin/eu.ecency.com.conf` and `us.ecency.com.conf` rather than a snippet +here.** This section used to carry a hand-copied `server { … }` block, which drifted and began contradicting the real config — it showed `add_header X-Cache-Tier $upstream_http_x_cache_tier always;`, which the tracked file explicitly forbids because the upstream already sets that header and nginx @@ -46,11 +67,6 @@ proxy_cache_background_update on; proxy_cache_lock on; ``` -The `$html_limited_bot` map lives at `http` level in `/etc/nginx/nginx.conf`, which is -**not** tracked (see `infra/origin/README.md`, "What these files depend on"). Its pattern -list must stay in step with `htmlLimitedBots` in `next.config.js`: if the app serves a -blocking render to an agent the map does not classify, that response and a browser's -share one cache entry. ## Why the bot UA class is in the cache key From 3b545b3ead900ae8e6860ba2ba661dad16febccd Mon Sep 17 00:00:00 2001 From: feruzm Date: Thu, 20 Aug 2026 19:49:54 +0000 Subject: [PATCH 3/3] Attribute the HEAD fix to the PR that made it, not the issue #1575 is the issue describing the fabricated 200; #1578 is the PR that removed it. The line read as though the issue did the work, which sends anyone following it to a report rather than a diff. Cites both now, since each answers a different question: the PR carries the change and the before/after measurements, the issue carries the reproduction. The main README already said #1578, so the two agree again. --- infra/origin/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/infra/origin/README.md b/infra/origin/README.md index dcf930cc33..4b52634b5d 100644 --- a/infra/origin/README.md +++ b/infra/origin/README.md @@ -51,7 +51,8 @@ stripped them and reported a clean run while every rate sat in prose two lines a It used to: `if ($request_method = HEAD) { add_header Cache-Control no-store; return 200; }`. That made every HEAD report a live page whatever the truth was — `GET /@good-karma/points` answered 307 while `HEAD` answered 200 — so link checkers, uptime probes and crawlers were all -told the wrong thing. Removed 2026-08-20 (#1575). +told the wrong thing. Removed 2026-08-20 by **#1578**, which closed #1575 — the PR carries +the change and the before/after measurements, the issue carries the reproduction. The `always` on the CORS and security `add_header`s is the **other half of the same fix**, not a tidy-up. While every HEAD was a fabricated 200 those headers applied, because 200 is in