diff --git a/.github/workflows/typecheck.yml b/.github/workflows/typecheck.yml index b3493d0e94..8095955e18 100644 --- a/.github/workflows/typecheck.yml +++ b/.github/workflows/typecheck.yml @@ -51,6 +51,11 @@ jobs: - name: Slim-entry convention audit run: node scripts/slim-entries-audit.mjs --fail + # This repo is PUBLIC and infra/origin holds production vhosts, so the audit keeps + # thresholds, source addresses, inline allowlists and secret markers out of them, and + # rejects a wildcard include on a file carrying a protective directive (nginx accepts + # one matching no files and reloads clean, i.e. it fails OPEN). --self-test runs first + # so a rule edited into uselessness fails here rather than passing silently. - name: Audit committed origin nginx config run: | node scripts/origin-config-audit.mjs --self-test diff --git a/CLAUDE.md b/CLAUDE.md index b209115d76..3cb72019ea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -65,6 +65,24 @@ pnpm lint pnpm typecheck ``` +**Four script audits also gate `.github/workflows/typecheck.yml`**, and `pnpm test` runs none +of them. ⛔ Copy the flags exactly: without `--fail` these REPORT and exit 0, so a violation +passes locally and fails in CI. + +```bash +node scripts/icon-scss-audit.mjs # also fails if a retired SCSS rule reappears +node scripts/icon-tsx-audit.mjs --fail # icon sizing, see docs/icons.md +node scripts/slim-entries-audit.mjs --fail # feed payload invariants +node scripts/origin-config-audit.mjs --self-test # prove the rules still fire, THEN enforce +node scripts/origin-config-audit.mjs --fail +``` + +`origin-config-audit` keeps thresholds, source addresses, inline allowlists, secret markers and +fail-open wildcard includes out of `infra/`. It reads **comments** too — an earlier version +stripped them and reported a clean run while every rate sat in prose two lines above. Run +`--self-test` first: it proves each rule still fires, so a regex edited into uselessness is +caught in the same step that relies on it. + ### Running Single Tests ```bash @@ -99,6 +117,12 @@ pnpm publish:ui - **packages/wallets** - Multi-chain wallet management (`@ecency/wallets`) - **packages/render-helper** - Markdown rendering utilities (`@ecency/render-helper`) - **packages/ui** - Shared UI component library (`@ecency/ui`) +- **infra/origin** - The web origin nginx vhosts (`eu`/`us.ecency.com.conf`), tracked since + 2026-08-20. ⛔ **This repo is public**: structure is committed, thresholds and addresses are + not — they live in host-only includes. Enforced by `scripts/origin-config-audit.mjs` in CI. + ⛔ CI does **not** deploy these; they are applied by hand to both boxes, and the rule is + apply-and-reload BEFORE committing so the tracked copy matches what is running. See + `infra/origin/README.md`. All packages use `workspace:*` protocol for local dependencies. The main app transpiles workspace packages during build (configured in `next.config.js`). diff --git a/README.md b/README.md index 2d035b4509..67bfb24fe7 100644 --- a/README.md +++ b/README.md @@ -126,10 +126,20 @@ expires. # Anonymous — should HIT after the first request curl -sI https://ecency.com/discover | grep -iE 'cache|tier' -# Logged-in — should always BYPASS +# Logged-in on the SAME path — still cached. /discover is the `list` tier, and most +# tiers are auth-class-equivalent, so logged-in users share one edge entry. curl -sI --cookie "active_user=alice" https://ecency.com/discover | grep -iE 'cache|tier' + +# Logged-in on a MUTE-FILTERED tier — this is what actually bypasses: +# `feed`, `feed-created` and `profile-feed` emit `private, no-store` when the +# active_user cookie is present, because each user's mutes filter it differently. +curl -sI --cookie "active_user=alice" https://ecency.com/@ecency/feed | grep -iE 'cache|tier' ``` +`curl -sI` sends HEAD, which reaches the app and returns the real status only since +#1578 — before that `location /` short-circuited every HEAD with a fabricated 200, so +these recipes reported nothing useful. + Expected headers on an anonymous hit: ```http @@ -141,8 +151,9 @@ CF-Cache-Status: HIT ### Infra configuration -Nginx and CF worker configs live in the infra repo; their behaviour is -documented here in `docs/cache/`. The rules are simple: **respect origin +The web origin nginx vhosts live in this repo at `infra/origin/` (tracked since +2026-08-20), guarded by `scripts/origin-config-audit.mjs` in CI; the CF worker config +lives in the infra repo. Behaviour is documented here in `docs/cache/`. The rules are simple: **respect origin `Cache-Control`**, **bypass on `active_user` cookie** and **preserve `x-cache-tier`** in the response headers. diff --git a/apps/self-hosted/hosting/nginx-multi-tenant.conf b/apps/self-hosted/hosting/nginx-multi-tenant.conf index 21120b88cc..8815193a80 100644 --- a/apps/self-hosted/hosting/nginx-multi-tenant.conf +++ b/apps/self-hosted/hosting/nginx-multi-tenant.conf @@ -128,8 +128,11 @@ server { # THIS BOX (host.docker.internal, an extra_hosts entry in the compose file), # deliberately not through public ecency.com: a Cloudflare hop would make # every tenant's signup look like one datacenter client (one IP for consent - # records, one bucket for rate limits, and a bot challenge would break the - # form outright). The reader's address travels as CF-Connecting-IP, set from + # records, one bucket for rate limits, and an EDGE bot challenge would break + # the form outright). Note the form DOES carry a Turnstile widget of its own + # since 2026-08-20, solved in-page before submit and verified by the relay; + # that is a different thing from an edge interstitial, which the reader could + # not clear. The reader's address travels as CF-Connecting-IP, set from # X-Real-IP, which the HOST's edge vhost asserted from its own remote_addr; # the origin's /api/ location forwards exactly that header pair to the app. # Managed instances only, by construction: only they sit behind this nginx. @@ -247,8 +250,11 @@ server { # THIS BOX (host.docker.internal, an extra_hosts entry in the compose file), # deliberately not through public ecency.com: a Cloudflare hop would make # every tenant's signup look like one datacenter client (one IP for consent - # records, one bucket for rate limits, and a bot challenge would break the - # form outright). The reader's address travels as CF-Connecting-IP, set from + # records, one bucket for rate limits, and an EDGE bot challenge would break + # the form outright). Note the form DOES carry a Turnstile widget of its own + # since 2026-08-20, solved in-page before submit and verified by the relay; + # that is a different thing from an edge interstitial, which the reader could + # not clear. The reader's address travels as CF-Connecting-IP, set from # X-Real-IP, which the HOST's edge vhost asserted from its own remote_addr; # the origin's /api/ location forwards exactly that header pair to the app. # Managed instances only, by construction: only they sit behind this nginx. diff --git a/apps/web/.env.template b/apps/web/.env.template index 4c43543747..65fd6617ce 100644 --- a/apps/web/.env.template +++ b/apps/web/.env.template @@ -56,5 +56,9 @@ NEWSLETTER_SERVICE_TOKEN= # deployment that has not configured the newsletter at all never reaches that code, since # the route already 503s without NEWSLETTER_API_URL and NEWSLETTER_SERVICE_TOKEN. # specs/deploy/newsletter-wiring pins it on the web service in both compose files. +# ⛔ The SITEKEY is inlined at BUILD time, not read at runtime, so it must reach the image +# build: apps/web/Dockerfile takes it as an ARG and both deploy workflows pass it as a build +# arg. Setting it only in the runtime environment has no effect, and the client silently falls +# back to the literal in features/shared/turnstile.tsx. NEXT_PUBLIC_TURNSTILE_SITEKEY= TURNSTILE_SECRET= diff --git a/docs/cache/README.md b/docs/cache/README.md index 0c7f39dfe7..b99e666daa 100644 --- a/docs/cache/README.md +++ b/docs/cache/README.md @@ -150,9 +150,14 @@ Implemented in this repo: - Entry-page TTL refined by post age via L1 Map + per-host Redis L2 - `scripts/purge-cache.sh` supports manual DMCA / moderation invalidation +Tracked here, applied by hand (CI does NOT deploy them): + +- Nginx (`ssrcache` zone) on each origin host — the vhosts are in `infra/origin/` + since 2026-08-20; see [nginx.md](./nginx.md) and `infra/origin/README.md` for the + public-repo contract that keeps thresholds and addresses off the record + Operated outside this repo: -- Nginx (`ssrcache` zone) on each origin host — see [nginx.md](./nginx.md) - Cloudflare worker `ecency-geo-router` — see [cloudflare-worker.md](./cloudflare-worker.md) Before production rollout of changes here, verify `x-cache-tier` values diff --git a/docs/cache/nginx.md b/docs/cache/nginx.md index 16de9ead8b..bc3474f6e4 100644 --- a/docs/cache/nginx.md +++ b/docs/cache/nginx.md @@ -1,7 +1,7 @@ # Nginx Cache Alignment -Nginx (`ssrcache` zone) sits between the CF worker and vision_web. It runs -on each origin server (eu/us/asia.ecency.com). Because the worker has +Nginx (`ssrcache` zone) sits between the CF worker and vision_web. It runs on each +origin server (EU and US; the asia origin was decommissioned). Because the worker has already keyed on auth-class and only forwards cacheable requests, **nginx does NOT need to gate on `active_user` cookie itself** — origin's `Cache-Control` is the source of truth. @@ -25,45 +25,49 @@ proxy_cache_path /var/cache/nginx/ssr levels=1:2 ## Per-host config +### The bot map is CANONICAL here — do not remove it + +⛔ This block is **not** documentation of the config, it IS a copy the test suite reads. +`apps/web/src/specs/features/next-middleware/social-bot-metadata.spec.ts` parses it out of +this file and asserts term-for-term parity with `htmlLimitedBots` in `next.config.js`. The +two drifting is invisible in dev, where nothing is cached, and in production means a page +primed by a browser is served to a crawler from the wrong cache namespace with streamed +metadata (#1257). Keep the fenced `nginx` block and the `"~*( … )"` shape — the test matches +on them. + +The live definition sits at `http` level in `/etc/nginx/nginx.conf`, which is not tracked. + ```nginx -# See "Why the bot UA class is in the cache key" below. map $http_user_agent $html_limited_bot { default ""; "~*(Googlebot|[\w-]+-Google|Google-[\w-]+|googleweblight|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|TelegramBot|WhatsApp|SkypeUriPreview|Yeti)" "|htmlbot"; } +``` -server { - listen 80; - server_name eu.ecency.com; # or us per host - - location / { - proxy_cache ssrcache; - proxy_cache_key "$request_uri$html_limited_bot"; - # Defer to origin Cache-Control. Fallback for responses without it. - proxy_cache_valid 200 0; - proxy_cache_valid any 30s; - # Stale-while-revalidate semantics - proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504; - proxy_cache_background_update on; - proxy_cache_lock on; - proxy_cache_lock_timeout 5s; - - # Observability - add_header X-Cache-Status $upstream_cache_status always; - add_header X-Cache-Tier $upstream_http_x_cache_tier always; - - proxy_pass http://127.0.0.1:3000; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_connect_timeout 5s; - proxy_send_timeout 20s; - proxy_read_timeout 20s; - } -} +### The rest of the vhost is tracked, so read it there + +⛔ **Read `infra/origin/eu.ecency.com.conf` and `us.ecency.com.conf` rather than a snippet +here.** This section used to carry a +hand-copied `server { … }` block, which drifted and began contradicting the real config +— it showed `add_header X-Cache-Tier $upstream_http_x_cache_tier always;`, which the +tracked file explicitly forbids because the upstream already sets that header and nginx +proxies it through, so adding it emitted the header **twice on every response**. It also +showed `always` on `X-Cache-Status`, which the real config deliberately omits: that value +is a cache diagnostic, not something we owe an error response. + +What matters conceptually, and is stable: + +```nginx +proxy_cache ssrcache; +proxy_cache_key "$request_uri$html_limited_bot"; # see the next section +proxy_cache_valid 200 0; # defer to origin Cache-Control +proxy_cache_valid any 30s; +proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504; +proxy_cache_background_update on; +proxy_cache_lock on; ``` + ## Why the bot UA class is in the cache key `htmlLimitedBots` in `apps/web/next.config.js` makes Next.js render metadata diff --git a/infra/origin/README.md b/infra/origin/README.md index 7593bbf9b0..4b52634b5d 100644 --- a/infra/origin/README.md +++ b/infra/origin/README.md @@ -42,8 +42,29 @@ What IS committed and deliberately so: **Comments count as published.** A threshold quoted in a comment is as disclosed as one in a directive, so the vhost comments name the include rather than the number. The audit -reads comments for exactly this reason: an earlier version stripped them and reported a -clean run while every rate sat in prose two lines above. +(`scripts/origin-config-audit.mjs`, run from `.github/workflows/typecheck.yml` as +`--self-test` then `--fail`) reads comments for exactly this reason: an earlier version +stripped them and reported a clean run while every rate sat in prose two lines above. + +## Why `location /` does not answer HEAD itself + +It used to: `if ($request_method = HEAD) { add_header Cache-Control no-store; return 200; }`. +That made every HEAD report a live page whatever the truth was — `GET /@good-karma/points` +answered 307 while `HEAD` answered 200 — so link checkers, uptime probes and crawlers were all +told the wrong thing. Removed 2026-08-20 by **#1578**, which closed #1575 — the PR carries +the change and the before/after measurements, the issue carries the reproduction. + +The `always` on the CORS and security `add_header`s is the **other half of the same fix**, not +a tidy-up. While every HEAD was a fabricated 200 those headers applied, because 200 is in +`add_header`'s default status list; deleting the block alone would have made HEAD reach real +404/429/5xx responses with the headers gone. `always` alone fixes nothing either, since an +`add_header` nested inside the `if` suppresses inheritance regardless. `X-Cache-Status` +deliberately keeps its default list: it is a cache diagnostic, not something we owe an error +response. + +⛔ Verify this against the origin, never the public hostname. Cloudflare normalises the request +and already answered 307 with headers, so `curl -I https://ecency.com/...` reports the bug as +already fixed. ## What these files depend on