Repository navigation
docs(migration): production upgrade runbook + search-indexing policy (already deployed) - #845
Merged
Merged
Conversation
Previously, POST /_/:room dispatched commands to the room Durable Object and checked authorization status via authVerdict(writeRes). However, if the DO rejected a command for non-auth reasons (such as a 413 error when a command exceeds sheet limits), the worker ignored writeRes.ok and unconditionally returned HTTP 202 Accepted. This created a silent write-loss path where callers were informed that commands were accepted when they were actually rejected by the DO. Propagate non-2xx status and response text from the Durable Object fetch response back to the caller in both JSON and plain-text dispatch handlers. Also include unit tests for DO error propagation, legacy room accessibility, oversize sheet handling, and ETHERCALC_AUTH kill-switch behavior.
…verify
The read_score() function in scripts/ratchet-verify.sh previously attempted to read a mutationScore property from Stryker's JSON report schema using jq and node. However, Stryker JSON reports do not contain a top-level mutationScore field; mutant results are recorded under .files[].mutants[] with status strings. As a result, read_score() returned empty, causing ratchet-verify.sh to fail with exit code 2 ('parse error').
Update read_score() in both jq and node paths to calculate the score from individual mutant statuses as (Killed + Timeout) / (Killed + Timeout + Survived + NoCoverage) * 100.
Add production upgrade runbook and supporting audit documentation in docs/migration/ to guide the EtherCalc modern stack deployment. Includes the production upgrade plan, delta audit between legacy and modern worker implementations, WebSocket skew and reconnect handling strategy, preflight validation results, and complete component inventory. Also update AGENTS.md key docs index.
Document the 413 rejection response and WebSocket 1008 close behavior in API.md under POST /_/:room. Add Unreleased changelog entries in Changes.txt for the POST /_/:room error propagation fix and the ratchet-verify Stryker JSON parser fix.
…itions and rollback mechanics
…y in upgrade plan
…d workerd version
Module scripts require an exact JS Content-Type, and Socket.IO 0.x clients parse handshake fields positionally — pin both contracts that real clients depend on.
Hosted Cloudflare Assets pass text/javascript through; standalone workerd rewrites octet-stream via mimeForPath. Document the split in the upgrade runbook and refresh the now-pinned socket.io notes.
Standalone config.capnp has no DB/cron/email bindings. Spell out empty room-index behavior, missing D1 audit/chat recovery, dead settimetrigger persistence, external /_timetrigger setup, PITR vs filesystem backup, and the upgrade-critical DO uniqueKey storage anchor.
Map every RoomDO cross-room hop (DO→DO vs Worker→DO), diff the /_do/* protocol from 149ebcf to HEAD, and record the Phase 2 gradual-ramp verdict: public-room traffic is DEGRADED (sheet-limit asymmetry) not BROKEN, while /_do/init-private is BROKEN against Phase 1 DOs and must stay unreachable until the ramp is at 100% (harden §9 item 9).
/_do/access 501 is not an authz hole: HTML/edit only skip UX redirect hints, DELETE is fail-closed, RoomDO remains the sole boundary, and a Phase 1 DO cannot be private. Keep init-private 501 as the real correctness reason Phase 2 must hit 100% before AUTH=1, stacked on the original rollback justification. Clarify workbook-kind unknown impact.
Record D1 database_size headroom as a go/no-go baseline, reframe mass PITR as bounded-window only, page private-room discovery under the 30s query ceiling, and budget multi-GB d1 export wall-clock.
Add inspect→branch→act cards at Phase 1 unknown outcome, Phase 2 ramp abort, and Phase 3 private-data rollback, plus a cutover-log artifact table and a soak pointer to the D1 capacity ceiling.
Move Card 1 immediately after the Phase 1 deploy fence and link §0.2
instead of restating CLI; collapse Card 3 to ordered commands; make the
Path B WAF default deny-all on both production hosts (optional operator
IP only) and require /_/, /_ws/, /{room}, /{room}.xlsx block checks
before Phase 1 traffic.
Re-running time-travel info after cutover yields a current bookmark, not the pre-cutover one. Recover only from saved output or a recorded UTC timestamp via §6.4; otherwise mark indeterminate.
Mark each companion as supporting evidence with runbook authority. Correct D1 scope (no sheet snapshots; private index exclusion vs unfiltered audit/chat), front-door POST /_/:room 413 propagation (superseding always-202), player.js MIME split, ~1.8M/10GB scale, and self-host absence of D1 — without erasing historical audit notes.
…notes Replace DELTA exec-summary "backup DO storage" with WAF/lockout/bounded PITR guidance and never-strip-meta:access. Mark PREFLIGHT gate 3/4 counts as matching current runbook §1 (1520/196), retiring draft 1514/195 discrepancy claims.
Feed the genuine oracle MINIMAL_SCSAVE (recorded pre-3.1.0) through createSocialCalcFactory()/createSpreadsheet and assert cell values plus semantic round-trip. Reclassify DELTA_AUDIT item j FORWARD-COMPATIBLE; document that first re-save is not byte-identical and that oracle "10/13" is an intentional static-asset subset, not three failures.
The suite loads all recorded fixtures and only hard-asserts nine named HTTP scenarios plus a semantic socialcalc.js check; it does not filter or fail an exact 13-scenario set.
Re-ran scripts/ratchet-verify.sh end-to-end (828s, exit 0). All six packages remain above break floors; worker is still narrowest at 90.03% (+0.03 vs 90%). robots.ts measured 100% (9 killed).
Every audited package sits within 1pp of its Stryker break floor; @ethercalc/worker is +0.03pp. Pre-cutover PRs can trip mutation-gate even when functional tests pass — run ratchet-verify locally first; equivalent mutants get written // Stryker disable justifications.
Room URLs are unlisted by design; stamp noindex on every non-landing response, serve a real robots.txt without Disallow so crawlers can still observe noindex headers, and keep only / indexable.
Record the operator runbook deliverable, the overturned 149ebcf premise, AUTH=1 single-ramp correction, cookie rename regression, robots restore caution, tip measurements, and remaining OPERATOR-VERIFY items.
Latest becomes the operator runbook session (false 149ebcf premise, AUTH=1 single ramp, cookie rename, robots restore); prior security audit compresses into Prior.
Rotation requires the former Latest paragraph under Prior unchanged aside from the label; reinstate the full security-audit wording.
Record the 570 Survived mutants from the Gate 7 Stryker JSON at tip 327fa3d (score 90.034%, floor 90, +0.03pp): 552 REAL GAP, 18 EQUIVALENT, 0 UNCERTAIN. Analysis only — no source/test/threshold changes. Point §8 mutation-fragility at MUTATION_SURVIVORS.md.
Pin room.ts/command-limits REAL GAP counts and expand ellipsized Original/Mutated/Reason cells; headline metrics unchanged.
Record CURRENT_PROD_VERSION_ID bd76bda5…, D1 size/Time Travel, and live probes in the §4.3 cutover log. Rewrite §0.1 so absent ETHERCALC_KEY is status-quo posture, not a hard provision demand.
Keep the candidate-range and still-stands tables inside one continuous blockquote after the baseline-pin wording landed.
Live compat/asset markers are consistent with pre-b7d8840 source uploads but not exclusive if dist/ethercalc/wrangler.json redirected.
Replace bare blank lines inside the STOP banner with quoted blanks so the whole banner renders as one blockquote.
Staging rehearsal proved Version-Overrides only select versions already in the active deployment. Document verified CLI forms, robots/X-Robots discriminators, sticky-cache rollback caveat, and 2026-08-10 results.
Canonical post-rollback edge proof is unique-room X-Robots-Tag; robots alone is forward-only after ship pollutes cache. Mark worker preflight and staging rehearsal checklist items satisfied at HEAD.
SHIP 0e024a32… ramped 0→10→50→100; CURRENT bd76bda5 remains rollback floor. Uncached probes green; edge purge still needed for sticky cache.
…compat CI clean installs only lockfile packages; probing .bun/socialcalc@3.0.8 orphans never works there. Add socialcalc-308=npm:socialcalc@3.0.8 devDependency and resolve the reverse-compat UMD through that alias without disturbing primary socialcalc@3.1.0.
Job cancelled at 20m mid-Stryker on worker-only src diff (run 31388973185) with zero score output. Local full ratchet ~828s; 45m ≈ 2.5× that baseline so CI can finish and report a real score.
Same stryker.conf and mutate scope; gate uses total mutationScore (not covered). Local 90.03 (+0.03) vs CI 90.61 (+0.61) is Timeout/ Killed/Survived flapping, not a second harness. Provenance on run 31390939451 / head 09673a4.
Correct socialcalc-308 scope (worker loads headless; alias only in headless reverse-compat test). Drop unproven Timeout-flapping / mutant-generation language; keep aggregate deltas only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status: already on production
This branch is already deployed to production
ethercalc.net.0e024a32-5ce8-4315-a4eb-a4c5c2a8f780@100%CURRENT_PROD_VERSION_ID = bd76bda5-3161-4576-b159-dbdb97d774c2Rollback (do not run unless aborting):
This PR exists so the source behind that Worker version is recoverable and reviewable. Do not squash or rewrite history in a way that drops production code (see
c789249note below).Why the plan changed (baseline correction)
Live probes on 2026-08-10 showed production was not tag
0.20260717.0/ commit149ebcf. Passkeys,AuthDO, and private rooms were already live (GET /_auth/whoami→enabled:true; root serves./static/passkey/ui.js). The original three-phase plan’s soak ofETHERCALC_AUTH="0"would have madegetSessionPrincipalnull and 403-locked every private room for its owner — a self-inflicted outage. Corrected procedure: single gradual ramp with auth left at"1"throughout (docs/migration/PROD_UPGRADE_PLAN.md§4; superseded three-phase material only in Appendix A).Candidate window remains
[d2afa90, b7d8840)under the working premise of a source-wrangler.tomlupload (exact git SHA still open — no version annotation; §4.0 generated-config redirect can mask compat/asset markers).What this ships (user-visible)
ec_sess→__Host-ec_sesswith no dual-read: every signed-in passkey user appears signed out until they passkey-login again. Credentials remain in AuthDO; only the browser cookie is orphaned.GET /robots.txt(text/plain,User-agent: *,Allow: /, noDisallow); non-landing paths getX-Robots-Tag: noindex, nofollow, noarchive; only/stays indexable. (Accidental dual-purpose revert restored viac789249.)mainpath: externalized root page scripts (static/index-bootstrap.jset al.), CSP/__Hosthardening, WS/frame caps, command-rejection 413 propagation, etc.Verification
test:nodetest:workers327fa3d(docs-only since;packages/+scripts/empty vs that tip)enabled:true; bootstrap 404→200enabled:falseLoad-bearing history note
Commit
c789249restores production search-indexing / robots behavior after an accidental dual-purpose-commit revert. Its subject may look docs-oriented, but it carries runtime code. Any squash, subject-based triage, or interactive rebase must not drop it or production robots/X-Robots behavior will regress on the next deploy.Outstanding (not blocking this PR open)
ethercalc.net(andwwwif needed). Uncached probes already prove ship is live; bare/robots.txtand bare/static/index-bootstrap.jscan still show sticky pre-ship cache hits until purge.ETHERCALC_MIGRATE_TOKENonly if PITR/migrate routes are in-window (production has run without it).Runbook entry point
docs/migration/PROD_UPGRADE_PLAN.md(+ companions indocs/migration/).Do not merge until review. Production is already at 100% ship; merging is documentation/source recovery, not a deploy trigger unless someone runs deploy-production separately.