Fix nightly QA setups for Cursor MicroVM (ubuntu+shim, Patroni bootstrap) - #1042
Draft
cursor[bot] wants to merge 87 commits into
Draft
Fix nightly QA setups for Cursor MicroVM (ubuntu+shim, Patroni bootstrap)#1042cursor[bot] wants to merge 87 commits into
cursor[bot] wants to merge 87 commits into
Conversation
MicroVM terminals ship docker-ce but no running dockerd and no systemd. This script starts dockerd in tmux and fixes socket permissions so manual QA provisioning (pmm-framework.py) can run locally. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
- provision-pmm.sh: scripted PMM Server setup (Jenkins staging parity)
- wait-pmm-ready.sh: correct readyz check (HTTP 200, body {})
- cleanup-pmm-microvm.sh: tear down server + QA containers
- PSMDB+PBM no-systemd entrypoint for MicroVM (PMM_QA_NO_SYSTEMD=1)
Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
MANUAL-QA-MICROVM.md is the expanded reference for the cloud agent prompt. provision-pmm.sh now accepts Jenkins-style DOCKER_ENV_VARIABLE extra -e flags. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Prompt content was copied to the Cursor Automation config; the committed runbook remains at qa-integration/MANUAL-QA-MICROVM.md. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
travagliad
marked this pull request as ready for review
July 2, 2026 10:51
Use runuser+nohup instead of su so pbm-agent stays alive after start. Required for MongoDB PBM backup QA on MicroVM. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
- Add qa-integration/scripts/pmm-ui-login.sh for playwright-cli login on Cloud Agent / MicroVM (session pmm-<TICKET>, programmatic /graph/login) - Expand .agents/workflows/pmmLogin.md with runtime routing table: playwright-cli vs Playwright MCP - Cross-link MANUAL-QA-MICROVM.md, workflowIndex, mcpRules, AGENTS.md Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Use --headed on playwright-cli open by default. Set PMM_UI_HEADED=0 to run headless when needed. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Add repo-level .playwright/cli.config.json (headless: false), pass --config from the login script regardless of cwd, and default DISPLAY to :1 so screen recordings capture a visible browser window. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Use --start-maximized and viewport: null in cli.config.json, and resize to 1920x1200 after open so headed sessions fill the MicroVM display. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
- Add SETUP-INVENTORY.md cataloging all 31 pmm-framework.py variants - PSMDB sharding: docker-compose-sharded.microvm.yaml + start-sharded-microvm.sh - PMM_QA_NO_SYSTEMD path: ubuntu base containers for PS/MySQL/haproxy/external - Fix create_minio_container.yml YAML, docker-py pin, ansible interpreter env - Prefer pmm-server container name for pmm-agent registration - Fix wait-pmm-ready.sh missing body file on first curl failure Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Add scripts/lib/cursor-vm.sh and scripts/cursor_vm.py so automation secrets only need IS_CURSOR_VM=1; PMM_QA_NO_SYSTEMD is derived internally. Wire IS_CURSOR_VM through pmm-framework, ansible playbooks, PSMDB compose helpers, provision/cleanup scripts, and docs. Add MicroVM-safe PostgreSQL service tasks for pdpgsql setups. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Correct external_setup cleanup shell syntax and remove stale MySQL containers before reprovisioning GR/replication setups. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
prepare_install_mysql.yml is included once per node; stop looping over all nodes inside the task file. Mount sockets under /var/run/mysqld instead of replacing /tmp, and capture initialize output for the temp root password. Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Co-authored-by: davi.travaglia <davi.travaglia@percona.com>
Remove invalid become on include_tasks in pdpgsql playbooks. Mount the existing postgresql-primary.conf and postgres-replica.conf files for pgsql replication. Add shm-size and ulimit to PXC container startup.
Remove leftover postgres container data with a root-owned alpine bind mount before recreating the replication cluster.
Use idempotent docker rm for mlaunch playbooks and add shm-size for container stability. Add run-failed-setups.sh for targeted retests.
Ensure MicroVM mode is active when rerun scripts lack the secret in the shell env. Use sudo to remove root-owned postgres data dirs.
PXC 8.0 SST requires xtrabackup 8.x; the playbook only installed xtrabackup-24 for 5.7, causing start_pxc to time out on MicroVM.
Vendor mongodb_user_setup.js was 404 from GitHub; copy into repo and docker cp it for ssl_mlaunch. Tolerate postgresql stop when data dir was removed on replica nodes during patroni setup.
- Patroni: allow replication from any Docker subnet in pg_hba, prefer basebackup over pgbackrest, wait for all cluster members running - PXC: set umask 022 before start_pxc so mysqld does not ignore world-writable node*.cnf files - ssl_mlaunch: add PSMDB 8.0 tarball fallback and mongosh support
Use entrypoint-no-systemd.sh and systemctl shim when IS_CURSOR_VM=1 so psmdb-server starts without /usr/sbin/init on MicroVM.
travagliad
marked this pull request as draft
July 3, 2026 18:03
Replica bootstrap (basebackup after pgbackrest miss) can finish after install_pmm_client. Wait for pg_isready on each node before registering. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Stop and kill Postgres before wiping replica data dirs, bring the primary online first, and clone replicas with basebackup only so port 5432 is free and bootstrap is not racing a promoting leader. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
cursor
Bot
changed the title
Export KRB5_KTNAME when starting mongod, keep containers alive after bootstrap, and wait for rs101 readiness in the sharded setup script. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
cursor
Bot
force-pushed
the
cursor/pmm-manual-qa-process-d487
branch
from
August 3, 2026 12:15
7fa136c to
58ebe85
Compare
Sharded compose has no keytabs volume; skip Kerberos env when keytab is missing, use SCRAM-only auth in sharded mongod configs, ensure /tmp is writable for the unix socket on rs101, and run entrypoint.sh explicitly on all sharded mongod services. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Cherry-pick PMM-7 admin-ajax.php resolver (replaces dead products-api.php) so PSMDB setups no longer fail with 404 during version lookup. Install iproute2 in PDPGSQL containers so PMM-T2224 can apply tc netem on loopback; plain ubuntu:24.04 base lacks tc unlike docker-systemd images. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Wait for mongodb_exporter restart before grepping pmm-agent.log for connectTimeoutMS=4000, matching the MySQL connection-timeout test pattern. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Use dynamic listen port for HAProxy setup to avoid runner collisions on 42100. Add shared PSMDB setup helpers that wait for mongod readiness and retry pmm-agent setup until Connected before adding services. Run PS 5.7 apt install non-interactively to avoid debconf hangs. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Remove setup-helpers.sh and revert start-rs-only/configure-extra-agents to their previous behavior. Keep only the targeted fixes: dynamic HAProxy port, inline pmm-agent connect retry in configure-agents.sh, and PS 5.7 noninteractive apt install. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Retry pmm-client RPM install in PSMDB Dockerfile when mirror metadata is inconsistent. Replace configure-agents retry loop with a short sleep after pmm-agent setup. Start mongod on all RS nodes in scheduled_test BeforeSuite after long inventory tests leave secondaries down. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Implement stop/restart mongod in the PSMDB entrypoint shim so backup tests that call systemctl stop/start mongod behave like they did with full systemd in the container. Revert the scheduled_test BeforeSuite workaround added for rs102 connection refused. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
systemctl stop mongod was a no-op in the PSMDB shim, breaking @bm-mongo backup tests that rely on stop/start mongod (inventory_test.js). Also fix is-active checks and always initialize patroni random_service_name_value. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
stop_mongod used pkill -x mongod which never matched the actual /usr/bin/mongod process, so systemctl stop mongod was a no-op and @bm-mongo backup tests failed deterministically. Use mongosh shutdown + pkill -f fallback, wait for readiness on start, and add test-systemctl-shim.sh (verified on rs101-rs103 after image rebuild). Co-authored-by: travagliad <travagliad@users.noreply.github.com>
The entrypoint auto-restart loop immediately brought mongod back after systemctl stop, breaking @bm-mongo backup tests. Add a mongod-stopped flag so explicit stop/start via the shim works as tests expect. Also redirect mongod logs, remove deprecated journal.enabled from configs (avoid startup warnings on stderr), increase start timeout to 120s, and extend sharded pmm-server wait to 240s with pmm-agent setup delay. Validated locally from clean build: test-systemctl-shim.sh passes on rs101-rs103 (stop stays down, start exits 0 with no stderr). Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Before/After hooks called systemctl start mongod but reused a stale MongoClient from BeforeSuite, causing Server selection timeout after mongod was stopped during backup tests. Reconnect via mongoConnect after ensuring mongod is up. BeforeSuite in scheduled_test now starts mongod on rs101-rs103 before pmm-admin add (rs102 was connection refused when its mongod was down). Setup: replace fixed sleep with wait-for-mongod-nodes.sh in start-rs-only. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
mongoConnect created a new MongoClient on every Before/After without closing the previous one, leaking connections and preventing codecept from exiting (job stuck in "Execute e2e tests" for 1h+). Close the old client before reconnecting, use directConnection=true for rs101:27027, and retry mongoConnect until ping succeeds after systemctl start mongod. Wait for all RS nodes in BeforeSuite before pmm-admin add on rs102/rs103. Entrypoint start_mongod now waits for rs.status().ok when possible. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
The supervisor loop waited on mongod's PID, but mongod is a grandchild (shell -> runuser -> mongod), so wait returned immediately and the loop spun forever, respawning mongosh (a ~250MB node process) continuously. Measured on a local reproduction of the @bm-mongo CI job: every PSMDB container burned 52-72% CPU with a load average of 18.75, which is what made mongo server selection and backup status checks time out. Poll with pgrep every 5s instead, and reserve mongosh for start/stop readiness checks. Revert the speculative hook retries in the backup tests; CodeceptJS routes helper calls through its recorder, so try/catch around I.mongoConnect never intercepts the failure. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Auto-restarting mongod broke every PBM physical restore. PBM stops mongod and runs its own standalone instance over the same dbPath to recover the oplog; the supervisor saw mongod missing and started a competing instance, so the restore died with "DBException in initAndListen, terminating" and every later backup failed because pbm-agent had already exited. The packaged units are the contract: mongod and pbm-agent declare no Restart, only pmm-agent is Restart=always. PMM brings the other two back through `systemctl restart`, which the shim already forwards. Supervise pmm-agent only and cover the expected behaviour in test-systemctl-shim.sh. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
PMM restarts mongod and pbm-agent after a physical restore by running
systemctl through its action runner, which sets cmd.Env to an empty slice
(agent/runner/actions/process_action.go). With no PATH, bash falls back to
/bin:/usr/bin, so runuser (/usr/sbin/runuser) was not found and mongod
never came back:
/entrypoint.sh: line 53: runuser: command not found
Reproduced locally with `env -u PATH systemctl start mongod`. rs102 and
rs103 stayed down for the rest of the run, which is why the mongo client
timed out in the Before hook and `pmm-admin add mongodb` on rs102 was
refused.
Co-authored-by: travagliad <travagliad@users.noreply.github.com>
PMM restarts pbm-agent with an empty environment, so PBM_MONGODB_URI was
missing and the agent exited immediately:
required flag mongodb-uri not set
That left pbm-agent down on every node after the first physical restore,
so all later backups reported BACKUP_STATUS_ERROR. Load
/etc/sysconfig/pbm-agent and /etc/sysconfig/mongod with `set -a` so the
values are exported the way the units' EnvironmentFile= does, and pass the
environment through runuser.
Verified with `env -i systemctl restart pbm-agent` on rs101-rs103:
pbm status now reports pbm-agent [v2.9.1] OK on all members.
Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Contributor
Author
|
Reproduced the Three real defects in the systemd replacement, all introduced with the 1. Supervisor busy-loop ( The loop ran 2. Supervision semantics inverted ( The packaged units are the contract: 3. PMM restarts services with an empty environment (
mongod never came back on rs102/rs103 and pbm-agent stayed down after the first physical restore, which is what produced the mongo I also reverted the speculative hook retries I had added to the backup tests: CodeceptJS routes helper calls through its recorder, so Results
|
Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Replacing docker-systemd with `sleep infinity` left a PID 1 that never
calls wait(), so every orphaned process turned into a zombie. `find /`
then fails on their /proc entries, which is what broke @pgsm-pmm-integration:
find: '/proc/7325/task/7325/net': Invalid argument
Reproduced with a bare container and confirmed the fix:
ubuntu:24.04 sleep infinity -> 3 zombies, find exit=1
ubuntu:24.04 --init sleep infinity -> 0 zombies, find exit=0
The PSMDB containers are unaffected because bash reaps reparented
children, which is why @bm-mongo passes there.
Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Leftover values in the invoking shell (COMPOSE_PROFILES=extra, PMM_CLIENT_VERSION=pmm3-rc) silently changed the setup: six mongo nodes instead of three and the RC client instead of the tarball, so the runs did not match CI. Re-exec with env -i, the way a GitHub runner starts a job. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
Co-authored-by: travagliad <travagliad@users.noreply.github.com>
- ci-repro-e2e.sh and test_list.txt only existed to reproduce CI locally. - test-systemctl-shim.sh is a manual probe that no pipeline runs. - mongoDB.js only needed closeClient() while the backup tests reconnected per hook; those hook changes are gone, so the helper goes back to main. - The mongod.conf edits are obsolete: journal.enabled only produced a startup warning on stderr, which the entrypoint now redirects to /var/log/mongo/mongod.log, and the entrypoint already unsets KRB5_KTNAME when the sharded compose has no keytab, so the configs can keep GSSAPI. Co-authored-by: travagliad <travagliad@users.noreply.github.com>
travagliad
added a commit
that referenced
this pull request
Aug 6, 2026
* Migrate Cursor QA cloud agents to Claude Code + Linode provisioning Ports the three cloud-agent workflows prototyped in the still-open Cursor PR (#1042, cursor/pmm-manual-qa-process-d487) to Claude Code: Test Runner, Test Healer, and Test Reporter as .claude/agents/, with their supporting skills under .claude/skills/. Since this is committed to the repo, any teammate opening percona/pmm-qa in Claude Code gets them automatically -- no separate environment snapshot or dashboard config to keep in sync. Replaces the MicroVM-Docker provisioning approach (which kept hitting "container is not running" on setups needing real systemd) with a throwaway Linode VM per run, provisioned by a new Terraform module (terraform/linode-runner/) and a new pmm-linode-provisioning skill. The VM runs the exact, unmodified qa-integration/pmm_qa/pmm-framework already in this checkout -- qa-integration/ itself is untouched, so there is still exactly one source of truth for provisioning, shared with Jenkins/EC2/CI. Cost-safety net for the Linode VMs is layered rather than relying on any single mechanism (SessionEnd hooks are known to be unreliable for cloud cleanup on abandoned/timed-out remote sessions): - down.sh as the mandatory last step of every agent workflow - a best-effort SessionEnd hook as a secondary attempt - reap.sh as the actual guarantee: a tag- and TTL-based sweep against the Linode API directly, independent of any run's local Terraform state (not yet wired to a live schedule -- cadence/TTL needs sign-off first) Also adds a SessionStart hook (gh, terraform, json-diff, pmm-framework's Python/Ansible deps, the local Playwright helper scripts) and a PreToolUse hook blocking any `git clone` of pmm-submodules. Terraform validate/fmt and shellcheck pass on everything new; live provisioning smoke test still pending a Linode API token. * Fix live-smoke-test findings: 443-only egress, missing ssh/rsync, tag filter Three real bugs surfaced by actually provisioning a Linode VM from this session rather than trusting static validation alone: 1. This controller (a Claude Code Remote session) can only make outbound raw TCP connections on port 443 -- confirmed against arbitrary hosts, not just our own VM, so it's a network-level restriction rather than an HTTP-proxy quirk. Direct SSH on 22 was dead on arrival. Fixed by adding sslh to cloud-init: it multiplexes real sshd and PMM's HTTPS UI over the one open port 443 by protocol-sniffing (a well-known, boring pattern -- not a bespoke exec service). The firewall now opens only 443; up.sh/ run.sh use `ssh -p 443`; the PMM Server docker run publishes to 127.0.0.1:8443 instead of host port 443, since sslh now owns that port. 2. openssh-client and rsync are not present in this environment's base image. Added both to session-start.sh. 3. reap.sh's Linode API call used `?tags=pmm-qa-ephemeral` as a query string, which the API silently ignores -- it returned every instance on the account, tag or no tag. Verified live against the real API that the only correct mechanism is the `X-Filter` request header; fixed and re-verified with --dry-run against the live account (0 false matches). Also verified end-to-end: a real Linode VM provisions, and terraform destroy actually removes it (confirmed via a follow-up API call, not just Terraform's own exit code). * Simplify Linode runner back to plain SSH (permissive network policy confirmed) The dedicated shared environment for these agents will use a permissive network policy, so the sslh port-443 multiplexing workaround (needed only for this scratch session's locked-down egress) is unnecessary complexity. Reverts to a plain 22+443 firewall and straight `ssh`/`rsync`. * Add package-lock.json for the Playwright helper scripts Locks the playwright version for .claude/scripts so SessionStart's `npm install` there is reproducible. * Self-destructing Linode VMs instead of a reaper Routine; git-clone, never rsync - Each instance now carries its own on-box systemd timer (default 24h, overridable) that deletes it via the Linode API using its own run tag -- no external cron/Routine, no risk of a scheduled job mistakenly reaping a still-active run. extend.sh reschedules the timer on a live instance instead of losing it mid-investigation. reap.sh is removed. - up.sh now has the box `git clone` a specific ref of percona/pmm-qa instead of rsyncing this session's working tree. Claude never edits code on the VM: a fix under test must be committed and pushed to a branch first (PMM_QA_REF on a fresh box, or sync.sh on a live one). Documented explicitly as a Never rule in both provisioning agents and the skill. * Rename test-healer to test-doctor, add ffmpeg/video evidence, clarify docs - test-healer -> test-doctor across agent file, skills, and docs. - Explicit bug-reproduction vs fix-verification phases in the workflow: confirm the failure reproduces before touching code, then re-verify the fix against the SAME VM/environment that reproduced it. - pw-record.js: Playwright's own video capture, transcoded to .mp4 via ffmpeg (now installed by the SessionStart hook) -- no external recorder. - docs/agents/AUTOMATIONS.md: replaced hedged language with confirmed facts about Claude Tag (no passive channel watching, @mention required) and Routine trigger types (schedule/API/GitHub-owned-repo only; a Jira Automation rule can hit a Routine's API trigger directly). * Restructure agents into test-runner / test-doctor / fb-validator test-doctor now watches pmm-qa's own scheduled main-branch CI (e2e/gssapi/helm/integration-cli), triggered directly by a notify-test-doctor job added to each workflow instead of polling. fb-validator replaces test-reporter and covers both outcomes of a Percona-Lab/pmm-submodules FB Tests run (green -> Jira evidence, red -> triage/fix), since that repo can't push a CI-trigger event to us and still needs a polling Routine. Docs and skill cross-references updated to match. * Centralize Test Doctor's CI trigger on workflow_run, add nightly-e2e-tests-matrix.yml Replace the per-workflow notify-test-doctor job (a hand-maintained needs: [...] + if: failure() list in each of the four scheduled workflows) with a single .github/workflows/notify-test-doctor.yml that watches all five main-branch runs via workflow_run and fires on GitHub's own computed run conclusion. This fixes a real gap: some of these pipelines pass their e2e-test step but still fail overall once a later Launchable step errors collecting results, and a per-job needs list has to be kept in sync by hand every time a job is added or renamed or it silently misses failures like that. workflow_run's conclusion isn't derived from any job list we maintain, so it can't miss them, and watching a sixth workflow going forward is a one-line addition instead of a new job in that workflow's own file. nightly-e2e-tests-matrix.yml is included too -- it's dispatched daily by the Jenkins pipeline in jenkins-pipelines, so it's matched by name rather than event == 'schedule' (it has no GitHub Actions cron of its own). Routine ID is now hardcoded in the workflow (not sensitive); only the bearer token stays a repo secret. * Fail notify-test-doctor when the routine token isn't configured A silent skip meant a missing secret could quietly disable the whole watcher with no visible signal. Fail the job instead so a missing TEST_DOCTOR_ROUTINE_TOKEN shows up as a red check, not silence. * Capture PMM AI Slack app design before it gets lost in chat Records what's been designed so far but never committed anywhere: the manifest, Socket Mode relay architecture (no public endpoint needed), the channel-to-routine static routing table, and the reply-as-bot mechanism (PMM_AI_SLACK_BOT_TOKEN + direct chat.postMessage instead of the Slack MCP connector, so replies show up as the bot instead of a person). Also notes a future agent idea for a prod/support channel -- reads a person's thread message and figures out whether it's a how-to question or an actual bug needing reproduction -- as an open, unbuilt idea (working name "support-triage", name not settled). Nothing here is deployed yet: no Slack app created, no relay process written, no Routine created for it. * Un-ignore .claude/integrations/ and actually add the Slack app files The previous commit's manifest.yaml and README.md were silently dropped by .gitignore's blanket .claude/* rule -- only agents/skills/hooks/scripts/settings.json had carve-outs. Add the missing carve-out and the two files it was hiding. * Fix firewall label length overflow and missing TF_VAR_linode_token in down.sh linode_firewall.runner's label prefixed 'fw-' onto the 63-char instance label, exceeding Linode's 32-char firewall label limit and failing terraform apply (and, since Terraform re-evaluates the full config for a destroy plan too, blocking terraform destroy the same way). Give the firewall its own independently-truncated label. down.sh checked LINODE_TOKEN was set but never forwarded it as TF_VAR_linode_token like up.sh does, so terraform destroy always failed with 'No value for required variable' before touching anything -- the primary cleanup mechanism was non-functional. Verified live: up.sh now provisions cleanly through the firewall resource, and down.sh now destroys all 5 resources successfully. * Replace SSH with an HTTPS-exec mechanism for the Linode runner Raw SSH (port 22) is unreachable from a Claude Code cloud session at any network access level (None/Trusted/Full/Custom) -- confirmed live, including with a generic non-Linode target, so it isn't a config gap to fix. Moving that admin channel to a non-443 port doesn't work either -- also confirmed live: the CONNECT tunnel succeeds but the TLS handshake gets reset right after the ClientHello, consistent with per-port traffic inspection rather than a simple port-based ACL. Every instance now runs a small bearer-token-authenticated HTTPS service on port 443 instead (cloud-init.yaml.tftpl installs it, run.sh/up.sh talk to it), addressed by a nip.io hostname derived from its IP -- the same proxy drops bare-IP connections outright and needs a hostname to route on, also confirmed live. This works from the default proxied-HTTPS environment, no special network policy needed, which the old SSH-based version required and most environments don't have. Since host port 443 is now the exec-server's, PMM Server binds an internal-only port (8443) instead of 443 -- client containers on the same docker network are unaffected (container-hostname discovery, not host-port-dependent), but the controller's own browser has no external port to reach PMM's UI on today. Documented as an open gap in the skill (step 4) and test-runner.md, not silently routed around. Validated end-to-end against a real Linode VM running the full PMM-15188 manual QA scenario (ssl_mysql provisioning, pmm-admin, pmm-encryption-rotation x2, log/DB checks) -- everything passed through the new mechanism with zero SSH usage. Supersedes the firewall-label and down.sh token fixes from the previous commit with equivalent versions (same two bugs, folded into this redesign since both touch the same lines). * Multiplex the exec-server and PMM's UI onto port 443 via nginx SNI routing Closes the UI-access gap from the previous commit: PMM Server's UI had no controller-reachable port since host 443 was the exec-server's alone. Tried moving the exec-server to a non-443 port first -- also confirmed live not to work (see the previous commit's message) -- so instead nginx's stream module (ssl_preread) now routes by SNI hostname on the single usable port: "exec-"-prefixed hostnames go to the exec-server (now loopback-only), everything else goes to PMM. Neither side's TLS is terminated by nginx, so both keep handling their own exactly as before. Validated live end-to-end through the actual module (not just an isolated test): provisioned a real instance, brought up PMM Server, hit its readyz directly from the controller over the plain hostname (200 {}), confirmed the exec channel still works at the same time, and tore down cleanly. run.sh/up.sh now construct the "exec-" prefixed hostname; the skill and README are updated to document both hostnames and drop the "UI access blocked" caveat -- it isn't, anymore. * Trim the investigation narrative out of the operational skill docs pmm-linode-provisioning's "Why HTTPS-exec, not SSH" section read like a debugging log (CONNECT tunnel details, ClientHello resets, "confirmed live, twice") -- fine for the module README (implementation reference, a different audience) but not for a skill an agent reads mid-task. It only needs the operational facts: no SSH, use run.sh, two hostname conventions. Renamed to "Accessing the VM" and cut to that. * Pin exec-server TLS to a per-run cert, harden scripts against path/regex bypass Address CodeRabbit review findings on PR #1138: - Generate the exec-server's TLS cert/key in Terraform (tls_private_key + tls_self_signed_cert, *.nip.io SAN) instead of on-box openssl at cloud-init time, so run.sh/up.sh verify against a CA they already know (--cacert) rather than skipping verification (-k). Real hostname verification isn't achievable given Linode's boot model (user_data is fixed before the IP exists), but chain verification against this run's uniquely generated key is. - Add a separate 32-char-truncated firewall_label (Linode's firewall label limit is stricter than the instance label limit). - Make the exec-server a ThreadingHTTPServer with a request-header read timeout, so one slow/half-open connection can't tie up the single- threaded handler indefinitely. - Validate run_id in up.sh/run.sh/down.sh as a single safe path component before it's used to build any filesystem path. - Add an ERR trap in up.sh so a failure after provisioning (health poll, cloud-init poll, clone) tears the instance down instead of leaking it. - Fix block-pmm-submodules-clone.sh's regex to catch `git -C ... clone` and other global-option forms, not just `git clone` literally. - Scope pmm-ui-login.js's Basic-auth header to PMM's own origin via page.route instead of setExtraHTTPHeaders (which sent it on every request the page made). Validate sessionId/PMM_URL/storage-state path in pmm-ui-login.js, pw-screenshot.js, and pw-record.js. - Trim session-start.sh to only what actually runs in this environment now that PMM/tests run entirely on the Linode VM (drop SSH client, Python venv, ansible-galaxy -- none of that runs on the controller). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfUFzc6H9QBPQAmL8Ya4gq * Address remaining CodeRabbit doc/workflow findings on PR #1138 - fb-validator.md / test-doctor.md: note that <PR>/<run_id> must be extracted as a plain numeric ID before use in any shell command; fix broken anchor links (MD051); clarify down.sh isn't needed when the red path stops before up.sh ever provisions a VM. - session-end-cleanup.sh: explain why scanning every runs/ dir is safe (one isolated VM per Claude Code cloud session, nothing to collide with). - Slack integration README: keep PMM_AI_SLACK_BOT_TOKEN in the relay's own secret store instead of the fired Routine's environment (the Routine calls a local relay endpoint to post its reply instead of hitting the Slack API directly); note the Socket Mode envelope_id ack + event_id dedup needed to avoid double-firing on Slack retries; tag the architecture code fence as text (MD040). - pmm-linode-provisioning/SKILL.md: generate ADMIN_PASSWORD per run instead of a fixed literal; propagate to pmm-ui-evidence/SKILL.md, which now also uses the exec-<ip>.nip.io hostname format instead of a bare IP. - notify-test-doctor.yml: also check head_branch == main so the watcher only ever fires for main-branch runs. - AUTOMATIONS.md: note LINODE_TOKEN should be a least-privilege, access-controlled token, and acknowledge it still flows into TF_VAR_linode_token / cloud-init user_data / tfstate as an accepted tradeoff of the current no-backend design. - pmm-fb-tests/references/fb-tests.md, pmm-git-diff/SKILL.md, pmm-linode-provisioning/references/SETUP-INVENTORY.md: tag an unlabeled fenced code block as text (MD040), replace the unsupported `gh pr diff -- path` positional argument with a supported gh api fetch, add the missing 4th table column header. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfUFzc6H9QBPQAmL8Ya4gq * Address CodeRabbit's second review pass on PR #1138 - fb-validator.md/test-doctor.md: distinguish the numeric GitHub identifier (PR / Actions run ID, for gh commands) from the separate Linode run-id (heal-<PR> / nightly-<workflow>-<date>, for up.sh/sync.sh/down.sh) -- my earlier "extract as a plain numeric ID" wording conflated the two. - fb-validator.md: use `gh pr checks --watch` so a still-pending FB build (exit code 8) isn't raced into the green/red branch. - session-start.sh: verify the downloaded Terraform archive against HashiCorp's published SHA-256 sums before `sudo unzip`; drop the `|| true` on the hooks chmod so a permission failure aborts the script instead of silently leaving a hook non-executable. - Slack README: note the `/reply` relay endpoint needs its own authentication (a per-run signed capability bound to channel/thread), not just token custody, once it's actually built. - pmm-git-diff/SKILL.md: add `set -euo pipefail`, fetch base/head into a `mktemp -d` instead of the worktree, and validate both are real JSON before diffing. - pmm-linode-provisioning/SKILL.md + pmm-ui-evidence/SKILL.md: persist ADMIN_PASSWORD to runs/<run_id>/admin_password instead of relying on an exported shell variable, since later steps can run in a separate shell where that export wouldn't exist. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfUFzc6H9QBPQAmL8Ya4gq * Scope session-end cleanup to this session's own Linode runs CodeRabbit correctly flagged that scanning every runs/ directory isn't actually safe: this hook has no CLAUDE_CODE_REMOTE gate, so it can also run against a local checkout, where two concurrent Claude Code sessions sharing the same working tree would each tear down the other's active VM. My earlier fix only annotated this as safe -- it wasn't. up.sh now tags each run directory with CLAUDE_CODE_SESSION_ID at provisioning time. session-end-cleanup.sh only tears down a run whose tag matches its own session ID, and skips entirely if it doesn't have one to compare -- the on-box self-destruct timer remains the actual guarantee either way. Also switch notify-test-doctor.yml's head_branch check from a hardcoded 'main' to github.event.repository.default_branch, per CodeRabbit's suggested diff -- avoids hardcoding the branch name. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfUFzc6H9QBPQAmL8Ya4gq * Pin PMM's own TLS cert in the UI scripts instead of trusting any cert CodeRabbit pushed back on my "left as intentional" reply for ignoreHTTPSErrors -- correctly: the cert doesn't need to be known in advance to pin it, only fetched after PMM comes up, over the exec-server channel that's already cert-pinned. Implemented and verified live against a local self-signed HTTPS server before wiring it in (confirmed: correct pin succeeds, wrong pin and no pin both fail closed). - pmm-linode-provisioning/SKILL.md step 2: after readyz, fetch PMM's cert via `run.sh ... openssl s_client | openssl x509` and save it to runs/<run_id>/pmm_cert.pem. - New .claude/scripts/lib/spki-pin.js: computes a cert's SPKI SHA-256 pin for Chromium's --ignore-certificate-errors-spki-list flag. - pmm-ui-login.js/pw-screenshot.js/pw-record.js: accept optional PMM_CERT_PATH -- when set, launch Chromium with the SPKI pin and ignoreHTTPSErrors:false; otherwise fall back to the old ignoreHTTPSErrors:true behavior (unchanged for non-PMM URLs like a GitHub Actions screenshot, which already have a real CA). - pmm-ui-login.js: replaced page.request.post for the login call with an in-page fetch via page.evaluate -- discovered live that Playwright's request context is a separate Node-side HTTP client that doesn't honor the Chromium launch flag, so the actual credential-carrying request would have silently kept trusting any cert even with a pin configured. - Also found and fixed live: npm-installed playwright (^1.56.0 resolved to 1.62.1) expects a Chromium revision that doesn't match what's pre-installed at /opt/pw-browsers -- all three scripts now pass executablePath explicitly per this environment's own guidance, or every real run of these scripts would fail outright. Also, from the same CodeRabbit round: renamed the firewall CIDR variable from allowed_ssh_cidr (leftover from the SSH design) to allowed_inbound_cidr, and documented that the same port carries PMM's own UI/API, not just the exec-server -- so PMM itself is reachable by anyone who knows the run's nip.io hostname while the instance is up, unless this is tightened. Fixed the pmm-git-diff jq validation gap CodeRabbit found: `jq empty` silently accepts a zero-byte file (confirmed live), replaced with `jq -e .` which correctly rejects it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfUFzc6H9QBPQAmL8Ya4gq * Require PMM's cert pin instead of falling back to trust-anything TLS CodeRabbit's follow-up review on c801e6e caught two real gaps in the TLS pinning I just added, both confirmed live before fixing: - pmm-ui-login.js only ever targets PMM, so the ignoreHTTPSErrors:true fallback for a missing PMM_CERT_PATH had no legitimate use -- it just meant a forgotten env var silently sent the admin password over an unverified connection. PMM_CERT_PATH is now required; the script exits before launching a browser if it's missing. - pw-screenshot.js/pw-record.js now default ignoreHTTPSErrors to false instead of true. They're still usable without a pin for non-PMM URLs (e.g. a GitHub Actions screenshot) since those already have a real CA and strict verification handles them fine -- verified live that a self-signed PMM cert with no pin now fails closed instead of silently succeeding, while a real CA and a correctly pinned PMM cert both still work. - pmm-ui-login.js: page.goto(pmmUrl) can follow a redirect to a different origin before the login POST is built. The origin-scoped Basic-auth route doesn't protect that separate request. Added an explicit check that page.url()'s origin still matches pmmOrigin after navigation, and switched the login/help-page URLs to build from pmmOrigin directly rather than the redirect-derived origin. Verified live with a mock server: a same-origin load succeeds, and a redirect to a different origin -- even reusing the exact pinned cert, so it's not just the TLS pin catching it -- is now rejected before the password would be sent. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfUFzc6H9QBPQAmL8Ya4gq * Fix set -e/&& masking, require explicit public-ingress opt-in, reject login redirects Three more findings from CodeRabbit's ongoing pass, all confirmed live before fixing: - pmm-git-diff/SKILL.md: `jq -e . base.json && jq -e . head.json` looked like it fails closed under `set -e`, but bash exempts every command in an && list except the last one from errexit -- confirmed live that a failing first check let the script fall through to json-diff on invalid data anyway. Replaced with a plain for-loop that explicitly exits 1 per file, which does abort correctly (verified). - terraform/linode-runner/variables.tf: `allowed_inbound_cidr` no longer defaults to 0.0.0.0/0 in Terraform itself -- up.sh now passes it explicitly from an ALLOWED_INBOUND_CIDR env var (still 0.0.0.0/0 unless set, since this environment still has no stable egress IP to scope it to by default), so opening PMM to the whole internet is a visible choice at the call site instead of a buried default. - pmm-ui-login.js: the finalOrigin check added last round only covers the initial page navigation. The login POST itself could still receive a 307/308 redirect and forward the admin password to a different origin. Added `redirect: "error"` to that fetch so it rejects instead of following. Verified live: a mock PMM that redirects the login POST now fails the request instead of forwarding credentials; a normal non-redirecting login still succeeds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfUFzc6H9QBPQAmL8Ya4gq * Rearchitect agents: extract Investigator + FB Reporter, merge FB Validator into Test Doctor Reconsidered the agent architecture end to end after noticing test-doctor and fb-validator duplicated almost their entire "dedup, reproduce, classify, fix, PR, cleanup" pipeline near-verbatim, differing only in the PR marker text and which repo's logs to read. - New agent `investigator`: the shared pipeline, extracted for real. Its core change from the old design isn't just deduplication -- it now classifies a failure as a test bug or a product regression from what actually reproduces on a throwaway VM, not from a git-log/diff correlation made before ever touching a VM. The old "did an upstream PR merge in that window" check still happens, but only as supporting evidence alongside a real reproduction, not as the sole basis for deciding not to reproduce at all. - New agent `fb-reporter`: split off the FB-Tests-green-evidence responsibility, since it has nothing to do with failure investigation. Retries past flakiness itself (`gh run rerun --failed`, capped at twice) before giving up -- previously this just gave up on the first red. - `fb-validator` is gone. Percona-Lab/pmm-submodules is a Percona-owned repo, not a third party's, so the "polling vs event-triggered" argument for keeping it a separate agent no longer holds once a notify workflow can be added there too (noted as a go-live item, since that requires access to that other repo). `test-doctor` now watches both sources and is purely a thin detect-and-handoff layer -- all judgment moved to `investigator`. - Investigator and FB Reporter are real agents (their own name/description, directly invokable ad hoc), not skills -- but test-doctor/test-runner reference them by reading the file directly and following it in the same session, rather than spawning them as nested subagents. Claude Code's own docs don't confirm whether a Routine-fired session can spawn a subagent via the Agent/Task tool, and both test-doctor and test-runner run as Routines -- reading the file directly sidesteps that unconfirmed dependency entirely while still sharing one source of truth. - Renamed all 6 skills to drop the redundant "pmm-" prefix (pmm-jira -> jira, pmm-fb-tests -> fb-tests, pmm-linode-provisioning -> linode-provisioning, pmm-git-diff -> git-diff, pmm-ui-evidence -> ui-evidence, pmm-repos -> repos) -- every agent/skill in this repo is already pmm-qa's own, the prefix said nothing. Updated every cross-reference across agents, skills, hooks, and docs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfUFzc6H9QBPQAmL8Ya4gq * Fold test-doctor into investigator -- it was too thin to be its own agent You're right that it was redundant. Once dedup/reproduce/classify/fix/PR all lived in investigator, test-doctor's remaining job -- parse whichever trigger payload arrived and extract a failure list -- isn't a separable concern, it's just investigator's own first step. Splitting "detect" from "investigate" into two files/two Routines only added an indirection with no real boundary behind it. - investigator.md absorbs test-doctor's "Being invoked" logic directly: CI trigger payload, FB trigger payload, or a direct ask all parse their own failure list as workflow step 0, before the existing dedup/reproduce/classify/fix/PR pipeline. - Down to 3 agents: test-runner, investigator, fb-reporter. - Renamed .github/workflows/notify-test-doctor.yml -> notify-investigator.yml, and TEST_DOCTOR_ROUTINE_TOKEN -> INVESTIGATOR_ROUTINE_TOKEN (not live yet, so free to rename). - AUTOMATIONS.md documents why the separate watcher was cut, and adds an "Updating the live Routines" table -- the three Routines already created from an earlier draft of this design need to be brought in line by hand (Test Doctor -> rename to Investigator, same trigger ID, new prompt; FB Validator -> currently broken, points at a deleted file). - Updated every remaining cross-reference (skills, hooks, docs) accordingly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfUFzc6H9QBPQAmL8Ya4gq * Add router agent for Slack, sharpen investigator's classification - New router.md: matches an @pmm-ai Slack mention to test-runner / investigator / fb-reporter (or answers directly), then reads and follows the matched agent's file in the same session. A mention never reaches test-runner directly anymore. - Slack triggering is the custom @pmm-ai app, not Claude Tag -- fixed across test-runner.md, AUTOMATIONS.md, and the slack integration README. - test-runner.md: every ticket has a linked submodules PR, so the fb-reporter hand-off is unconditional now. - investigator.md: being asked directly can now mean an open-ended question or a suspected customer-reported bug, not just a known failing test -- added its own reproduce-then-classify outcome set (not-a-bug-here's-the-right-way / genuine-bug-report / didn't reproduce). Folds the earlier "support-triage" idea into investigator instead of a separate agent. - investigator.md: an FB-triggered test-bug fix that anticipates a not-yet-merged upstream PMM/Grafana PR now opens as a draft PR marked blocked-on that PR, instead of a normal ready-to-merge one. * Unify investigator into one pipeline, no known-failure/question fork The earlier split (3a known-failure vs 3b question/suspected-bug) was artificial -- investigator can't actually know which shape of input it has until it's already dedup'd and reproduced, so both steps work identically either way. Now there's one dedup -> reproduce -> classify pipeline; the only real branch is inside classification, where a genuine bug routes to product (report) or test (fix, checking FB-triggered + open-upstream-PR for the blocked draft PR case). A CI/FB-triggered "matches intended behavior" case folds into the test- bug fix path instead of being a dead end, since there's no one to explain a reply to. * Drop redundant gate from investigator's classify step "Was someone waiting for an answer?" was unnecessary -- landing on "matches intended behavior" already implies someone described a scenario to check, since a CI/FB failure that reproduces is never that outcome (it's always a confirmed bug). Renamed the two outcomes to make that self-evident: "described scenario is not an actual bug" and "CI failure / scenario-described bug confirmed", the latter going straight to the test-vs-product decision. * Land approved workflow diagrams into AUTOMATIONS.md Embeds the four mermaid flowcharts (Test Runner, Investigator, FB Reporter, PMM AI/Router) as GitHub-native mermaid fences next to each workflow's existing prose -- same content approved in the scratch artifact, now checked into the repo instead of living only there. * Trim fb-reporter/investigator per review feedback, broaden investigator's scope fb-reporter.md: cut restated invocation rationale, redundant "who to report to" detail, and a Never-list item already enforced by the existing clone-block hook; simplified two more Never items to avoid repeating the retry count and job-handoff wording stated elsewhere. investigator.md: description was scoped too narrowly to "pmm-qa test failures" -- it also diagnoses a customer-reported product problem with no failing test involved at all, which the mermaid flow already reflects via the product-vs-test routing. Rewrote the description around that, and cut a label phrase from the intro that wasn't pulling its weight. AUTOMATIONS.md: added a go-live checklist item -- Percona-Lab is a different GitHub org than percona, so the connector/gh auth needs to actually reach it for fb-reporter and investigator's FB-source checks. * Fix down.sh: pass allowed_inbound_cidr, init before destroy Found by actually running test-runner end-to-end: down.sh errored outright with "No value for required variable" because allowed_inbound_cidr has no default in variables.tf (made required, non-public-by-default, in an earlier review round) and down.sh never passed it. Also add a defensive `terraform init` before destroy -- the run's own .terraform dir should already be initialized by up.sh, but destroy shouldn't have to assume that's still true. * Address second round of review comments (34 missed by the first pass) investigator.md: note that a job's own pass/fail can mask a Launchable- quarantined or below-confidence-threshold test underneath it (CI and FB sources both), add a zero-investigation exception to "always reproduce" for genuinely obvious cases, mention json-diff alongside git-diff for the product-bug cross-check, drop the "not built yet" planning note (that belongs in AUTOMATIONS.md's checklist, not the agent's own behavior file), remove the now-redundant pmm-submodules clone rule (the hook already enforces it), and trim several explanatory asides. test-runner.md: mention json-diff, stop re-explaining what the linode-provisioning skill already documents, let fb-reporter be invoked as a sub-agent or directly instead of mandating one path, drop the Developers-only-visibility Never item (the jira skill already mandates it) and the clone rule (hook-enforced), add an exception to the qa-integration/ modify rule for tickets that objectively require a setup change, and set concrete criteria + Playwright-not-CodeceptJS guidance for the automation-adds-a-test decision. Hooks and notify-investigator.yml: trimmed several over-explained comments to one line each, no logic changes. AUTOMATIONS.md: checked off the GitHub connector go-live item. * Update go-live checklist to reflect actual state Live Routines confirmed updated and working (Test Runner unchanged, Test Doctor renamed to Investigator, FB Validator resolved). Marked the items that are actually done, and made the remaining ones concrete about why they're waiting: submodules notify workflow needs a session scoped to that repo (can't cross-add a different GitHub org mid-session), Jira Automation and the PMM AI Slack app need manual work outside what a session here can do. Also flags a real finding: gh CLI in this kind of interactive session can't reach Percona-Lab/pmm-submodules at all (403, access not enabled for the session) -- unclear yet whether that's specific to this session type or also true for a live Investigator/FB Reporter run, needs confirming from an actual Routine fire. Added the per-person Router routing idea as a future item. * Fold per-person routing idea into the checklist, drop the extra section Per feedback -- keep it as one more go-live checklist item instead of its own section; same content, just not split out. * Trim down.sh comment, add shared-environment checklist item down.sh: cut the explanatory comment on ALLOWED_INBOUND_CIDR per review feedback, same as the other trims this round. AUTOMATIONS.md: added a checklist item for a team-wide shared Claude Code environment with LINODE_TOKEN set once at the environment level, so individual teammates don't need their own Linode token. * Note the submodules notify-workflow PR is open and verified --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Single PR for MicroVM manual QA provisioning and versioned PMM cloud agent roles on
percona/pmm-qa.MicroVM provisioning (
qa-integration/)provision-pmm.sh,lib/wait-pmm-ready.sh,cleanup-pmm-microvm.sh,start-docker-microvm.shPMM_QA_NO_SYSTEMD=1)MANUAL-QA-MICROVM.mdrunbookVersioned cloud agents (
.cursor/).cursor/agents/test-runner.md,test-healer.md,test-reporter.md(+ Fase 3 workers)pmm-jira,pmm-fb-tests,pmm-provisioning,pmm-ui-evidence,pmm-repos,pmm-jenkins-staging(renamed frompmm-manual-test)pmm-submodulesclone.cursor/environment.jsonfor CloudPMMsnapshot.gitignore: track.cursor/agents/, skills, hooks,environment.jsonDocs (
docs/agents/)README.md,SETUP.md,ROLES.md,TROUBLESHOOTING.md,VALIDATION.md,AUTOMATIONS.md,ADMIN_REQUEST.md,HANDOFF.md,CANVAS_SYNC.mddocs/agents/AUTOMATIONS.md)#pmm-aicanvas synced fromROLES.mdPost-merge checklist
main— linear history)environment.jsonlandsdocs/agents/AUTOMATIONS.mddocs/agents/ADMIN_REQUEST.mdto team admindocs/agents/VALIDATION.mdchecklist on a real cloud runBranch rebased on latest
main(conflicts inAGENTS.md+external_setup.ymlresolved).