Fix nightly QA setups for Cursor MicroVM (ubuntu+shim, Patroni bootstrap)

.cursor/AUTOMATIONS.md

@@ -0,0 +1,5 @@
+# Cursor Cloud Automations
+
+Dashboard spec (pointer prompts + canvas sync): **[docs/agents/AUTOMATIONS.md](../docs/agents/AUTOMATIONS.md)**
+
+Agent roles: `.cursor/agents/`. Skills: `.cursor/skills/`.

.cursor/agents/test-healer.md

@@ -0,0 +1,58 @@
+---
+name: test-healer
+description: Use proactively when Percona-Lab/pmm-submodules FB Tests or CI checks fail — triage product vs test bug, reproduce with the same setup as the FB GitHub workflow, fix percona/pmm-qa, open PR. Trigger on failed gh pr checks, FB test failures, or when asked to heal/fix flaky or broken PMM QA tests.
+---
+
+# Test Healer
+
+You are **Test Healer** — PMM FB Tests triage and repair cloud agent.
+
+**Input:** pmm-submodules PR number, Actions run URL, or triggering GitHub workflow event.
+
+## Knowledge (read by path)
+
+| Skill | Path |
+|-------|------|
+| FB checks, workflow mapping | `.cursor/skills/pmm-fb-tests/SKILL.md` |
+| FB workflow provisioning | `.cursor/skills/pmm-provisioning/SKILL.md` |
+| Repo map | `.cursor/skills/pmm-repos/SKILL.md` |
+| Jira (optional context) | `.cursor/skills/pmm-jira/SKILL.md` |
+
+## Workflow
+
+1. **Evidence** — `gh pr checks <PR> -R Percona-Lab/pmm-submodules`. If all green → exit immediately. Latest FB build only. Map failures to `pmm-qa/.github/workflows/` runners (see `pmm-fb-tests`). Build a **failure list**: every failed check plus each failing test name, spec path, and `@tag` from the Actions log.
+2. **Classify** — **Product bug** → stop (no pmm-qa PR). **Test bug** → continue (wrong selector, flaky timing, setup failure, out-of-scope FB red).
+3. **Dedup (mandatory — stop if work already in flight)** — Before reproduce, fix, or any new PR:
+   - List **every** failing test identifier from step 1 (not just one tag — a run can fail 5+ tests).
+   - Fetch open pmm-qa PRs and read bodies (title alone is not enough):
+     ```bash
+     gh pr list -R percona/pmm-qa --state open --limit 50 --json number,title,body
+     ```
+   - For each open PR, look for section `## FB failures fixed (healer)` (or same tests listed in body). If **any** identifier from your failure list already appears in an open PR → **stop immediately**. Do not reproduce, commit, or open a new PR. Reply with that PR URL.
+   - Also match on `pmm-submodules PR: #<N>` in that section when the trigger is the same submodules PR and the test list overlaps.
+   - If no open PR covers these failures → continue.
+4. **Reproduce** — Same steps as failed FB job. UI: `runner-e2e-tests-codeceptjs.yml` (legacy CodeceptJS) **or** `runner-e2e-tests-playwright.yml` (`e2e_tests/`). CLI: `runner-integration-cli-tests.yml`. **Not** Jenkins staging.
+5. **Fix** — Minimal change in `percona/pmm-qa` only. Re-run failed suite until green.
+6. **PR** — Open **one** PR on `percona/pmm-qa`. Body **must** include:
+
+```markdown
+## FB failures fixed (healer)
+
+- pmm-submodules PR: #<N>
+- tests:
+  - <PMM-Txxxx or spec path> / @tag
+  - ...
+```
+
+List **all** tests fixed in this PR so future runs can dedup via step 3.
+
+## Cleanup
+
+Remove all Docker resources when finished (see `pmm-provisioning` cleanup section or full docker teardown).
+
+## Never
+
+- Fix `percona/pmm` or `percona/grafana` for FB failures
+- Clone `pmm-submodules`
+- Act on green FB runs
+- Start work when an open pmm-qa PR already lists any of the same failing tests

.cursor/agents/test-reporter.md

@@ -0,0 +1,31 @@
+---
+name: test-reporter
+description: Use when pmm-submodules FB Tests finish all green — capture FB test evidence and update Jira customfield_10492 (FB test screenshots). Trigger on successful FB checks, green CI on pmm-submodules, or when asked to attach FB screenshot to a PMM Jira ticket.
+---
+
+# Test Reporter
+
+You are **Test Reporter** — attach FB test evidence to Jira when all checks pass.
+
+**Input:** pmm-submodules PR number or linked Jira ticket from trigger context.
+
+## Knowledge (read by path)
+
+| Skill | Path |
+|-------|------|
+| FB checks (green gate) | `.cursor/skills/pmm-fb-tests/SKILL.md` |
+| Jira field update, attachments | `.cursor/skills/pmm-jira/SKILL.md` |
+| UI screenshot of Actions run | `.cursor/skills/pmm-ui-evidence/SKILL.md` |
+| Repo map | `.cursor/skills/pmm-repos/SKILL.md` |
+
+## Workflow
+
+1. **Verify green** — `gh pr checks <PR> -R Percona-Lab/pmm-submodules` — any `fail` → **stop** (no screenshot, no Jira update).
+2. **Screenshot** — FB Tests Actions run (not PR checks page). Use **computer use** per `pmm-ui-evidence`.
+3. **Update Jira** — `customfield_10492` with run URL and image attachment via Atlassian MCP.
+
+## Never
+
+- Attach green screenshot when checks failed
+- Clone `pmm-submodules`
+- Post Jira **comments** — this role only updates `customfield_10492` (and attachments). Comments are Test Runner / human QA.

.cursor/agents/test-runner.md

@@ -0,0 +1,40 @@
+---
+name: test-runner
+description: Use proactively for PMM manual QA on a Jira ticket — read requirements, verify against code, provision PMM on the cloud VM, execute tests (terminal + UI), post Developers-only Jira results, optionally open a pmm-qa PR. Trigger when the user asks to test a ticket, run QA, verify a PMM fix, or debug manual test steps for PMM-XXXX.
+---
+
+# Test Runner
+
+You are **Test Runner** — PMM manual QA cloud agent.
+
+**Input:** Jira key (e.g. `PMM-15196`) from the user, webhook `issueKey`, or Slack/Jira message. Natural language is fine ("please test PMM-15196") — you do not need a rigid slash prefix when the role file is loaded.
+
+## Knowledge (read by path before acting)
+
+| Skill | Path |
+|-------|------|
+| Jira read/write, visibility | `.cursor/skills/pmm-jira/SKILL.md` |
+| FB checks, JNKPercona | `.cursor/skills/pmm-fb-tests/SKILL.md` |
+| Docker, MicroVM, pmm-framework | `.cursor/skills/pmm-provisioning/SKILL.md` |
+| PR diffs, JSON dashboards | `.cursor/skills/pmm-git-diff/SKILL.md` |
+| UI screenshots / recordings | `.cursor/skills/pmm-ui-evidence/SKILL.md` |
+| Repo map, gh rules | `.cursor/skills/pmm-repos/SKILL.md` |
+
+Read each file when its step needs it. Do not guess field IDs or setup commands.
+
+## Workflow
+
+1. **Read ticket** — Atlassian MCP: summary, AC, `customfield_10083`, `customfield_10492`, dev links, comments. Cross-check with `pmm-git-diff` on linked `percona/pmm` / `percona/grafana` PRs.
+2. **Plan** — Short test plan: criteria, `DOCKER_ENV_VARIABLE`, `CLIENTS` / DB needs, post-provision steps, FB images from latest JNKPercona comment on linked pmm-submodules PR (`gh` only).
+3. **Provision** — Follow `pmm-provisioning` skill (`.cursor/environment.json` + `qa-integration/pmm_qa/pmm-framework/pmm-framework`).
+4. **Execute** — Terminal for API/CLI; **computer use** for UI per `pmm-ui-evidence`.
+5. **Report** — One Jira comment, **Developers visibility only** (see `pmm-jira`). Include pass/fail per criterion, artifact paths, blockers. Do not mark pass if criteria failed.
+6. **Automation decision** — After manual QA: if a minimal `pmm-qa` test adds clear value, implement and open PR to `percona/pmm-qa` only. Otherwise stop after Jira comment.
+
+## Never
+
+- Open PRs to `percona/pmm` or `percona/grafana`
+- `git clone` `Percona-Lab/pmm-submodules`
+- Modify `qa-integration/` in this agent PR (provisioning fixes are separate upstream PRs)
+- Trust "How to test" without reading PR diff
+- Post public Jira comments on QA results

.cursor/environment.json

@@ -0,0 +1,4 @@
+{
+  "install": "sudo service docker start 2>/dev/null || true; docker info >/dev/null 2>&1 || sudo dockerd >/tmp/dockerd.log 2>&1 & sleep 5; if ! command -v gh >/dev/null 2>&1; then curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg 2>/dev/null; sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg; echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main\" | sudo tee /etc/apt/sources.list.d/github-cli.list >/dev/null; sudo apt-get update -qq && sudo apt-get install -y gh; fi; npm install -g json-diff; QA_ROOT=\"${PWD}\"; [ -d pmm-qa ] && QA_ROOT=\"${PWD}/pmm-qa\"; cd \"${QA_ROOT}/codeceptjs-e2e\" && npm ci && npx playwright install --with-deps; cd \"${QA_ROOT}/cli\" && npm ci; cd \"${QA_ROOT}/qa-integration/pmm_qa\" && python3 -m venv virtenv && . virtenv/bin/activate && pip install --upgrade pip setuptools && pip install -r requirements.txt 'docker>=7.0.0'; ansible-galaxy collection install -p \"${HOME}/.ansible/collections\" community.docker >/dev/null 2>&1 || true; chmod +x \"${QA_ROOT}/qa-integration/pmm_qa/pmm-framework/pmm-framework\" \"${QA_ROOT}/.cursor/scripts/\"*.sh 2>/dev/null || true",
+  "start": "TMUX_CONF=/exec-daemon/tmux.portal.conf; if ! docker info >/dev/null 2>&1; then if ! tmux -f \"$TMUX_CONF\" has-session -t =dockerd 2>/dev/null; then tmux -f \"$TMUX_CONF\" new-session -d -s dockerd -c /tmp -- sudo dockerd; fi; for _ in $(seq 1 30); do [ -S /var/run/docker.sock ] && break; sleep 1; done; docker info >/dev/null 2>&1 || sudo chmod 666 /var/run/docker.sock; fi; export ANSIBLE_COLLECTIONS_PATH=\"${HOME}/.ansible/collections:${ANSIBLE_COLLECTIONS_PATH:-/usr/lib/python3/dist-packages/ansible_collections}\""
+}

.cursor/hooks.json

@@ -0,0 +1,19 @@
+{
+  "version": 1,
+  "hooks": {
+    "beforeShellExecution": [
+      {
+        "command": ".cursor/hooks/block-pmm-submodules-clone.sh",
+        "matcher": "git clone",
+        "failClosed": false
+      }
+    ],
+    "stop": [
+      {
+        "type": "prompt",
+        "prompt": "If this run performed PMM QA (test-runner or test-reporter role) and reported results to Jira, confirm a comment was added with Developers-only visibility (not public). If results were produced but no such Jira comment exists, return a followup_message instructing the agent to post using .cursor/skills/pmm-jira/SKILL.md visibility rules. Otherwise allow completion. Input: $ARGUMENTS",
+        "loop_limit": 2
+      }
+    ]
+  }
+}

.cursor/hooks/block-pmm-submodules-clone.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+# Block cloning Percona-Lab/pmm-submodules — use gh only.
+input=$(cat)
+command=$(echo "$input" | jq -r '.command // empty')
+
+if echo "$command" | grep -qiE 'git\s+clone.*pmm-submodules'; then
+  echo '{
+    "permission": "deny",
+    "user_message": "Cloning pmm-submodules is not allowed. Use gh pr checks and gh api on Percona-Lab/pmm-submodules instead.",
+    "agent_message": "Never clone pmm-submodules. Use gh only per pmm-repos skill."
+  }'
+  exit 0
+fi
+
+echo '{ "permission": "allow" }'
+exit 0

.cursor/scripts/pmm-ui-login.sh

@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# Log into PMM UI via playwright-cli (Cloud Agent / MicroVM).
+# Avoids the Grafana login form and self-signed TLS issues.
+#
+# Usage:
+#   export PMM_URL='https://127.0.0.1'      # optional, default below
+#   export ADMIN_PASSWORD='pmm3admin!'        # optional
+#   .cursor/scripts/pmm-ui-login.sh PMM-14576
+#
+# Session name: pmm-<SESSION_ID> (reuse the same id for follow-up UI commands).
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+PW_CONFIG="${REPO_ROOT}/.playwright/cli.config.json"
+
+SESSION_ID="${1:?usage: $0 <SESSION_ID>  e.g. PMM-14576}"
+PMM_URL="${PMM_URL:-https://127.0.0.1}"
+ADMIN_PASSWORD="${ADMIN_PASSWORD:-pmm3admin!}"
+PLAYWRIGHT_SESSION="pmm-${SESSION_ID}"
+
+export PLAYWRIGHT_MCP_IGNORE_HTTPS_ERRORS="${PLAYWRIGHT_MCP_IGNORE_HTTPS_ERRORS:-1}"
+export DISPLAY="${DISPLAY:-:1}"
+
+if ! command -v playwright-cli >/dev/null 2>&1; then
+  echo "ERROR: playwright-cli not found on PATH" >&2
+  exit 1
+fi
+
+AUTH_TOKEN="$(printf 'admin:%s' "$ADMIN_PASSWORD" | base64 -w0 2>/dev/null || printf 'admin:%s' "$ADMIN_PASSWORD" | base64)"
+
+PW_CLI=(playwright-cli --config="$PW_CONFIG")
+OPEN_ARGS=(-s="$PLAYWRIGHT_SESSION" open --headed "$PMM_URL")
+if [ "${PMM_UI_HEADED:-1}" = "0" ]; then
+  OPEN_ARGS=(-s="$PLAYWRIGHT_SESSION" open "$PMM_URL")
+fi
+
+"${PW_CLI[@]}" "${OPEN_ARGS[@]}"
+
+PMM_UI_WIDTH="${PMM_UI_WIDTH:-1920}"
+PMM_UI_HEIGHT="${PMM_UI_HEIGHT:-1200}"
+"${PW_CLI[@]}" -s="$PLAYWRIGHT_SESSION" resize "$PMM_UI_WIDTH" "$PMM_UI_HEIGHT" >/dev/null 2>&1 || true
+
+# shellcheck disable=SC2016
+"${PW_CLI[@]}" -s="$PLAYWRIGHT_SESSION" run-code "async page => {
+  const base = page.url().match(/^https?:\\/\\/[^/]+/)[0];
+  await page.setExtraHTTPHeaders({ Authorization: 'Basic ${AUTH_TOKEN}' });
+  await page.route('**/api/user/auth-tokens/rotate', r => r.fulfill({ status: 200, contentType: 'application/json', body: '{}' }));
+  await page.route('**/v1/users/me', r => r.fulfill({ status: 200, contentType: 'application/json', body: JSON.stringify({ alerting_tour_completed: true, product_tour_completed: true, snoozed_pmm_version: '', user_id: 1 }) }));
+  await page.route('**/v1/server/updates?force=**', r => r.fulfill({ status: 200, contentType: 'application/json', body: JSON.stringify({ installed: {}, last_check: new Date().toISOString(), latest: {}, update_available: false }) }));
+  const res = await page.request.post(base + '/graph/login', { data: { user: 'admin', password: '${ADMIN_PASSWORD}' } });
+  if (res.status() >= 400) throw new Error('Login failed: HTTP ' + res.status() + ' ' + await res.text());
+  await page.goto(base + '/pmm-ui/help');
+}"
+
+echo "PMM UI login OK (session=${PLAYWRIGHT_SESSION}, url=${PMM_URL})"

.cursor/settings.json

@@ -0,0 +1,7 @@
+{
+  "plugins": {
+    "atlassian": {
+      "enabled": true
+    }
+  }
+}

.cursor/skills/pmm-fb-tests/SKILL.md

@@ -0,0 +1,56 @@
+---
+name: pmm-fb-tests
+description: Analyze Percona-Lab/pmm-submodules FB Tests via gh pr checks, JNKPercona build comments, flaky triage, and map failures to pmm-qa GitHub workflow runners. Use when reading FB test status, finding server/client docker versions, or deciding what failed in FB CI.
+---
+
+# PMM FB Tests
+
+**pmm-submodules** — use `gh` only. **Never** `git clone` this repo.
+
+## Collect checks
+
+```bash
+gh pr checks <SUBMODULES_PR> -R Percona-Lab/pmm-submodules
+```
+
+- **Latest FB build only** — older comments/checks are invalid
+- Ignore JNKPercona "API tests have succeded/failed" comments
+
+## JNKPercona build comment (latest only)
+
+```bash
+gh api repos/Percona-Lab/pmm-submodules/issues/<PR>/comments \
+  --jq '[.[] | select(.user.login == "JNKPercona" and (.body | contains("Staging instance:"))) | {created_at, body}] | sort_by(.created_at) | .[-1]'
+```
+
+| Field in comment | Use as |
+|------------------|--------|
+| Server docker | `DOCKER_VERSION` / `PMM_SERVER_IMAGE` |
+| Watchtower docker | `WATCHTOWER_VERSION` |
+| Client tarball | `CLIENT_VERSION` |
+| Client docker | **ignore** for `CLIENT_VERSION` |
+
+## Map failures to workflows
+
+| Failed check pattern | pmm-qa workflow | Runner |
+|---------------------|-----------------|--------|
+| `@* UI tests` | `fb-e2e-suite.yml` | `runner-e2e-tests-codeceptjs.yml` (legacy) or `runner-e2e-tests-playwright.yml` (`e2e_tests/`) |
+| `CLI tests *` | `fb-integration-suite.yml` | `runner-integration-cli-tests.yml` |
+
+Extract `setup_services` / `tags_for_tests` or `services_list` / `cli_tag` from the failed job inputs.
+
+## Flaky triage
+
+Mark each failure: **relevant** (overlaps ticket) / **flaky** / **out of scope**. Only expand manual scope for **relevant** failures.
+
+## Green gate (Test Reporter)
+
+```bash
+gh pr checks <PR> -R Percona-Lab/pmm-submodules 2>&1 | grep -E '\tfail\t'
+```
+
+Any output → do **not** attach green screenshot to Jira.
+
+## Detail
+
+See [references/fb-tests.md](references/fb-tests.md) (full templates and screenshot workflow).

.cursor/skills/pmm-manual-test/fb-tests.md → .cursor/skills/pmm-fb-tests/references/fb-tests.md

@@ -110,6 +110,31 @@ Save screenshots to a temp path the user can find (e.g. workspace root or `%TEMP
 
 ## Update Jira
 
+### Comment visibility (mandatory)
+
+**Every Jira comment** on `perconadev.atlassian.net` PMM tickets MUST be restricted to the **Developers** role. Omitting visibility posts a **public** comment (visible to reporters/customers) — never do that for QA notes, triage, or test results.
+
+**Always** pass visibility when calling a comment tool:
+
+| MCP server | Tool | Required parameter |
+|------------|------|--------------------|
+| `user-mcp-atlassian` | `jira_add_comment` | `visibility: "{\"type\":\"role\",\"value\":\"Developers\"}"` |
+| `plugin-atlassian-atlassian` | `addCommentToJiraIssue` | `commentVisibility: {"type": "role", "value": "Developers"}` |
+
+Example (`user-mcp-atlassian`):
+
+```json
+jira_add_comment(
+  issue_key: "PMM-14915",
+  body: "*QA note (dev)* — ...",
+  visibility: "{\"type\":\"role\",\"value\":\"Developers\"}"
+)
+```
+
+If the MCP tool does not accept a visibility parameter, **do not post** — show the draft to the user and ask them to paste it with **Restrict to → Developers**.
+
+`jira_update_issue` on custom fields below does **not** use comment visibility (different mechanism).
+
 ### Custom fields
 
 | Field | ID | Use |

.cursor/skills/pmm-git-diff/SKILL.md

@@ -0,0 +1,41 @@
+---
+name: pmm-git-diff
+description: Read and summarize git diffs for percona/pmm and percona/grafana PRs linked to a PMM Jira ticket. Use before planning manual QA or when Test Runner needs PR scope. Includes JSON dashboard diff guidance.
+---
+
+# PMM git diff
+
+Read `.cursor/skills/pmm-repos/SKILL.md` for repo rules. Never clone `pmm-submodules`.
+
+## Find and diff PRs
+
+```bash
+gh search prs "<JIRA_KEY>" --repo percona/pmm --json number,title,url
+gh search prs "<JIRA_KEY>" --repo percona/grafana --json number,title,url
+gh pr diff <n> --repo percona/pmm
+```
+
+Return: files changed, behavioral summary, gaps in "How to test", suggested manual checks.
+
+## Large Grafana dashboard JSON diffs
+
+Grafana PRs often change dashboard JSON under `grafana/public/` or packaged dashboards. A raw `gh pr diff` can be **thousands of lines** of minified JSON — do not read the full diff inline.
+
+1. List changed JSON files: `gh pr diff <n> --repo percona/grafana --name-only | grep -E '\.json$'`
+2. For each file, use a structural JSON diff tool (installed in the PMM cloud environment):
+
+```bash
+gh pr diff <n> --repo percona/grafana -- path/to/dashboard.json | json-diff /dev/stdin
+# or save base/head and compare:
+json-diff base.json head.json
+```
+
+3. Summarize **what panels/queries/alerts changed**, not every byte.
+
+`json-diff` is installed globally via `.cursor/environment.json` (`npm install -g json-diff`).
+
+## When diff is too large even for json-diff
+
+- Read the PR description and linked Jira AC first
+- Use `gh api` to fetch only the files list and line change counts
+- Ask the dev for a testing focus if dashboard churn is unrelated to the ticket