fix: increase Test_Server_CapabilityError timeout to prevent flaky race (CRE-4319)

core/capabilities/remote/executable/server_test.go

@@ -211,7 +211,7 @@ func Test_Server_CapabilityError(t *testing.T) {
 
 	numCapabilityPeers := 4
 
-	callers, srvcs := testRemoteExecutableCapabilityServer(ctx, t, &commoncap.RemoteExecutableConfig{}, &TestErrorCapability{}, 10, 9, numCapabilityPeers, 3, 100*time.Millisecond, nil)
+	callers, srvcs := testRemoteExecutableCapabilityServer(ctx, t, &commoncap.RemoteExecutableConfig{}, &TestErrorCapability{}, 10, 9, numCapabilityPeers, 3, 10*time.Second, nil)
 
 	for _, caller := range callers {
 		_, err := caller.Execute(t.Context(),

tools/test/.agents/skills/_shared-jira-flaky-ops/README.md

@@ -0,0 +1,83 @@
+---
+name: shared-jira-flaky-ops-readme
+description: Index of shared JIRA operations for flaky-test skills. Defines the canonical slim-record schema, required env, and how to use these files from another skill.
+---
+
+# Shared JIRA Flaky-Test Operations
+
+This directory contains self-contained reference files for JIRA operations specific to flaky-test workflows. Any skill that needs to interact with JIRA around flaky tests should read the relevant file here instead of duplicating the logic.
+
+## When to use
+
+These files are **includable references**, not user-facing skills (no `SKILL.md`). Read the relevant file(s) when you need to perform one of the operations listed below. Each file is self-contained: it declares its inputs, steps, and outputs. You do not need to read files you are not using.
+
+## Required environment
+
+All operations require:
+- `cloudId` — the Atlassian cloud ID (from `mcp__atlassian__getAccessibleAtlassianResources` or cached from a prior call).
+- `accountId` — the current user's Atlassian account ID (from `mcp__atlassian__atlassianUserInfo`, cached in `phase_outputs.phase0`).
+
+## Operations index
+
+| File | Purpose | Key inputs |
+|------|---------|------------|
+| `investigation-comment.md` | Comment format for Investigation Updates; parsing prior-attempt comments | jira_key, outcome, field values |
+| `abandon-ticket.md` | Mid-flight abandonment: unassign → Open → Investigation Update comment (ABANDONED) | jira_key, reason |
+| `transition-ticket.md` | Transition a ticket to a semantic target state | jira_key, target |
+| `claim-ticket.md` | Assign to self and transition to "In Progress" | jira_key, accountId |
+| `fetch-flaky-tickets.md` | JQL search loop: fetch N eligible flaky-test tickets for a project key | KEY, N, cloudId, current_repo, nav_tool, lsp_available, repo root |
+| `validate-flaky-ticket.md` | Validate a single explicitly-provided ticket and build its slim record | jira_key, ci_run_url, cloudId, current_repo, nav_tool, lsp_available, repo root |
+| `recheck-ownership.md` | Verify the ticket is still assigned to us before touching files or pushing | jira_key, accountId |
+
+## How to include from another skill
+
+1. Read the relevant file(s) from this directory.
+2. Collect the declared inputs from your skill's context.
+3. Follow the file's steps exactly.
+4. Use the declared output schema to integrate the result back into your skill's state.
+
+Example: to claim a ticket, read `claim-ticket.md` and follow its steps with your `jira_key` and `accountId`.
+
+---
+
+## Canonical slim-record schema
+
+Defined here once. Referenced by `fetch-flaky-tickets.md` and `validate-flaky-ticket.md`. Both files must produce records conforming to this schema. Do not redefine it in either file.
+
+```json
+{
+  "jira_key":            "KEY-NNN | null",
+  "local_id":            "local-N | null",
+  "title":               "string",
+  "description":         "string",
+  "trunk_test_case_url": "https://app.trunk.io/.../test/{UUID} | null",
+  "test_case_id":        "{UUID} | null",
+  "package":             "github.com/owner/repo/path | null",
+  "test_name":           "TestFoo | TestFoo/subtest_name",
+  "previous_attempts":   [
+    {
+      "outcome":               "INCONCLUSIVE | PARTIAL_FIX | MISMATCH | SKIP_TOP_LEVEL | RETURNED_TO_QUEUE | ABANDONED | FIXED",
+      "date":                  "YYYY-MM-DD",
+      "summary":               "string",
+      "excluded_approaches":   ["string"],
+      "rejection_reasons":     ["string"],
+      "recommended_next_step": "string | null",
+      "full_text":             "string"
+    }
+  ],
+  "ci_run_url":          "string | null",
+  "provided_log_path":   "string | null",
+  "provided_log_text":   "string | null"
+}
+```
+
+**Field rules:**
+- `jira_key`: non-null in JIRA modes; null in local mode.
+- `local_id`: non-null in local mode (`local-1`, `local-2`, …); null in JIRA modes.
+- `test_case_id`: `customfield_13010` (bare UUID). If absent, extract UUID from `https://app.trunk.io/*/test/{UUID}` in description. Null only if neither yields a value. Always null in local mode.
+- `package`: `customfield_13009`. Null if absent.
+- `test_name`: `customfield_13007` (full path including subtest). If absent, longest `TestXxx`/`testXxx` token from title.
+- `trunk_test_case_url`: scan description for `https://app.trunk.io/*/test/{UUID}`; null if not found. Display only.
+- `previous_attempts`: parsed per `investigation-comment.md` parsing rules. Empty array in local mode.
+- `ci_run_url`: null in project mode (no upfront URL syntax). Populated from `KEY@URL` syntax in direct-ticket mode, or from a 3a fallback prompt.
+- `provided_log_path`, `provided_log_text`: null in JIRA modes; populated from `--log <path>` in local mode.

tools/test/.agents/skills/_shared-jira-flaky-ops/abandon-ticket.md

@@ -0,0 +1,34 @@
+---
+name: abandon-ticket
+description: Mid-flight abandonment procedure for a claimed flaky-test ticket. Unassigns, transitions back to Open, and writes an ABANDONED Investigation Update comment. Must never be skipped once a ticket has been claimed.
+---
+
+# Abandon Ticket
+
+Apply whenever a claimed ticket is stopped mid-flight — regardless of reason. This includes: user cancels, user skips, verdict is INCONCLUSIVE, PARTIAL_FIX is reverted, ownership conflict detected, SUT/AMBIGUOUS/INFRA auto-queue return, or session ends early.
+
+**Never leave a claimed ticket in "In Progress" with no assignee action.**
+
+## Inputs
+
+- `jira_key` — the JIRA ticket key
+- `reason` — one sentence describing why work stopped (used in "What was investigated" section)
+- `accountId` — the current user's Atlassian account ID (to confirm we own it before unassigning)
+
+## Steps
+
+Execute in order:
+
+1. `mcp__atlassian__editJiraIssue` → unassign the issue (set `assignee` to null).
+2. Follow `transition-ticket.md` with `jira_key` and `target = "Open"`.
+3. Follow `investigation-comment.md` to write an `addCommentToJiraIssue` call:
+   - **Outcome**: ABANDONED
+   - **What was investigated**: `reason` (the reason work stopped).
+   - **Hypothesis**: N/A
+   - **What was tried**: N/A
+   - **Why it didn't hold**: N/A
+   - **Recommended next step**: N/A
+
+## Output
+
+No structured output. Caller continues with other issues after this completes.

tools/test/.agents/skills/_shared-jira-flaky-ops/claim-ticket.md

@@ -0,0 +1,31 @@
+---
+name: claim-ticket
+description: Assign a flaky-test ticket to the current user and transition it to In Progress. Used after a ticket is approved for investigation.
+---
+
+# Claim Ticket
+
+## Inputs
+
+- `jira_key` — the JIRA ticket key (e.g. `CRE-5719`)
+- `accountId` — the current user's Atlassian account ID (from `phase_outputs.phase0`)
+
+## Steps
+
+Execute in order — wait for each step to succeed before proceeding:
+
+1. `mcp__atlassian__editJiraIssue` → assign the issue to `accountId` (set `assignee.accountId = accountId`). Wait for success.
+2. Follow `transition-ticket.md` with `jira_key` and `target = "In Progress"`.
+   - If the transition fails: log available transitions and stop. Do not leave the ticket assigned without transitioning.
+
+## Output
+
+```json
+{ "success": true, "jira_key": "KEY-NNN" }
+```
+
+or on failure:
+
+```json
+{ "success": false, "jira_key": "KEY-NNN", "error": "<reason>" }
+```

tools/test/.agents/skills/_shared-jira-flaky-ops/fetch-flaky-tickets.md

@@ -0,0 +1,79 @@
+---
+name: fetch-flaky-tickets
+description: JQL search loop that fetches N eligible flaky-test tickets for a project key, filters out cross-repo and system-test tickets, resolves each to a local test function, and returns slim records. Extracted from phase2a.
+---
+
+# Fetch Flaky Tickets
+
+## Inputs
+
+- `KEY` — JIRA project key (e.g. `CRE`)
+- `N` — number of eligible records to return
+- `cloudId` — Atlassian cloud ID
+- `current_repo` — `{owner}/{repo}` extracted from `git remote get-url origin`
+- `nav_tool` — `"lsp"` | `"crg"` (from `phase_outputs.phase0`)
+- `lsp_available` — boolean (from `phase_outputs.phase0`)
+- repo root path
+
+## Output
+
+```json
+{
+  "slim_records": [ /* see README.md slim-record schema */ ],
+  "skipped": { "cross_repo": 0, "system_tests": 0, "not_found": 0 }
+}
+```
+
+Never return raw JIRA API objects — caller only receives slim records.
+
+## Slim-record schema
+
+See `README.md` in this directory for the canonical schema. In the records produced here:
+- `jira_key` non-null, `local_id` null.
+- `ci_run_url` always null (project mode has no upfront URL syntax; 3a fallback may prompt later).
+- `provided_log_path`, `provided_log_text` null.
+
+## Loop
+
+```
+results = []
+cursor = null
+while len(results) < N:
+  fetch N issues via mcp__atlassian__searchJiraIssuesUsingJql:
+    jql:           project = {KEY} AND labels = "flaky-test" AND status = "Open" ORDER BY created DESC
+    fields:        ["summary", "description", "comment", "status", "assignee",
+                    "customfield_13010", "customfield_13009", "customfield_13007"]
+    maxResults:    N
+    nextPageToken: cursor  (omit on first call)
+
+  for each issue (in order):
+    1. Repo check (zero-cost): extract {owner}/{repo} from customfield_13009
+       (2nd + 3rd segments after github.com/). Mismatch → skip (cross_repo++).
+       If customfield_13009 absent, scan description for
+       https://github.com/{owner}/{repo} or a "Repo:" / "Repository:" field.
+
+    2. System-tests exclusion (zero-cost): if customfield_13009 starts with
+       github.com/smartcontractkit/chainlink/system-tests/ → skip (system_tests++).
+
+    3. Test function check: extract top-level function name from customfield_13007
+       (part before first /), fall back to longest TestXxx token in title if absent.
+       - nav_tool="lsp" or lsp_available=true: LSP definition lookup
+       - nav_tool="crg": mcp__code-review-graph__semantic_search_nodes_tool
+       - last resort only: grep -rl "func {TestName}" .
+       Not found → skip (not_found++).
+
+    4. Eligible: build slim record (see schema), append to results.
+       Stop once len(results) == N.
+
+  cursor = nextPageToken from response
+  if no more pages: break
+```
+
+## Field extraction rules
+
+- `test_case_id`: `customfield_13010` (bare UUID). If absent, extract UUID from `https://app.trunk.io/*/test/{UUID}` in description. Null only if neither yields a value.
+- `package`: `customfield_13009`. Null if absent.
+- `test_name`: `customfield_13007` (full path including subtest, e.g. `TestFoo/subtest`). If absent, longest `TestXxx`/`testXxx` token from title.
+- `trunk_test_case_url`: scan description for `https://app.trunk.io/*/test/{UUID}`; null if not found. Display only.
+- `previous_attempts`: parse per `investigation-comment.md` parsing rules.
+- If any custom field is absent from the search response, call `mcp__atlassian__getJiraIssue` with `fields=["summary","description","comment","status","assignee","customfield_13010","customfield_13009","customfield_13007"]` for that issue as a fallback.

tools/test/.agents/skills/_shared-jira-flaky-ops/investigation-comment.md

@@ -0,0 +1,59 @@
+---
+name: investigation-comment
+description: Authoritative format for Investigation Update JIRA comments and rules for parsing previous-attempt comments. Read by any phase or skill that writes to or reads from JIRA investigation history.
+---
+
+# Investigation Update Comment Format
+
+## Writing a comment
+
+Every `mcp__atlassian__addCommentToJiraIssue` call for an investigation update uses the structure below. Always include all five sections — write `N/A` for any that don't apply. **Style: concise and matter-of-fact. 1–3 sentences per section. No narrative padding, no hedge words.**
+
+```markdown
+## Investigation Update — {OUTCOME} · {YYYY-MM-DD}
+
+**Outcome**: {OUTCOME}
+**Investigator**: {display name from atlassianUserInfo}
+**Classification**: {TEST | SUT | AMBIGUOUS | INFRA} (confidence: {high | low | none}) | N/A
+
+### What was investigated
+{The failure mode and where analysis focused.}
+
+### Hypothesis
+{The proposed root cause, or N/A.}
+
+### What was tried
+{The fix or approach applied or proposed, or N/A.}
+
+### Why it didn't hold
+{Objections, test results, or reason the fix was rejected or reverted — or N/A.}
+
+### Recommended next step
+{Concrete actionable direction for the next investigator, or N/A.}
+```
+
+**Outcome values:**
+| Value | When to use |
+|-------|------------|
+| `FIXED` | Fix verified, PR created |
+| `INCONCLUSIVE` | Debate unresolved — no fix applied |
+| `PARTIAL_FIX` | Fix applied, tests still failed, reverted |
+| `RETURNED_TO_QUEUE` | SUT / AMBIGUOUS / INFRA classification |
+| `CLOSED_SUBTEST` | Failure originates in a `t.Run` subtest, not the top-level function |
+| `ABANDONED` | Mid-flight stop for any reason |
+
+---
+
+## Parsing previous-attempt comments
+
+Scan JIRA comments for `## Investigation Update — {OUTCOME}`. For each match extract:
+
+- `outcome` — the OUTCOME token from the heading
+- `date` — the date after `·`
+- `full_text` — the full comment text
+- `excluded_approaches` — content of `### What was tried` (skip if "N/A")
+- `rejection_reasons` — content of `### Why it didn't hold` (skip if "N/A")
+- `recommended_next_step` — content of `### Recommended next step` (null if "N/A")
+- `summary` — content of `### What was investigated`
+
+Fall back to keyword scanning for non-standard-format comments.

tools/test/.agents/skills/_shared-jira-flaky-ops/recheck-ownership.md

@@ -0,0 +1,35 @@
+---
+name: recheck-ownership
+description: Verify that a claimed ticket is still assigned to the current user before touching files or pushing. Used in phase4 (before applying fixes) and phase5 (before pushing).
+---
+
+# Recheck Ownership
+
+## Inputs
+
+- `jira_key` — the JIRA ticket key
+- `accountId` — the current user's Atlassian account ID (cached from `phase_outputs.phase0`)
+
+## Steps
+
+1. Call `mcp__atlassian__getJiraIssue` with `fields=["assignee"]`.
+2. Compare `assignee.accountId` to `accountId`.
+
+## Output
+
+```json
+{ "result": "ok" }
+```
+
+or if reassigned:
+
+```json
+{ "result": "reassigned", "reassigned_to": "<displayName>" }
+```
+
+## Caller responsibility
+
+If `result = "reassigned"`:
+- Report: *"KEY-NNN is now assigned to {displayName} — reach out before proceeding."*
+- Follow `abandon-ticket.md` for this ticket.
+- Continue with remaining issues.

tools/test/.agents/skills/_shared-jira-flaky-ops/transition-ticket.md

@@ -0,0 +1,33 @@
+---
+name: transition-ticket
+description: Generic transition operation for flaky-test JIRA tickets. Takes a semantic target state, resolves it to the actual transition name, and applies it. Referenced by claim-ticket.md and abandon-ticket.md.
+---
+
+# Transition Ticket
+
+## Inputs
+
+- `jira_key` — the JIRA ticket key (e.g. `CRE-5719`)
+- `target` — semantic state: `"In Progress"` | `"In Review"` | `"Open"` | `"Won't Do"` | `"Done"`
+
+## Steps
+
+1. Call `mcp__atlassian__getTransitionsForJiraIssue` with `jira_key`.
+2. Match `target` to an available transition using the alias table below. Pick the **first alias that appears** in the response.
+3. Call `mcp__atlassian__transitionJiraIssue` with the matched transition ID.
+   - For closing targets (`"Won't Do"`, `"Done"`): if the transition supports a `resolution` field, set `resolution = "Won't Do"` (fallback: `"Won't Fix"`).
+4. Output: `{ "success": true, "transition_name_used": "<actual name>" }` on success, or `{ "success": false, "error": "<reason>", "available_transitions": ["..."] }` if no alias matched.
+
+## Target alias table
+
+| Semantic target | Try these names in order |
+|----------------|--------------------------|
+| `In Progress` | "In Progress", "In Development", "Active", "Start Progress" |
+| `In Review` | "In Review", "In Code Review", "Code Review", "Review" |
+| `Open` | "Open", "Reopen", "Backlog", "To Do", "Reopened" |
+| `Won't Do` | "Won't Do", "Won't Fix", "Reject", "Close", "Done" |
+| `Done` | "Done", "Closed", "Resolved", "Close", "Resolve" |
+
+## Error handling
+
+If no alias matches any available transition: log all available transition names and return `success: false`. The caller decides how to proceed (stop, skip, or pick manually).