diff --git a/.claude/skills/vector-components-maturity-eval/SKILL.md b/.claude/skills/vector-components-maturity-eval/SKILL.md new file mode 100644 index 0000000000000..82e0753636350 --- /dev/null +++ b/.claude/skills/vector-components-maturity-eval/SKILL.md @@ -0,0 +1,292 @@ +--- +name: vector-components-maturity-eval +description: Evaluates all Vector component maturity levels and writes a monthly markdown report to .claude/skill-reports/maturity-YYYY-MM.md. Use when asked to evaluate component maturity or generate the monthly maturity report. +--- + +You are the Vector Component Maturity Evaluator. Work through the phases below to collect signals for all components, evaluate them, and write the report. + + +## Maturity Criteria + +From `website/content/en/docs/architecture/guarantees.md`: + +**Stable** requires ALL of: + +- >50 production users for a sustained period without issue (proxy: age + zero open bugs) +- >4 months community testing (proxy: file age in git) +- API stable and unlikely to change (proxy: low config churn) +- No major open bugs + +**Beta**: Does not meet stable criteria — use with caution in production. +**Deprecated**: Will be removed in next major version. + +## Signal Priority + +1. **Open bugs** (highest weight) — open GitHub issues with issue type `Bug` mentioning this component +2. **Test coverage** (second) — integration test exists? line coverage % from Datadog CI? +3. Equal weight: age, config churn (6 months), docs quality (AI judgment) + +--- + +## Phase 1: Inventory + +```bash +# All canonical component CUE files (exclude generated/ subdirs) +find website/cue/reference/components/sources \ + website/cue/reference/components/transforms \ + website/cue/reference/components/sinks \ + -maxdepth 1 -name "*.cue" | sort + +# Integration test directories +ls tests/integration/ +``` + +--- + +## Phase 2: Bulk Signal Collection + +Use single shell loops to collect all signals at once — do not make one Bash call per component. + +### 2a. Open GitHub bugs + +Issues use the GitHub issue **Type** field. The type name is `Bug`. + +```bash +gh issue list --state open --search "type:Bug" --json number,title,url,labels --limit 500 2>/dev/null +``` + +Store the full list. You will map bugs to components in Phase 4 by scanning titles and labels. + +### 2b. Component age — date each CUE file was first committed + +```bash +for kind in sources transforms sinks; do + for f in website/cue/reference/components/${kind}/*.cue; do + name=$(basename "$f" .cue) + first_date=$(git log --follow --format="%ad" --date=short -- "$f" 2>/dev/null | tail -1) + echo "${kind}/${name}|${first_date}" + done +done +``` + +### 2c. Config churn — commits to CUE file in last 6 months + +Count commits to both the hand-written component file and its generated counterpart (the generated file carries the actual configuration API and may change without touching the top-level file). + +```bash +for kind in sources transforms sinks; do + for f in website/cue/reference/components/${kind}/*.cue; do + name=$(basename "$f" .cue) + generated="website/cue/reference/components/${kind}/generated/${name}.cue" + paths=("$f") + [ -f "$generated" ] && paths+=("$generated") + count=$(git log --since="6 months ago" --oneline -- "${paths[@]}" 2>/dev/null | sort -u | wc -l | tr -d ' ') + echo "${kind}/${name}|${count}" + done +done +``` + +### 2d. Code coverage from the merged-lcov artifact + +```bash +# Derive artifact name from the workflow file — single source of truth +artifact_name=$(grep -A2 'Upload merged coverage artifact' \ + .github/workflows/coverage.yml | grep 'name:' | awk '{print $2}') + +[ -z "$artifact_name" ] && { echo "ERROR: could not find artifact name in .github/workflows/coverage.yml — check the 'Upload merged coverage artifact' step" >&2; exit 1; } +echo "Artifact name: $artifact_name" + +# Find the latest successful coverage run on master +run_id=$(gh run list --workflow=coverage.yml --branch=master --status=success \ + --limit=1 --json databaseId --jq '.[0].databaseId') + +[ -z "$run_id" ] && { echo "ERROR: no successful coverage run found on master" >&2; exit 1; } +echo "Coverage run: $run_id" + +# Download the artifact +rm -rf /tmp/vector-coverage +gh run download "$run_id" -n "$artifact_name" --dir /tmp/vector-coverage \ + || { echo "ERROR: $artifact_name artifact missing — check .github/workflows/coverage.yml has an 'Upload merged coverage artifact' step and the run predates it" >&2; exit 1; } + +# Emit file|coverage_pct for every source file in the report +awk ' + /^SF:/ { file=$0; sub(/^SF:/, "", file); lf=0; lh=0 } + /^LF:/ { lf=$0; sub(/^LF:/, "", lf) } + /^LH:/ { lh=$0; sub(/^LH:/, "", lh) } + /^end_of_record/ { if (lf+0 > 0) printf "%s|%.1f\n", file, (lh/lf)*100 } +' /tmp/vector-coverage/merged-lcov.info +``` + +**Mapping files to components:** For each component `` of type ``, collect all LCOV lines whose path starts with `src//` (catches both `src/sources/kafka.rs` and `src/sources/kafka/`). Average the coverage percentages across those lines. Components with no matching path are marked `n/a`. + +This resolves shared-directory sources (e.g. `src/sources/prometheus/scrape.rs` → `prometheus_scrape`) and shared utilities (e.g. `src/sources/prometheus/parser.rs` → counted for all three prometheus sources) without any explicit path mapping. + +Record for each component: `coverage_pct` (number or `"n/a"`). + +### 2e. Integration test presence + +From the `ls tests/integration/` output, note which component names have a matching directory. A component has an integration test if `tests/integration/` or a close variant exists. + +--- + +## Phase 3: Read CUE Files + +Read each component's CUE file in batches of 10–15 (parallel Read calls in a single response). Extract: + +- `development` value — `"stable"`, `"beta"`, or `"deprecated"` +- Whether `how_it_works` has substantive prose. If it references a shared CUE object, read that referenced object and judge the resolved prose; shared populated docs count as substantive. +- Whether `description` (top-level) is meaningful: at least two sentences explaining what the component does and when to use it +- Whether there are non-trivial `examples` in the configuration section + +**Docs quality judgment**: mark docs as `complete`, `partial`, or `minimal`. + +- `complete`: all three present (description, how_it_works prose, examples) +- `partial`: one or two present +- `minimal`: none meaningful or all are placeholders/references + +--- + +## Phase 4: Match Bugs to Components + +For each issue from Phase 2a, match it to components using two signals: + +1. **Component labels**: issues are often labeled `source: `, `sink: `, or `transform: `. A label match is authoritative — count the issue for that component without needing a title match. +2. **Title scan**: scan the title for canonical component names from the CUE filenames. Only count a title match when the component name appears unambiguously (e.g. `"kafka source: ..."`, `"[loki sink]"`, or the name as a standalone token next to "source", "sink", or "transform"). + +**Avoid false matches on generic terms.** Names like `file`, `http`, `socket`, `vector`, `console`, and `internal` appear in many issue titles without referring to a specific component. Apply the title-match rules strictly for these. + +Count matched open bugs per component. If an issue mentions multiple components, count it for each. If a title is ambiguous and carries no component label, do not count it; collect these in an "Unmatched / ambiguous" list in the report's Reference section for manual review. + +--- + +## Phase 5: Evaluate Each Component + +For every component, assign one recommendation: + +| Rec | Meaning | +| --- | --- | +| **promote** | Beta → stable candidate | +| **keep** | No change warranted | +| **watch** | Stable with concerning signals | +| **deprecate-candidate** | Little activity, superseded, or already deprecated in CUE | + +**Promote** (beta only): 0–1 open bugs AND (integration test OR coverage ≥ 50%) AND age > 4 months AND churn ≤ 5 commits AND docs at least `partial`. + +**Watch** (stable only): ≥ 3 open bugs, OR churn > 10 commits (API instability), OR (coverage < 10% AND no integration test). + +Use judgment for borderline cases. A component with 2 bugs but a long stable history is different from one with 2 bugs filed in the last month. + +--- + +## Phase 6: Write Report + +Create the output directory and write the report: + +```bash +mkdir -p .claude/skill-reports +``` + +Write to `.claude/skill-reports/maturity-YYYY-MM.md` using the actual current year and month. + +--- + +### Report format + +```markdown +# Vector Component Maturity Report — YYYY-MM + +_Generated: YYYY-MM-DD. N sources · N transforms · N sinks (N total)._ + +--- + +## Summary + +| Category | Count | +|----------|-------| +| Promote candidates (beta → stable) | N | +| Near misses (one criterion short) | N | +| Watch list (stable with concerns) | N | +| Deprecation candidates | N | +| No change | N | + +--- + +## Promotion Candidates + +_Beta components that strictly meet all stable criteria: 0–1 open bugs, integration test or coverage ≥ 50%, age > 4 months, churn ≤ 5 commits, docs at least `partial`._ + +| Component | Type | Open Bugs | Int Tests | Coverage | Age | Churn (6mo) | Docs | +|-----------|------|-----------|-----------|----------|-----|-------------|------| +| `name` | source | 0 | ✓ | 72% | 18mo | 2 | complete | + +--- + +## Near Misses + +_Beta components that fail exactly one promotion criterion. List the blocking criterion._ + +| Component | Type | Open Bugs | Int Tests | Coverage | Age | Churn (6mo) | Docs | Blocking | +|-----------|------|-----------|-----------|----------|-----|-------------|------|----------| + +--- + +## Watch List + +_Stable components with signals worth a human look._ + +| Component | Type | Open Bugs | Notes | +|-----------|------|-----------|-------| +| `name` | sink | 4 | 2 labeled critical | + +--- + +## Deprecation Candidates + +| Component | Type | Notes | +|-----------|------|-------| + +--- + +## Full Inventory + +
+Beta components (N) + +| Component | Type | Open Bugs | Int Tests | Coverage | Age | Churn | Docs | Rec | +|-----------|------|-----------|-----------|----------|-----|-------|------|-----| + +
+ +
+Stable components (N) + +| Component | Type | Open Bugs | Int Tests | Rec | +|-----------|------|-----------|-----------|-----| + +
+``` + +Notes column: five words max. Keep prose minimal. Tables over paragraphs. All issue number references must be hyperlinked: in markdown use `[#NNNNN](https://github.com/vectordotdev/vector/issues/NNNNN)`, in HTML use `#NNNNN`. + +--- + +## Phase 7: Done + +The report is complete. Tell the user where the file was written. Do not publish anywhere — distribution is a separate decision made by the user after reviewing the report. + +--- + +## Reference + +- CUE files at `website/cue/reference/components/{sources,transforms,sinks}/` are authoritative (ignore `generated/` subdirs) +- `gh` is pre-authenticated for `vectordotdev/vector` +- Bugs are identified by the GitHub issue **Type** field (`type:Bug` in search) — issues use the Type field, not labels +- Working directory is the Vector repo root +- **Coverage artifact**: produced by the `Upload merged coverage artifact` step in the `coverage-upload` job (`.github/workflows/coverage.yml`), retained for 90 days. The artifact name is read from the workflow file at runtime — if Phase 2d fails to find it, check that step exists and the name matches. +- **Coverage thresholds**: ≥50% = well covered, 20–50% = partial, <20% = low, n/a = no data + +**Parent/shared CUE files**: Some CUE files define shared configuration for families of components and have no `development` field of their own (children inherit it). Known true parent files (exclude from per-component inventory): `sinks/aws_cloudwatch.cue`, `sinks/datadog.cue`, `sinks/gcp.cue`, `sinks/humio.cue`, `sinks/influxdb.cue`, `sinks/sematext.cue`, `sinks/splunk_hec.cue`. `sinks/statsd.cue` and `sources/syslog.cue` are real components whose `development` value is inherited — read the child CUE files to resolve it and include them in the inventory. + +**Integration test name mapping**: Integration test directory names use hyphens, not underscores (e.g. `tests/integration/docker-logs/` maps to `docker_logs`, `tests/integration/windows-event-log/` maps to `windows_event_log`). Some test directories cover multiple components under a shared umbrella (e.g. `aws/` covers all `aws_*` sources and sinks, `gcp/` covers all `gcp_*` sinks, `prometheus/` covers `prometheus_scrape`, `prometheus_exporter`, and `prometheus_remote_write`). + +**CUE age caveat**: Many component CUE files show a first-commit date of 2020-10-xx, which reflects the batch import of the website CUE system — not the actual component introduction date. Treat these dates as lower bounds and note the caveat in the report. diff --git a/.github/actions/spelling/allow.txt b/.github/actions/spelling/allow.txt index 7ac468f99526c..794ac298146bb 100644 --- a/.github/actions/spelling/allow.txt +++ b/.github/actions/spelling/allow.txt @@ -44,6 +44,7 @@ Arnova arshiyasolei Asus Atlassian +atlassian atleastonce atmostonce Attab @@ -69,6 +70,7 @@ bitflags bitnami bitwidth blackbox +blockquotes Blaupunkt Blusens buildname @@ -85,6 +87,7 @@ Ceph Chromecast Citrix cksum +claude Cloudflare Cloudfone Cmx @@ -237,6 +240,7 @@ HTTPDATE https Huawei humungus +hyperlinked Hyundai icecream Ideapad @@ -322,10 +326,12 @@ LYF macbook Malata manden +maxdepth maxmind maxminddb Maxthon MCRF +mcp Mediacom Medion MEF @@ -386,6 +392,7 @@ ntapi ntfs Odys onig +oneline opendal Openpeak OPENPGP @@ -579,3 +586,4 @@ grpcurl linting lexers binstalled +lcov diff --git a/.github/workflows/semantic.yml b/.github/workflows/semantic.yml index 616c5d15b83cc..5ac3f34414157 100644 --- a/.github/workflows/semantic.yml +++ b/.github/workflows/semantic.yml @@ -35,6 +35,7 @@ jobs: scopes: | administration + ai api api top api tap diff --git a/.gitignore b/.gitignore index a713eeb297f9a..4030e169d9ea1 100644 --- a/.gitignore +++ b/.gitignore @@ -71,3 +71,6 @@ local/ # vscode .vscode/ + +# Claude skill-generated reports (published to Confluence, not tracked) +.claude/skill-reports/