chore(dev): add AI-assisted component maturity evaluation skill#25518
chore(dev): add AI-assisted component maturity evaluation skill#25518pront wants to merge 8 commits into
Conversation
Adds a Claude Code project skill (`/vector-components-maturity-eval`) that evaluates the maturity of all Vector components monthly and publishes a report to Confluence. The skill collects signals from CUE metadata, git history, GitHub issues, and test coverage, then uses AI judgment to recommend promote/keep/watch/deprecate per component. Also excludes `.claude/skill-reports/` from git — reports are generated locally and published to Confluence, not tracked in the repo. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
pront
changed the title
github-actions
Bot
added
the
domain: ci
pront
added
the
no-changelog
- Remove auto-publish to Confluence (Phase 7); report stays local - Add Near Misses section separate from strict Promotion Candidates - Harden Phase 4 bug matching: flag generic names as ambiguous - Align frontmatter description path with actual output path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 49f8617d27
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| add_path "src/${kind}/${name}.rs" | ||
| add_path "src/${kind}/${name}" |
There was a problem hiding this comment.
When this runs for source components whose implementations live under shared directories, the only paths added are src/sources/<name>.rs and src/sources/<name>, so repo search shows existing tests in src/sources/prometheus/{scrape,pushgateway,remote_write}.rs and src/sources/file_descriptors/{stdin,file_descriptor}.rs are never counted. That makes the generated report mark stable components such as prometheus_scrape, stdin, and file_descriptor as having zero unit tests, which can put them on the watch list incorrectly and understate beta promotion candidates.
Useful? React with 👍 / 👎.
The type: bug label is deprecated; bugs are now identified via the GitHub issue Type field (type:Bug search qualifier). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
prometheus_scrape, prometheus_remote_write, prometheus_pushgateway, stdin, and file_descriptor all live under shared subdirectories. Without explicit mappings the unit test count falls back to zero, incorrectly putting stable components on the watch list. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8333461f95
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| --- | ||
|
|
||
| ## Phase 2: Bulk Signal Collection | ||
|
|
There was a problem hiding this comment.
For open issues that still carry the legacy type: bug label but have no GitHub Issue Type set, this query drops them before Phase 4, so affected components can be reported as having no open bugs. I checked the current tracker and found examples such as #21837 with label type: bug and Type: No type; the bug collection should include both type:Bug and label:"type: bug" (or otherwise merge the legacy label path) until those issues are migrated.
Useful? React with 👍 / 👎.
| Issues use the GitHub issue **Type** field (not a label). The type name is `Bug`. | ||
|
|
||
| ```bash | ||
| gh issue list --state open --search "type:Bug" --json number,title,url --limit 500 2>/dev/null |
There was a problem hiding this comment.
When an open bug has the component only in its GitHub labels, this command discards that signal and Phase 4's title scan won't count it for the component. I checked the current tracker and found open bugs such as #24984 with label source: opentelemetry but a title that does not contain the canonical opentelemetry component name, so the report can undercount bugs and incorrectly promote or clear watch status unless labels are included and matched.
Useful? React with 👍 / 👎.
| if [ ${#paths[@]} -eq 0 ]; then | ||
| count=0 | ||
| elif command -v rg >/dev/null 2>&1; then | ||
| count=$(rg -n "$test_attr_regex" "${paths[@]}" 2>/dev/null | wc -l | tr -d ' ') |
There was a problem hiding this comment.
For components whose directory contains feature-gated integration_tests.rs, this recursive rg counts those integration tests as unit tests even though Phase 2e tracks integration coverage separately. For example, src/sources/windows_event_log/integration_tests.rs has 34 #[tokio::test] entries gated by the integration-test feature, so beta components can satisfy the unit tests > 10 promotion criterion without actually having that many unit tests.
Useful? React with 👍 / 👎.
| - Bugs are identified by the GitHub issue **Type** field (`type:Bug` in search), not by label — the old `type: bug` label is deprecated | ||
| - Working directory is the Vector repo root | ||
|
|
||
| **Parent/shared CUE files**: Some CUE files define shared configuration for families of components and have no `development` field of their own (children inherit it). These will appear as "unknown" status when grepped. Known parent files: `sinks/aws_cloudwatch.cue`, `sinks/datadog.cue`, `sinks/gcp.cue`, `sinks/humio.cue`, `sinks/influxdb.cue`, `sinks/sematext.cue`, `sinks/splunk_hec.cue`, and possibly `sinks/statsd.cue`, `sources/syslog.cue`. Exclude these from per-component counts; note them separately. |
There was a problem hiding this comment.
Excluding sinks/statsd.cue and sources/syslog.cue from per-component counts drops two real components from the monthly inventory whenever their inherited development field is not resolved by grep. Both have concrete implementations (src/sinks/statsd and src/sources/syslog.rs) and inherit the stable/common classes from the socket components, so they should be resolved through inheritance rather than omitted from counts and recommendations.
Useful? React with 👍 / 👎.
2 participants
Summary
Adds a Claude Code project skill that evaluates the maturity of all Vector components on a monthly cadence and publishes the results to Confluence.
.claude/skills/vector-components-maturity-eval/SKILL.mdand is invoked via/vector-components-maturity-evalin any Claude Code session on this repocommonly_used), git history (component age, config churn), GitHub issues (type: buglabel), and test coverage (unit + integration).claude/skill-reports/from git — reports are generated locally and published to Confluence, not tracked in the repoFirst report to be published soon after we gain more confidence in this skill.
Vector configuration
N/A — this is a developer tooling addition, not a Vector configuration change.
How did you test this PR?
Ran the skill manually and generated the first monthly report. Verified signal collection (CUE parsing, git log, GitHub API, test detection), AI evaluation, and Confluence publishing end-to-end.
Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.