Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- Clarified namespace guidance for `extra` extension keys and examples ([#56](https://github.com/ctrf-io/ctrf/pull/56)).
- Clarified immutability guidance for emitted CTRF report artifacts ([#55](https://github.com/ctrf-io/ctrf/pull/55)).
- Clarified `tags` as simple keyless classifications and `labels` as structured key-value test metadata ([#54](https://github.com/ctrf-io/ctrf/pull/54)).
- Allowed non-empty arrays of strings, numbers, and booleans as label values ([#54](https://github.com/ctrf-io/ctrf/pull/54)).
26 changes: 18 additions & 8 deletions examples/comprehensive.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,19 @@
"timestamp": "2025-11-24T12:00:00Z",
"generatedBy": "example-ci",
"extra": {
"pipelineStage": "e2e",
"trigger": "pull_request"
"myorg.ci": {
"pipelineStage": "e2e",
"trigger": "pull_request"
}
},
"results": {
"tool": {
"name": "example-runner",
"version": "3.5.0",
"extra": {
"plugins": ["retry", "screenshots", "video"]
"myorg.ci": {
"plugins": ["retry", "screenshots", "video"]
}
}
},
"summary": {
Expand Down Expand Up @@ -216,13 +220,17 @@
"testEnvironment": "staging",
"healthy": false,
"extra": {
"nodeVersion": "18.0.0",
"region": "us-west-2"
"myorg.ci": {
"nodeVersion": "18.0.0",
"region": "us-west-2"
}
}
},
"extra": {
"shardIndex": 1,
"totalShards": 4
"myorg.ci": {
"shardIndex": 1,
"totalShards": 4
}
}
},
"insights": {
Expand Down Expand Up @@ -267,7 +275,9 @@
"buildUrl": "https://ci.example.com/builds/442",
"commit": "f6e5d4c3b2a1",
"extra": {
"branch": "main"
"myorg.ci": {
"branch": "main"
}
}
}
}
162 changes: 149 additions & 13 deletions spec/ctrf.md
Original file line number Diff line number Diff line change
Expand Up @@ -333,8 +333,6 @@ A CTRF document MUST include:
- `specVersion` - MUST follow Semantic Versioning 2.0.0.
- `results` - MUST contain the results of a single test execution run.

Additional top-level fields MUST be included within an `extra` object.

### 4.3. Schema Conformance

The document MUST validate against the normative JSON Schema provided in Appendix A.
Expand All @@ -353,10 +351,50 @@ Consumers MUST reject any document containing unknown properties outside of `ext

### 4.6. Extension Mechanism

- Objects named `extra` MAY contain arbitrary data and are the only supported extensibility points in CTRF.
- Producers MAY include tool-specific or domain-specific metadata within an `extra` object.
- Consumers MAY require particular fields within an `extra` object for their platform or workflow; such fields MUST NOT appear outside `extra`.
- Consumers SHOULD ignore unrecognized fields inside any `extra` object unless they explicitly depend on them.
Objects named `extra` are the only supported extensibility points in CTRF.

Producers MAY include tool-specific or domain-specific metadata within an `extra` object.
Consumers MUST ignore unrecognized fields within any `extra` object and MUST NOT reject documents solely due to their presence.

The `extra` object defines a mapping of extension keys to values.

#### 4.6.1. Extension Keys

Each property within an `extra` object represents a distinct extension.

Extension keys:

- MUST be strings
- SHOULD be namespaced to avoid collisions
- SHOULD use a stable, recognizable prefix identifying the extension owner (e.g. `myorg.myproject`)
- SHOULD include a delimiter (e.g. `.` or `/`) separating the namespace from the identifier

The `ctrf.*` namespace is reserved for CTRF-defined extensions.
Producers other than CTRF MUST NOT use the `ctrf.*` namespace.
Comment thread
Copilot marked this conversation as resolved.
Outdated

Extension keys are opaque identifiers.
Consumers MUST NOT interpret extension keys as hierarchical paths.

A producer SHOULD use the same namespace prefix consistently across all `extra` objects within a single document.
If a producer contributes extensions at multiple levels (for example, at the top level, within `results`, `summary`, or individual tests), the same namespace prefix SHOULD be used for all of that producer's extension keys throughout the document.

#### 4.6.2. Extension Values

Each extension key maps to a value.

- The value MAY be any valid JSON type.
- Extensions containing multiple related fields SHOULD use an object value.
- Extensions with a single value MAY use a scalar.

Producers SHOULD group related extension data under a single extension key.

#### 4.6.3. Extension Semantics

The structure and semantics of each extension value are defined by the extension producer.

- CTRF does not define or validate extension value structure.
- Consumers MUST treat unknown extension values as opaque.
Comment thread
Copilot marked this conversation as resolved.
Outdated
- Consumers MUST NOT assume the presence or meaning of any extension unless explicitly supported.

### 4.7. Versioning and Forward Compatibility

Expand Down Expand Up @@ -2985,15 +3023,19 @@ It includes:
"timestamp": "2025-11-24T12:00:00Z",
"generatedBy": "example-ci",
"extra": {
"pipelineStage": "e2e",
"trigger": "pull_request"
"myorg.ci": {
"pipelineStage": "e2e",
"trigger": "pull_request"
}
},
"results": {
"tool": {
"name": "example-runner",
"version": "5.0.0",
"extra": {
"parallel": true
"myorg.ci": {
"parallel": true
}
}
},
"summary": {
Expand All @@ -3009,7 +3051,9 @@ It includes:
"stop": 1700000312000,
"duration": 12000,
"extra": {
"reportedBy": "summary-module"
"myorg.ci": {
"reportedBy": "summary-module"
}
}
},
"tests": [
Expand Down Expand Up @@ -3111,11 +3155,15 @@ It includes:
"testEnvironment": "staging",
"healthy": false,
"extra": {
"container": "node:18"
"myorg.ci": {
"container": "node:18"
}
}
},
"extra": {
"runShard": 1
"myorg.ci": {
"runShard": 1
}
}
},
"insights": {
Expand All @@ -3139,7 +3187,95 @@ It includes:
"source": "main branch - last stable build",
"buildNumber": 9800,
"extra": {
"note": "Release candidate baseline"
"myorg.ci": {
"note": "Release candidate baseline"
}
}
}
}
```

### D.6. CTRF Document with Namespaced Extra Objects

This example demonstrates how producers SHOULD use namespaced extension keys within `extra` objects, grouping related fields under a single key as an object value, and using the same namespace prefix consistently across all `extra` objects in the document.

Two producers contribute extensions in this example:

- `myorg.ci` (the CI system) contributes extensions at the top level, `results`, `summary`, and per-test level — using the `myorg.ci` namespace key consistently throughout.
- `acme.test-management` (a test management integration) contributes extensions at the per-test level only — using the `acme.test-management` namespace key consistently.

The `ctrf.*` namespace is reserved for CTRF-defined extensions and MUST NOT be used by other producers.

```json title="CTRF document with namespaced extra objects"
{
"reportFormat": "CTRF",
"specVersion": "0.0.0",
"extra": {
"myorg.ci": {
"pipeline": {
"stage": "e2e",
"triggerEvent": "pull_request",
"region": "us-east-1"
}
}
},
"results": {
"tool": {
"name": "example-runner",
"version": "2.0.0"
},
"summary": {
"tests": 2,
"passed": 1,
"failed": 1,
"skipped": 0,
"pending": 0,
"other": 0,
"start": 1700000000000,
"stop": 1700000005000,
"extra": {
"myorg.ci": {
"retryBudgetUsed": 2,
"retryBudgetTotal": 10
}
}
},
"tests": [
{
"name": "checkout flow completes",
"status": "passed",
"duration": 2000,
"extra": {
"myorg.ci": {
"featureFlag": "new-checkout-ui"
},
"acme.test-management": {
"issueKey": "SHOP-1234",
"testCycleName": "Sprint 42 Regression"
}
}
},
{
"name": "payment fails gracefully",
"status": "failed",
"duration": 3000,
"message": "Expected error banner to be visible",
"extra": {
"myorg.ci": {
"featureFlag": "new-payment-ui"
},
"acme.test-management": {
"issueKey": "SHOP-1235",
"testCycleName": "Sprint 42 Regression"
}
}
}
],
"extra": {
"myorg.ci": {
"deploymentId": "deploy-abc123",
"environment": "staging"
}
}
}
}
Expand Down
Loading