Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,28 @@ Kingfisher is a Rust secret scanner, live credential validator, revocation helpe
- After markdown/doc changes, verify local documentation links when practical.
- After `docs-site/` source changes, rebuild with `docs-site/.venv/bin/mkdocs build -f docs-site/mkdocs.yml` when practical so generated output stays in sync.

## Crate Versions And Publication

- Before finishing changes to a publishable crate, check whether its packaged contents
differ from the same version on crates.io. Packaged tests, examples, README files,
dependency requirements, licenses, and generated data count, even without runtime/API changes.
- Crate versions are independent of the CLI tag and Python SDK version. Bump an
already-published crate when its package changes; do not bump every crate for a CLI
release or bump an unpublished prepared version unnecessarily. Choose patch/minor/major
versions according to the public API compatibility contract.
- Update dependent minimum requirements and `Cargo.lock` with each internal crate bump.
Recheck dependent packages too: changing their published manifests can require their
own version bumps. Regenerate rule-bundle provenance after manifest/lockfile changes;
never hand-edit its hashes or bypass the immutable-package comparison.
- Build fresh archives with `cargo package --locked` for `kingfisher-core`,
`kingfisher-rules`, `kingfisher-scanner`, and `kingfisher-bin`, then run the read-only
`python3 scripts/publish-crates.py --tag v<CLI-version>` plan before declaring a release
ready. Keep package verification enabled; `--allow-dirty` is only for local pre-commit checks.
- For a metadata-only correction after a successful binary release, commit and merge
it into `main`, then run `make release crates` from a clean, current `main` checkout.
Preserve the existing public tag and artifacts. Runtime changes need a new binary
release, not a broader recovery exception. See [publishing guidance](docs/PUBLISHING.md).

## Cross-Platform Tests

- New and modified tests must work on Windows (x64 and arm64), macOS, and Linux. Review fixture setup and subprocess calls for platform assumptions even when developing on Unix.
Expand Down
5 changes: 4 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,14 @@ All notable changes to this project will be documented in this file.
requests also refuse redirects; configure final canonical URLs instead.
- Prepare Python SDK **1.6.0** with explicit analysis and source-attribute scanning,
and rules/scanner crates **3.0.0** for public Rust schema changes. Existing
YAML/TOML rule files and Python APIs remain compatible; core stays at **1.0.4**.
YAML/TOML rule files and Python APIs remain compatible; core uses the compatible **1.0.5** patch.
Rust callers should follow the [3.0 migration guide](docs/LIBRARY.md#migrating-to-rules-and-scanner-30).

### Improvements

- Correct crates.io release metadata for the packaged core regression tests and
verify the regenerated rules-manifest hash during crates-only recovery. Verified
metadata-only recovery does not require another Python SDK release.
- Reduce malformed-container and recognized-placeholder false positives with
conservative, bounded offline private-key checks shared by the CLI and SDKs.
These checks do not establish cryptographic or live validity. See the
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ gpui-kit = { version = "=0.6.6", optional = true }
command-group = { version = "5.0.1", optional = true }
image = { version = "0.25", default-features = false, features = ["png"], optional = true }
# Library crates
kingfisher-core = { version = "1.0.4", path = "crates/kingfisher-core" }
kingfisher-core = { version = "1.0.5", path = "crates/kingfisher-core" }
kingfisher-rules = { version = "3.0.0", path = "crates/kingfisher-rules" }
kingfisher-scanner = { version = "3.0.0", path = "crates/kingfisher-scanner", features = ["validation", "archives", "context", "extraction", "git", "__cli-internals"] }

Expand Down
2 changes: 1 addition & 1 deletion crates/kingfisher-core/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "kingfisher-core"
version = "1.0.4"
version = "1.0.5"
description = "Core types and traits for Kingfisher secret scanner"
edition.workspace = true
rust-version.workspace = true
Expand Down
2 changes: 1 addition & 1 deletion crates/kingfisher-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ semantic versioning; breaking changes require a new major version.

```toml
[dependencies]
kingfisher-core = "1.0.4"
kingfisher-core = "1.0.5"
```

```rust
Expand Down
2 changes: 1 addition & 1 deletion crates/kingfisher-rules/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ __scanner-internals = []

[dependencies]
# Internal dependencies
kingfisher-core = { version = "1.0.4", path = "../kingfisher-core" }
kingfisher-core = { version = "1.0.5", path = "../kingfisher-core" }

# Serialization
serde.workspace = true
Expand Down
2 changes: 1 addition & 1 deletion crates/kingfisher-rules/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ These records document what was imported; they do not establish upstream authors
Use rules/scanner `3.0.0` together. Existing YAML/TOML rule files remain compatible;
public Rust schema fields and enum variants require source updates. See the
[3.0 migration guide](../../docs/LIBRARY.md#migrating-to-rules-and-scanner-30).
Core stays at 1.0.4; Python SDK 1.6.0 preserves its existing Python API.
Core uses the compatible 1.0.5 patch; Python SDK 1.6.0 preserves its existing Python API.

## Migrating to 2.0

Expand Down
6 changes: 3 additions & 3 deletions crates/kingfisher-rules/generated/provenance.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@
},
"bundle_format": "gzip / KFRULES v1 / converted YAML",
"generator_input_sha256": {
"Cargo.lock": "366caf1b91c054c61214f99542ff47cb27dcac8698dfcbe12c83e14c851c9ff8",
"Cargo.toml": "2e2eb92b537116ca4ef4f31a432b9744aff138c05976e8cb59d00be57970a73e",
"Cargo.lock": "b3d5780ad992ae01aec14c8c6e6436d54b5fe83bb832ed064d537cb4981d5133",
"Cargo.toml": "49aee40b7285ac3e1be70f7a8bb403d13f4b066b3956aa3ed0b2c98f4188aa7e",
"LICENSE": "43070e2d4e532684de521b885f385d0841030efa2b1a20bafb76133a5e1379c1",
"crates/kingfisher-rules/Cargo.toml": "c83c5aa9e2e2e9790f6eb032d0e97b481b9054b33e74b07cd59609ab1ac49b17",
"crates/kingfisher-rules/Cargo.toml": "a4312ea684367264638bc6c6aa5379697828b1e3e93b6ee024337b37608a9311",
"crates/kingfisher-rules/build_support/betterleaks.rs": "831d97ea9edcfbb841b7ef5b962e48a6d3ed26ed4c2e85109ac2b49f9425ebfa",
"crates/kingfisher-rules/build_support/builtin_docs.rs": "3edb2b444acfb2a7ade1ba7f6e1a3cbac14e6a743e01156293683fcdb4a77225",
"crates/kingfisher-rules/build_support/imported_capabilities.rs": "2ddeaf7ffdcbedb55d1ccb13eb010ad04a5f451cecd91fd418033c1390242914",
Expand Down
2 changes: 1 addition & 1 deletion crates/kingfisher-scanner/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ validation-all = ["validation"]

[dependencies]
# Internal dependencies
kingfisher-core = { version = "1.0.4", path = "../kingfisher-core" }
kingfisher-core = { version = "1.0.5", path = "../kingfisher-core" }
kingfisher-rules = { version = "3.0.0", path = "../kingfisher-rules", features = ["__scanner-internals"] }

# Error handling
Expand Down
2 changes: 1 addition & 1 deletion crates/kingfisher-scanner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ and [rule format guide](../../docs/RULES.md).
Use rules/scanner `3.0.0` together. Existing YAML/TOML rule files remain compatible;
public Rust schema fields and enum variants require source updates. See the
[3.0 migration guide](../../docs/LIBRARY.md#migrating-to-rules-and-scanner-30).
Core stays at 1.0.4; Python SDK 1.6.0 preserves its existing Python API.
Core uses the compatible 1.0.5 patch; Python SDK 1.6.0 preserves its existing Python API.

## Migrating to 2.0

Expand Down
9 changes: 5 additions & 4 deletions docs-site/docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,7 @@ description: "Kingfisher release history: new features, rules, bug fixes, and im

All notable changes to this project will be documented in this file.

## [v2.15.0] (Unreleased)

Changes since the published v2.14.0 release.
## [v2.15.0]

### Added

Expand Down Expand Up @@ -56,11 +54,14 @@ Changes since the published v2.14.0 release.
requests also refuse redirects; configure final canonical URLs instead.
- Prepare Python SDK **1.6.0** with explicit analysis and source-attribute scanning,
and rules/scanner crates **3.0.0** for public Rust schema changes. Existing
YAML/TOML rule files and Python APIs remain compatible; core stays at **1.0.4**.
YAML/TOML rule files and Python APIs remain compatible; core uses the compatible **1.0.5** patch.
Rust callers should follow the [3.0 migration guide](reference/library.md#migrating-to-rules-and-scanner-30).

### Improvements

- Correct crates.io release metadata for the packaged core regression tests and
verify the regenerated rules-manifest hash during crates-only recovery. Verified
metadata-only recovery does not require another Python SDK release.
- Reduce malformed-container and recognized-placeholder false positives with
conservative, bounded offline private-key checks shared by the CLI and SDKs.
These checks do not establish cryptographic or live validity. See the
Expand Down
6 changes: 3 additions & 3 deletions docs-site/docs/reference/library.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ description: "Embed secret detection, live validation, and explicit revocation i

For in-process Python detection, validation and revocation, see the [Python SDK](../reference/python-bindings.md).

The prepared library releases are **kingfisher-core 1.0.4**,
The prepared library releases are **kingfisher-core 1.0.5**,
**kingfisher-rules 3.0.0**, and **kingfisher-scanner 3.0.0**.
They require Rust **1.99** or newer and are versioned independently of the
`kingfisher-bin` CLI, currently **2.15.0**. See [publishing](https://github.com/mongodb/kingfisher/blob/main/docs/PUBLISHING.md).
Expand Down Expand Up @@ -881,7 +881,7 @@ runner.


The first published `1.0.0` established the stable contract. The rules and scanner
3.0.0 releases start their next stable major; core stays on the compatible 1.x API at 1.0.4. Within each
3.0.0 releases start their next stable major; core stays on the compatible 1.x API at 1.0.5. Within each
stable major:

- Existing reachable public Rust items, signatures, fields, trait implementations,
Expand Down Expand Up @@ -912,7 +912,7 @@ behavior; review and behavioral tests remain required.
### Migrating to rules and scanner 3.0

Upgrade both crates to `"3.0.0"`; scanner 3.x exposes rules 3.x types.
Existing YAML/TOML files and Python APIs remain compatible. Core stays at 1.0.4;
Existing YAML/TOML files and Python APIs remain compatible. Core uses the compatible 1.0.5 patch;
CLI 2.15.0 and Python SDK 1.6.0 are independently versioned.

- `HttpRequest` literals need `fallback_urls: Vec::new()`; `url` remains a string.
Expand Down
6 changes: 3 additions & 3 deletions docs/LIBRARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ For in-process Python detection, validation and revocation, see the [Python SDK]

[← Back to README](../README.md)

The prepared library releases are **kingfisher-core 1.0.4**,
The prepared library releases are **kingfisher-core 1.0.5**,
**kingfisher-rules 3.0.0**, and **kingfisher-scanner 3.0.0**.
They require Rust **1.99** or newer and are versioned independently of the
`kingfisher-bin` CLI, currently **2.15.0**. See [publishing](PUBLISHING.md).
Expand Down Expand Up @@ -878,7 +878,7 @@ runner.


The first published `1.0.0` established the stable contract. The rules and scanner
3.0.0 releases start their next stable major; core stays on the compatible 1.x API at 1.0.4. Within each
3.0.0 releases start their next stable major; core stays on the compatible 1.x API at 1.0.5. Within each
stable major:

- Existing reachable public Rust items, signatures, fields, trait implementations,
Expand Down Expand Up @@ -909,7 +909,7 @@ behavior; review and behavioral tests remain required.
### Migrating to rules and scanner 3.0

Upgrade both crates to `"3.0.0"`; scanner 3.x exposes rules 3.x types.
Existing YAML/TOML files and Python APIs remain compatible. Core stays at 1.0.4;
Existing YAML/TOML files and Python APIs remain compatible. Core uses the compatible 1.0.5 patch;
CLI 2.15.0 and Python SDK 1.6.0 are independently versioned.

- `HttpRequest` literals need `fallback_urls: Vec::new()`; `url` remains a string.
Expand Down
21 changes: 14 additions & 7 deletions docs/PUBLISHING.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The manifests currently prepare these releases; a Git tag does not override Carg

| Package | Version | Published targets |
| ------- | ------- | ----------------- |
| `kingfisher-core` | `1.0.4` | `kingfisher_core` library |
| `kingfisher-core` | `1.0.5` | `kingfisher_core` library |
| `kingfisher-rules` | `3.0.0` | `kingfisher_rules` library |
| `kingfisher-scanner` | `3.0.0` | `kingfisher_scanner` library |
| `kingfisher-bin` | `2.15.0` | `kingfisher` executable and library |
Expand All @@ -30,8 +30,9 @@ The Python SDK (`kingfisher-secret-scanner`) is independently versioned at

The libraries follow semantic versioning and do not inherit the CLI version.
Rules/scanner 3.0.0 add public Rust schema fields and Expr variants; upgrade both
crates together. Existing YAML/TOML files remain compatible. Core is unchanged at
1.0.4. Python SDK 1.6.0 adds compatible APIs for explicit analysis and source attributes.
crates together. Existing YAML/TOML files remain compatible. Core 1.0.5 packages
the added regression tests without changing its public API. Python SDK 1.6.0 adds
compatible APIs for explicit analysis and source attributes.
See the [3.0 migration guide](LIBRARY.md#migrating-to-rules-and-scanner-30).
`kingfisher-rule-bundle` is `0.1.0` with `publish = false`. The external
`kingfisher-vectorscan` dependency is versioned separately.
Expand Down Expand Up @@ -273,21 +274,27 @@ lockfile, provenance and documentation corrections into `main`, then dispatch:

```sh
gh workflow run release.yml --repo mongodb/kingfisher --ref main \
-f crates_only=true -f release_run_id=39339690046 \
-f crates_only=true -f release_run_id=40610865292 \
-f expected_sha="$(git rev-parse HEAD)"
```

Use the original binary release's run ID. The recovery gate requires its successful
build/test/release jobs, verifies the public tag, and rejects runtime source, dependency,
feature or catalog changes. Only the core patch version and corresponding dependency,
lockfile and provenance metadata may change; the independent SDK version correction
lockfile and provenance metadata may change. Regenerated root manifest, lockfile and
rules-manifest hashes must match their committed files; all other provenance stays
unchanged. The independent SDK version correction
from 1.4.0 to 1.5.0 and its matching lockfile entry are also allowed. Library tests and all four extracted-crate
build checks still run before authentication and publication. The ordinary release
path retains its exact tag/commit guards. A runtime change needs a new binary release.

The Python SDK preflight reuses the same strict metadata check, so a verified core
version/dependency-metadata correction does not require another SDK release. Actual
SDK/shared runtime or unrelated dependency changes still require a new SDK version.

The original failed run uses its original commit and remains historical; rerunning it
cannot pick up a later metadata correction. The new recovery run publishes the missing
crate versions without retagging `v2.14.0` or republishing its assets.
crate versions without retagging `v2.15.0` or republishing its assets.

## Independent terminal commands

Expand Down Expand Up @@ -321,7 +328,7 @@ publishing, first create fresh archives:

```sh
cargo package --locked -p kingfisher-core -p kingfisher-rules -p kingfisher-scanner -p kingfisher-bin
python3 scripts/publish-crates.py --tag v2.14.0
python3 scripts/publish-crates.py --tag v2.15.0
python3 -m unittest discover -s scripts/tests -v
```

Expand Down
10 changes: 6 additions & 4 deletions scripts/check-crates-recovery.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,20 @@
DEPENDENTS = ("Cargo.toml", "crates/kingfisher-rules/Cargo.toml",
"crates/kingfisher-scanner/Cargo.toml")
PROVENANCE = "crates/kingfisher-rules/generated/provenance.json"
PROVENANCE_INPUTS = ("Cargo.toml", "Cargo.lock")
# The core requirement bump also changes the rules manifest's recorded input hash.
# Manifest semantics are restricted above the provenance comparison in check_metadata.
PROVENANCE_INPUTS = ("Cargo.toml", "Cargo.lock", "crates/kingfisher-rules/Cargo.toml")
PLATFORMS = ("Linux x64", "Linux arm64", "macOS x64", "macOS arm64", "Windows x64", "Windows arm64")
REQUIRED_JOBS = {"Verify release ref and version", "Public GitHub Release", *PLATFORMS} | {
f"tests / {platform}" for platform in PLATFORMS
}
MAINTENANCE_FILES = {
"README.md", "CHANGELOG.md", "docs/LIBRARY.md", "docs/PUBLISHING.md",
"AGENTS.md", "README.md", "CHANGELOG.md", "docs/LIBRARY.md", "docs/PUBLISHING.md",
"crates/kingfisher-core/README.md", "crates/kingfisher-rules/README.md",
"crates/kingfisher-scanner/README.md", "docs-site/docs/changelog.md",
"docs-site/docs/reference/library.md", ".github/RELEASING.md",
".github/workflows/release.yml", "scripts/check-crates-recovery.py",
"scripts/tests/test_crates_recovery.py", "Makefile",
"scripts/tests/test_crates_recovery.py", "scripts/tests/test_publish_crates.py", "Makefile",
"scripts/release-packages.py", "scripts/tests/test_release_packages.py",
"scripts/check-sdk-release.py", "scripts/tests/test_sdk_preflight.py",
"scripts/tests/test_python_sdk_release.py", "scripts/tests/test_release_makefile.py",
Expand Down Expand Up @@ -120,7 +122,7 @@ def toml(commit, path):
raise ValueError(f"Recovery provenance does not match {path}; regenerate the bundle")
after["generator_input_sha256"][path] = before["generator_input_sha256"][path]
if before != after:
raise ValueError("Recovery changes rule-bundle provenance beyond workspace manifest/lock hashes")
raise ValueError("Recovery changes rule-bundle provenance beyond approved metadata input hashes")


def github_json(path, token):
Expand Down
16 changes: 16 additions & 0 deletions scripts/check-sdk-release.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
means an unused version; an existing empty or yanked release is still reserved.
"""
import argparse
import importlib.util
import json
import os
from pathlib import Path
Expand All @@ -29,6 +30,11 @@
SHA = re.compile(r"[0-9a-f]{40}\Z")
SHARED_CRATES = ("kingfisher-python", "kingfisher-core", "kingfisher-rules", "kingfisher-scanner")

_recovery_spec = importlib.util.spec_from_file_location(
"crates_recovery", ROOT / "scripts/check-crates-recovery.py")
crates_recovery = importlib.util.module_from_spec(_recovery_spec)
_recovery_spec.loader.exec_module(crates_recovery)


def sdk_version(root=ROOT):
with (root / MANIFEST).open("rb") as stream:
Expand Down Expand Up @@ -130,6 +136,16 @@ def runtime_path(path):

def runtime_changed(root, base, head):
paths = git(root, "diff", "--name-only", "-z", "--no-renames", base, head, "--").split("\0")
if "crates/kingfisher-core/Cargo.toml" in paths:
# Reuse the strict recovery gate, not a blanket exception for manifest or
# version edits. Source/dependency drift or unverified provenance falls
# back to the normal SDK runtime checks and still requires a new release.
try:
crates_recovery.check_metadata(root, base, head)
except (ValueError, KeyError, TypeError, OSError, subprocess.CalledProcessError):
pass
else:
return False
if any(runtime_path(path) for path in paths):
return True
if "Cargo.lock" in paths:
Expand Down
Loading
Loading