Repository navigation
test: add versioned, freeze and watch cases to the characterisation harness - #447
Merged
Merged
Conversation
Require reference/index.qmd in every Quarto project left by a case whose Gauntlet spec lists export names, matching the rendered Gauntlet checks. A missing test-side index fails the case even when the base also lacks it. The report names the missing file and gives the base-side result. Call package variations `variant`, since they can change more than the configuration, and allow extra command arguments per case. Record the variant, reference expectation and command order in manifests. Reject recordings with another manifest schema to avoid reading older recordings as the new format.
Build gdtest_user_guide_cli, then freeze one user-guide page. Freeze prepares the build directory afresh. With no post-render files in that directory, its changes expose differences from build preparation: CLI reference pages, shared sidebars, the interlinks index and Quarto config. Report these known defects in a separate section with each side's result. Known gaps never fail the run. Report a gap as closed when only the base side shows it.
Build gdtest_kitchen_sink in a generated Git repository. Tag the original package v0.1, then add one public function and tag that commit v0.2. Build version 0.2 from the checkout; version 0.1 takes its API from tag v0.1, so its reference lacks the new function. Fixed identities, dates and an empty Git config give repository copies identical commit hashes. Exclude .git from recordings because Git rewrites its stat data in a copy. For this case only, let stub Quarto write placeholder pages in _site so the versioned build can assemble the site. Cover the assembled site, version map, aliases and redirects. Record the Git version too.
Run great-docs build --watch on gdtest_page_tags. For this case only, keep stub Quarto preview running until interrupted. After preview starts, append a line to a user-guide page and wait up to 10 seconds for it to reach another package file. Then interrupt the process group as Ctrl-C would. Restore default interrupt handling in child processes so the run also works when background shells pass on ignored interrupts. Fail the case if preview never starts, the command exits with a non-zero status or outlives the interrupt, or the output contains a traceback. Watch mode currently copies no edited source into the build directory. Report the missing rebuild as a known gap without failing the run, and compare each side's rebuild result like other recorded results.
Cover the layout that predates docs/ by moving the configuration and user guide of gdtest_page_status to the package root. Leave the module unset so the build must discover the documented package and write the site to great-docs/.
Enable the hero in a variant of gdtest_homepage_ug_subdirs. The two Gauntlet packages that blend the first user-guide page into the homepage show no hero, leaving that part of the homepage uncovered.
Disable the skill link in a variant of gdtest_go_cli to leave its README homepage margin empty. As a Go project, it has no PyPI link or Python API for llms.txt. Existing Gauntlet README homepages all have metadata margins.
Update module documentation and Makefile help for freeze and watch mode, the 40-case default tier and the Git requirement. Explain that failed checks fail a run while known gaps are reported without failing it.
📚 Preview this PR's docs locallyFetch the just-built site and open it in your browser: # this exact build
great-docs preview --run 40541930659
# or always the latest build for this PR
great-docs preview --pr 447Auth & tips
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR widens the characterisation harness that guards the
core.pyrefactor. It ran onlygreat-docs initandbuild, so a refactor could break versioned builds,great-docs freezeorgreat-docs build --watchwithout the harness noticing. The default tier grows from 34 to 40 cases: a versioned build from a generated repository with two tags, a freeze after a build, a watch-mode run with a source edit and an interrupt, a configuration at the package root withoutmodule:, a homepage hero blended from the user guide, and a README homepage with an empty metadata margin. Builds whose Gauntlet spec lists exports must also leavereference/index.qmd.Two existing defects show up in the new cases: freeze prepares files differently from a build, and watch mode never picks up an edited source. The report lists them as known gaps, which do not fail the run, so a fix shows as the gap closing.