Docs/guillotina docs v2 batch master

.github/workflows/continuous-integration.yml

@@ -81,3 +81,37 @@ jobs:
         uses: codecov/codecov-action@v1
         with:
           file: ./coverage.xml
+
+  docs:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout the repository
+        uses: actions/checkout@v2
+
+      - name: Setup Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: "3.11"
+
+      - name: Install docs dependencies
+        run: |
+          pip install -r requirements.txt
+          pip install -r docs-requirements.txt
+          pip install -e .
+
+      - name: Generate LLM and crawler assets
+        run: |
+          python docs/scripts/generate_llms_assets.py
+
+      - name: Validate llms assets
+        run: |
+          python docs/scripts/check_llms_assets.py
+
+      - name: Build documentation
+        run: |
+          python -m sphinx -b html docs/source docs/build/html
+
+      - name: Check links
+        run: |
+          python -m sphinx -b linkcheck docs/source docs/build/linkcheck

AGENTS.md

@@ -0,0 +1,35 @@
+# AGENTS.md
+
+## Project Overview
+
+- Purpose: Guillotina async REST API/resource database project.
+- Main stack: Python package with Sphinx documentation, pytest tests, and setuptools packaging.
+- Key entrypoints: `guillotina.commands:command_runner`, console scripts `guillotina` and `g`.
+
+## Development Commands
+
+- Install dev environment: follow `README.rst` with a virtualenv, requirements, contrib requirements, and editable install with test extras.
+- Docs linkcheck used in this workspace: `.venv/bin/python -m sphinx -b linkcheck docs/source docs/build/linkcheck`.
+- Syntax smoke for Sphinx config: `python -m compileall -q docs/source/conf.py`.
+- Tests from repo docs: `./bin/pytest guillotina`.
+
+## Validation
+
+- For docs-only changes, run Sphinx linkcheck when dependencies are available.
+- Linkcheck currently includes known unrelated failures for localhost example URLs and may be affected by external DNS availability.
+
+## Deployment Notes
+
+- Iskra workspace policy applies: do not deploy this repo unless deployment is explicitly requested.
+- Record the working branch in Ops Tracker closeout evidence.
+
+## Constraints / Gotchas
+
+- Iskra repos require a dedicated task branch; do not commit directly to `main` or `master`.
+- Keep untracked virtualenvs and local generated artifacts out of commits.
+- Do not commit secrets, local credentials, or kubeconfig material.
+
+## Task Closeout Notes
+
+- Record commit, branch, validation, and no-deploy status in Ops Tracker.
+- Update `CHANGELOG.rst` for task outcomes when closing tracked work.

CHANGELOG.rst

@@ -4,6 +4,35 @@ CHANGELOG
 7.0.7 (unreleased)
 ------------------
 
+- Docs: Replace stale linkcheck-ignored external references for Hypercorn and
+  plone.restapi users/groups, and clear the docs linkcheck ignore list
+  [nilbacardit26]
+- Docs: Expose Developer Guides section from documentation index
+  [nilbacardit26]
+- Docs: Add final documentation quality and rollout readiness report
+  [nilbacardit26]
+- Docs: Add contributor docs and maintenance cadence documentation
+  [nilbacardit26]
+- Docs: Add retrieval benchmark queries and lexical coverage checker script
+  [nilbacardit26]
+- CI: Add docs quality gate job (build, linkcheck, llms checks)
+  [nilbacardit26]
+- Docs: Add generated llms/robots/sitemap assets and Sphinx root publishing config
+  [nilbacardit26]
+- Docs: Add core concepts pages (storage, catalog, middleware, task vars, security model)
+  [nilbacardit26]
+- Docs: Add core concepts pages (architecture, object model, traversal, request/response, transactions)
+  [nilbacardit26]
+- Docs: Add section index pages for concepts, how-to, and operations
+  [nilbacardit26]
+- Docs: Rework root documentation navigation toward Diataxis sections
+  [nilbacardit26]
+- Docs: Phase 1 hygiene updates (stray text cleanup, Python version refresh, link fixes)
+  [nilbacardit26]
+- Docs: Add baseline docs IA artifacts (inventory, redirect map, ADR-001)
+  [nilbacardit26]
+- Docs: Add ADR for LLM discovery policy decisions (canonical URL, bots, size cap)
+  [nilbacardit26]
 - Docs: Update documentation and configuration settings
 - Chore: Update sphinx-guillotina-theme version to 1.0.9
   [rboixaderg]

README.rst

@@ -2,7 +2,7 @@ Introduction
 ============
 
 .. image:: https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat
-   :target: http://guillotina.readthedocs.io/en/latest/
+   :target: https://guillotina.readthedocs.io/en/latest/
 
 .. image:: https://github.com/plone/guillotina/workflows/CI/badge.svg
    :target: https://github.com/plone/guillotina/actions
@@ -31,7 +31,7 @@ Introduction
    :target: https://hub.docker.com/r/guillotina/guillotina
    :alt: Docker Cloud Build Status
 
-Please `read the detailed docs <http://guillotina.readthedocs.io/en/latest/>`_
+Please `read the detailed docs <https://guillotina.readthedocs.io/en/latest/>`_
 
 
 This is the working project of the next generation Guillotina server based on asyncio.
@@ -99,13 +99,13 @@ Read more `Guillotina-Heroku <https://github.com/guillotinaweb/guillotina-heroku
 Getting started with development
 --------------------------------
 
-Using pip (requires Python > 3.7)
+Using pip (requires Python >= 3.10)
 
 .. code-block:: shell
 
     git clone git@github.com:plone/guillotina.git
     cd guillotina
-    python3.7 -m venv .
+    python3 -m venv .
     ./bin/pip install -r requirements.txt
     ./bin/pip install -r contrib-requirements.txt
     ./bin/pip install -e '.[test]'

docs-requirements.txt

@@ -7,3 +7,4 @@ sphinx-autodoc-typehints==3.0.1
 docutils==0.21
 readme-renderer==43.0
 async-asgi-testclient<2.0.0
+PyYAML==6.0.2

docs/ABOUT_DOCS.rst

@@ -1,21 +1,58 @@
-Meta Docs
-=========
+About The Documentation
+=======================
 
-Our docs are built with sphinx.
+This project documentation is organized to support both:
 
-In order to build the docs locally, you need a sphinx running with the requirements
-in the `docs-requirements.txt` file.
+- human readers (contributors, operators, integrators), and
+- machine-assisted workflows (retrieval and LLM navigation).
 
+Documentation model
+-------------------
 
-We use a mix of RestructuredText and MarkDown in these docs because, well,
-we're difficult I guess.
+Primary structure is Diataxis-aligned:
 
+- ``getting-started`` style content (quickstart, quick tour, training)
+- ``how-to`` task-oriented guidance
+- ``concepts`` explanatory and architecture pages
+- ``reference`` API-oriented material
+- ``operations`` deployment and migration guidance
+- ``contrib`` optional integrations
+
+Machine-readable layer
+----------------------
+
+Generated assets are published at docs root:
+
+- ``llms.txt``
+- ``llms-full.txt``
+- ``robots.txt``
+- ``sitemap.xml``
+
+These files are generated from curated sources using:
+
+- ``docs/scripts/generate_llms_assets.py``
+- ``docs/source/_llm/curation.yml``
+
+Contributor expectations
+------------------------
+
+When adding/updating docs:
+
+1. Keep page titles and section headings stable and descriptive.
+2. Add concise summaries for concept-heavy pages.
+3. Prefer extending current pages before introducing duplicates.
+4. Preserve link compatibility when moving content (stubs/cross-links).
+5. Regenerate LLM/crawler assets when curated pages change.
+
+Quality checks
+--------------
+
+Run locally before opening a PR:
 
 .. code-block:: shell
 
-  python3.7 -m venv .
-  source bin/activate
-  pip install guillotina
-  pip install -r docs-requirements.txt
-  cd docs
-  make html
+   python docs/scripts/generate_llms_assets.py
+   python docs/scripts/check_llms_assets.py
+   python docs/scripts/check_retrieval_coverage.py --top-k 5 --min-coverage 0.90
+   python -m sphinx -b html docs/source docs/build/html
+   python -m sphinx -b linkcheck docs/source docs/build/linkcheck

docs/MAINTENANCE.md

@@ -0,0 +1,40 @@
+# Documentation Maintenance
+
+## Ownership
+- Primary maintainers: Guillotina core maintainers and active docs contributors.
+- Review model:
+  - docs changes should be reviewed alongside code changes that alter behavior.
+  - concept/reference updates should include cross-link checks.
+
+## Cadence
+- Per PR: run docs build and CI docs job.
+- Weekly: review failed linkcheck results and fix/update external references.
+- Monthly: review LLM curation quality and prune noisy/low-value sections.
+- Per release: verify migration docs and changelog links remain correct.
+
+## Recurring checks
+- `llms-full.txt` size remains under configured budget (2 MB).
+- duplicate ratio in `llms-full.txt` stays under CI threshold.
+- retrieval benchmark coverage remains at or above 90% target.
+- no new broken internal links in docs build/link outputs.
+
+## Regeneration workflow
+```bash
+python docs/scripts/generate_llms_assets.py
+python docs/scripts/check_llms_assets.py
+python docs/scripts/check_retrieval_coverage.py --top-k 5 --min-coverage 0.90
+python -m sphinx -b html docs/source docs/build/html
+python -m sphinx -b linkcheck docs/source docs/build/linkcheck
+```
+
+## When structure changes
+- Update `docs/source/index.md` navigation first.
+- Keep compatibility stubs/cross-links for moved high-traffic pages.
+- Update `docs/source/_llm/curation.yml` and regenerate assets.
+- Update benchmark queries if discovery paths changed significantly.
+
+## Known external-link policy
+Some legacy external links may become unavailable over time. Prefer:
+1. replacing with canonical upstream links when possible,
+2. documenting ignored links in `docs/source/conf.py` when unavoidable,
+3. creating follow-up tasks to remove stale references.

docs/plan-artifacts/adr-001-doc-ia.md

@@ -0,0 +1,41 @@
+# ADR-001: Documentation IA Naming and Migration Strategy
+
+- Status: Accepted
+- Date: 2026-02-25
+- Scope: Baseline decisions before IA refactor tasks
+
+## Context
+A baseline audit was required before changing documentation structure to avoid link
+breakage and naming drift.
+
+## Audit findings (stale assumptions checked against real files)
+1. There is no existing `concepts/`, `how-to/`, or `operations/` directory in
+   `docs/source/` yet.
+2. The docs entrypoint currently mixes tutorial/reference/developer/deploy
+   concerns in `docs/source/index.md`.
+3. `docs/Makefile` expects `../bin/sphinx-build`; this path is absent in this
+   workspace unless a local venv is manually created there.
+4. Legacy pages (`quickstart`, `quick-tour`, `developer/*`, `installation/*`)
+   are externally linkable and should remain stable during migration.
+
+## Decisions
+1. Canonical IA naming for this migration:
+   - `getting-started/`
+   - `how-to/`
+   - `concepts/`
+   - `reference/`
+   - `operations/`
+   - `contrib/`
+   - `migration/`
+2. Keep backward compatibility by using stubs/cross-links for moved high-traffic
+   pages.
+3. Prefer incremental content mapping over one-shot large moves.
+4. Treat `contrib/` and `migration/` as stable top-level sections in this phase
+   to reduce migration risk.
+
+## Artifacts
+- Inventory: `docs/plan-artifacts/docs_inventory.csv`
+- Redirect map: `docs/plan-artifacts/redirect_map.csv`
+
+## Gate result
+- No unresolved naming/path conflicts identified for starting Phase 1 and Phase 2.

docs/plan-artifacts/adr-002-llm-discovery-policy.md

@@ -0,0 +1,46 @@
+# ADR-002: LLM Discovery and Crawl Policy
+
+- Status: Accepted
+- Date: 2026-02-25
+- Related plan: `DOCUMENTATION_IMPROVEMENT_PLAN_LLM_READY.md` (section 8 decisions)
+
+## Context
+The documentation improvement plan requires explicit decisions before implementing
+`llms.txt`, `llms-full.txt`, robots directives, and sitemap/canonical metadata.
+
+## Decision 1: Canonical production docs URL base
+- Canonical base URL: `https://guillotina.readthedocs.io/en/latest/`
+- Rationale:
+  - Existing references already point to ReadTheDocs.
+  - The repository has an active `.readthedocs.yml` configuration.
+  - `latest` is the stable, version-agnostic entrypoint for most users and agents.
+
+## Decision 2: Bot access policy by environment
+- Public production docs:
+  - Allow standard crawling for documentation pages.
+  - Keep explicit user-agent blocks available for rapid policy tightening if needed.
+- Non-production/preview/private docs:
+  - Default policy: disallow all crawling (`User-agent: *`, `Disallow: /`).
+- Rationale:
+  - Public docs should remain discoverable.
+  - Non-production/private content should not be indexed or harvested.
+
+## Decision 3: `llms-full.txt` size threshold
+- Initial maximum size: `2 MB`.
+- Rationale:
+  - Keeps context payload bounded for retrieval and prompt usage.
+  - Reduces duplication/noise risk.
+
+## Decision 4: Changelog and migration inclusion in curation
+- `llms.txt`:
+  - Include links to changelog and migration landing pages.
+- `llms-full.txt`:
+  - Exclude changelog and migration body content by default.
+  - Include only if a future policy change explicitly opts in.
+- Rationale:
+  - Preserve discovery of historical docs while keeping long-form corpus focused.
+
+## Operational notes
+- If project maintainers choose a different canonical host/version policy,
+  update this ADR and regenerate LLM assets in the same change set.
+- CI checks should enforce the `2 MB` cap and detect large duplicate ratios.