Skip to content

Every page advertises llms.txt and llms-full.txt, but they are only generated for projects with an API reference — docs-only sites ship two dead links #350

Closed

Description

@Greg-Hitchon

Summary

The AI / Agents margin section appends links to llms.txt and llms-full.txt unconditionally. _generate_llms_txt and _generate_llms_full_txt both return early when _quarto.yml has no api-reference key.

For any project without an API reference — a pure documentation site — the files are never written and the links 404.

Root cause

core.py, building the margin section. The Skills item is gated; the two below it are not:

if self._config.skill_enabled:
    ...
    ai_items.append(f'<a href="skills.html">Skills{_sparkle_svg}</a><br>')
ai_items.append("[llms.txt](llms.txt)<br>")
ai_items.append("[llms-full.txt](llms-full.txt)<br>")

margin_sections.append(f"\n#### {get_translation('ai_agents', lang)}\n")

core.py, _generate_llms_txt (and identically _generate_llms_full_txt):

quarto_yml = self.project_path / "_quarto.yml"
if not quarto_yml.exists():
    return
...
if "api-reference" not in config:
    return

The generator is called unconditionally during the build and the step is logged as Created llms.txt + llms-full.txt, so the build reports success while having written nothing.

Minimal repro

Any great-docs.yml with sections: and no api-reference in the generated _quarto.yml. Build, then:

$ find . -name 'llms*.txt'
(nothing)

$ grep -c 'llms.txt' _site/index.html
2

Confirmed on our own site: index.html and skills.html each carry both links, and no llms*.txt exists anywhere in the project or the built output.

Impact

Two dead links on the front page of every documentation-only site, in a section specifically aimed at agents — which are the readers least likely to report a 404 and most likely to be confused by one.

It is also quietly misleading: the build log says the files were created.

Suggested fix

Gate the links on the same condition the generators use — or, better, have the generators produce a section index for non-API projects rather than returning. The sections: structure already carries titles and page lists, which is most of what llms.txt wants.

If gating is the preferred route, the minimal version is to have _generate_llms_txt report whether it wrote anything and append the links only when it did, so the two cannot drift apart again.

Workaround

We generated the two files ourselves for several days, then deleted that and now strip the two links from the built HTML instead — our site is not published, so an index for crawlers had no reader either way.

Version

great-docs 0.17.0 (latest on PyPI at time of writing), Python 3.13, Windows.

Activity

  1. reopened this on Sep 17, 2026
  2. tyza66 commented on Oct 22, 2026

    @tyza66
    Contributor

    Hi, thanks for the clear report. I picked this up and opened #410.

    The root cause turned out to be one step past #351. That PR gated the places
    that advertise the files on _llms_txt_available(), but the code deciding
    whether the files are actually written still had three separate opinions about
    when to run:

    • _generate_llms_full_txt() only required an api-reference block with a
      package, so an empty sections list still produced a header-only
      llms-full.txt.
    • _prepare_for_freeze() called both generators directly, bypassing the
      predicate build() gates on, so the freeze path could leave files behind
      that a full build never writes.

    The fix introduces _llms_txt_quarto_config() as the single source of truth for
    the structural half of the guard, has the predicate and both generators share
    it, and makes _prepare_for_freeze() honour the same predicate as build().
    Tests are parametrized over six _quarto.yml shapes and assert the generators
    and the freeze path agree with the predicate in every case.

    Disclosure: this change was produced with the help of an AI coding agent, which
    used Alpha as its tool.
    Everything in the PR was reviewed by me and verified against the test suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions