Add build script for llms.txt and markdown docs

[Azaya89]

Fixes #1730

Description

This PR adds a build script for generating llms.txt and the relevant markdown docs.

• Reworked https://github.com/panel-extensions/panel-material-ui/blob/main/scripts/build_llms_txt.py to generate hvPlot-focused markdown/llms.txt from thedoc/ directory:
• excludes doc/user_guide/ since it will be removed in the future
• copy existing .md files and convert in-scope notebooks to markdown
• Updated pixi.toml docs tasks
• Also updated .github/workflows/docs.yaml build step

Unrelated

• Removed Streaming from doc/user_guide/index.md to clear the doc build warnings.

AI Disclosure

Tool & Model: GPT-5.3 Codex
Usage:

• I have tested all AI-generated content in my PR.
• I take responsibility for all AI-generated content in my PR.

Screenshot of generated llm.txt page

[maximlt]

How long does the _docs-markdown step take? Are we going to have to add this script to every repo or are we planning to upstream that to nbsite or some other place?

[Azaya89]

How long does the _docs-markdown step take?

It's actually surprisingly fast. Didn't extend the build time significantly in any noticeable way.

Are we going to have to add this script to every repo or are we planning to upstream that to nbsite or some other place?

We may have to add it to every repo because each repo will have different sets of docs it needs to convert to markdown.

[ahuang11]

One thing to keep in mind is token efficiency, i.e. try to cull useless info. For example, hvplot.hvPlot.area is a bit excess in my opinion, and can be simplified to area under API Manual. Also each section ideally would have a caption, like in https://panel-material-ui.holoviz.org/llms.txt:

## Component Reference

Detailed reference documentation for every Panel Material UI component, organized by category.```

[Azaya89]

I've updated the script.

[ahuang11]

Can you show the new output for some?

[Azaya89]

Can you show the new output for some?

/markdown/ref/api/manual/hvplot.hvPlot.area.md

Parts of /markdown/ref/plotting_options/legend.md

[Azaya89]

I've updated the script to convert all the docs (except the user guide section). PR description also updated.

[maximlt]

I have left a couple of questions.

Another one is that I checked how the markdown is made available on panel-material-ui. I went to https://panel-material-ui.holoviz.org/how_to/customize_palette.html and simply replaced .html with .md. It didn't work. Looking into the code I understand the URL needs to start with /markdown and indeed the file is available at https://panel-material-ui.holoviz.org/markdown/how_to/customize_palette.md. Do you know why the /markdown approach was chosen?

Could you please share a zip of the markdown folder built by this script? That would help review the changes.

We use a sphinx extension to set up redirects (e.g. when we remove/move a page, we redirect to a new one). I guess that's not supported here, but that it's also not so bad as LLMs can probably figure out where to find the new docs?

Is it planned to let people download directly the markdown file of a page, with a link/button from the page? It doesn't have to be done in this PR, I'm just asking if it's planned.

Will hvPlot's documentation be updated to let users know how they can use these markdown files and the holoviz skills? Again not expected to be done in this PR, I just want to know what is planned or not.

[maximlt]

Isn't that making the docs build much slower?

[maximlt]

Why do we need to set -d .sphinx-doctrees?

[Azaya89]

This was due to Sphinx parallel build issues. I first encountered the problem here: https://github.com/holoviz/holoviz/actions/runs/27723798712/job/82015403537

It's not much slower IMO.

[maximlt]

Ok but this is another repository. Does hvPlot suffer from the same error?

How much slower is it without being parallel? On which machine did it run?

[Azaya89]

Ok but this is another repository. Does hvPlot suffer from the same error?

Yes. I think the problem is upstream from Sphinx

How much slower is it without being parallel? On which machine did it run?

I didn't do a comparison tbh but it ran on my mac pretty fast. These are basically just text files so it wasn't slow.

[maximlt]

I didn't do a comparison tbh but it ran on my mac pretty fast. These are basically just text files so it wasn't slow.

The Sphinx build runs all the notebooks, it is not just processing text files.

Yes. I think the problem is upstream from Sphinx

Ok so let's check whether the problem is still present upstream before disabling anything.

[Azaya89]

Another one is that I checked how the markdown is made available on panel-material-ui. I went to panel-material-ui.holoviz.org/how_to/customize_palette.html and simply replaced .html with .md. It didn't work. Looking into the code I understand the URL needs to start with /markdown and indeed the file is available at panel-material-ui.holoviz.org/markdown/how_to/customize_palette.md. Do you know why the /markdown approach was chosen?

I'm not sure why but I imagine it was to keep the markdown docs separate from the html docs and to also make the link structures stable and easily findable.

Could you please share a zip of the markdown folder built by this script? That would help review the changes.

markdown.zip

We use a sphinx extension to set up redirects (e.g. when we remove/move a page, we redirect to a new one). I guess that's not supported here, but that it's also not so bad as LLMs can probably figure out where to find the new docs?

We didn't really move a page here. We just made new ones. I'm not sure I understand the question here...

Is it planned to let people download directly the markdown file of a page, with a link/button from the page? It doesn't have to be done in this PR, I'm just asking if it's planned.

It wasn't originally planned but it sounds like a good idea we can probably do as well, maybe in a new PR.

Will hvPlot's documentation be updated to let users know how they can use these markdown files and the holoviz skills? Again not expected to be done in this PR, I just want to know what is planned or not.

Yes, this is planned.

[maximlt]

I'm not sure why but I imagine it was to keep the markdown docs separate from the html docs and to also make the link structures stable and easily findable.

It's easily findable if /markdown is a standard/convention. Is it one? I'm fine with it, I just want to understand.

We didn't really move a page here. We just made new ones. I'm not sure I understand the question here...

This is about the future. Imagine this scenario:

• We publish the site with the markdown artifacts, so an HTML page X.html has a Markdown file X.md
• We rename the page from X to Y, and set up a redirect X->Y
• Users visiting X.html get automatically redirected to Y.html; Users visiting X.md don't get automatically redirected to Y.md

I'm fine with that too but want to make sure we understand the consequences of what we do.

It wasn't originally planned but it sounds like a good idea we can probably do as well, maybe in a new PR.

For sure in another PR, and very likely in nbsite (I don't want us to start copy/pasting code from a repo to another, that's a maintenance nightmare).

Yes, this is planned.

Cool!!

[Azaya89]

It's easily findable if /markdown is a standard/convention. Is it one? I'm fine with it, I just want to understand.

I don't think it's the most popular convention but I prefer this one tbh.

This is about the future. Imagine this scenario:

• We publish the site with the markdown artifacts, so an HTML page X.html has a Markdown file X.md
• We rename the page Y, and set up a redirect X->Y
• Users visiting X.html get automatically redirected to X.md; Users visiting X.md don't get automatically redirected to Y.md

I'm fine with that too but want to make sure we understand the consequences of what we do.

OK, in that case, it's not a problem because the via the pixi config, the markdown docs are rebuilt on every docs build. So if a file name is changed, the markdown file name will also be changed automatically. An LLM only needs to know the current file name and then it easily finds the corresponding .md equivalent.

[ahuang11]

I do like the idea of just replacing .html with .md without having to completely change the directory. I wonder if we can set up redirects using https://sphinxext-rediraffe.readthedocs.io/en/latest/

[maximlt]

I don't think it's the most popular convention but I prefer this one tbh.

What I am asking is if it is a convention at all or something we came up with?

OK, in that case, it's not a problem because the via the pixi config, the markdown docs are rebuilt on every docs build. So if a file name is changed, the markdown file name will also be changed automatically. An LLM only needs to know the current file name and then it easily finds the corresponding .md equivalent.

Let's ignore redirects for now (which are not supported but I think it'd not too bad as I care that we set up redirects for humans, less so for bots). Can you explain how an agent is supposed to find out about these markdown pages? Does it need to read the llms.txt file? If so, do agents read the llms.txt file automatically? Are the holoviz-skills involved in any way in all of that?

I haven't followed the work you all have done in this area so sorry for all the questions but I'd like to understand what we are setting up.

I wonder if we can set up redirects using https://sphinxext-rediraffe.readthedocs.io/en/latest/

Not sure that would work as the extension might check for the existence of the new file, and we do the html->md conversion in a post-build step and not part of the Sphinx machinery.

[Azaya89]

What I am asking is if it is a convention at all or something we came up with?

We didn't come up with it. For the record, I still prefer this one as it is even easier to bundle it all in one place and export or do other things with it.

Can you explain how an agent is supposed to find out about these markdown pages? Does it need to read the llms.txt file? If so, do agents read the llms.txt file automatically? Are the holoviz-skills involved in any way in all of that?

Ideally the agent reads the llms.txt files automatically which directs them to the relevant markdown files. However, the plan is also to add directives in the skills files akin to what is already done with PMUi below:

[maximlt]

We didn't come up with it. For the record, I still prefer this one as it is even easier to bundle it all in one place and export or do other things with it.

Ok good to know. Which other sites/projects use this convention?

Ideally the agent reads the llms.txt files automatically which directs them to the relevant markdown files. However, the plan is also to add directives in the skills files akin to what is already done with PMUi below:

Makes sense.

[ahuang11]

Ok good to know. Which other sites/projects use this convention?

:) pmui. This might be a question for @philippjfr

I'd like to have this concluded sooner than later and we can apply it across all our repos, HoloViews in particular.