Add indexers for common documentation portals

[yarikoptic]

The overall idea is to get assistant specifications more declarative and just to point to corresponding websites.
Indexers (introduced here) should be able to figure out per URL an index of pages to point to. The inspiration was the https://github.com/yarikoptic/qp/blob/main/src/assistants/figpack-assistant/retrieveFigpackDocs.tsx#L71 and the

❯ curl --silent https://flatironinstitute.github.io/figpack/doc-pages.json | head
{
  "docPages": [
    {
      "title": "API Reference",
      "url": "https://flatironinstitute.github.io/figpack/api/index.html",
      "sourceUrl": "https://flatironinstitute.github.io/figpack/_sources/api/index.md.txt",
      "includeFromStart": true
    },
    {
      "title": "Developer Guide",
...

so with this script (claude vibe coded) it is already possible to get such listing for various doc engines (with -o would be just to file without occluding logging)

• mkdocs

❯ ./doc_pages_index.py https://bids-specification.readthedocs.io/en/stable/ --format figpack | head
Detecting documentation type for https://bids-specification.readthedocs.io/en/stable/
✓ Detected: MkDocs documentation
Fetching MkDocs search index from: https://bids-specification.readthedocs.io/en/stable/search/search_index.json
Found 43 unique pages (filtered from 1458 entries)
Attempting to detect source repository from 'Edit this page' links...
✓ Detected source repository: https://github.com/bids-standard/bids-specification/blob/master/src
{
  "docPages": [
    {
      "title": "Changelog",
      "url": "https://bids-specification.readthedocs.io/en/stable/CHANGES.html",
      "sourceUrl": "https://github.com/bids-standard/bids-specification/blob/master/src/CHANGES.md",
      "includeFromStart": true
    },
    {
      "title": "Arterial Spin Labeling",

• sphinx

❯ ./doc_pages_index.py https://docs.datalad.org/en/stable/ --format figpack | head
Detecting documentation type for https://docs.datalad.org/en/stable/
✓ Detected: Sphinx documentation
Fetching Sphinx search index from: https://docs.datalad.org/en/stable/searchindex.js
Found 143 documents
Checking for _sources/ directory...
✓ Found _sources/ directory (will use for source URLs)
{
  "docPages": [
    {
      "title": "Acknowledgments",
      "url": "https://docs.datalad.org/en/stable/acknowledgements.html",
      "sourceUrl": "https://docs.datalad.org/en/stable/_sources/acknowledgements.rst.txt",
      "includeFromStart": true
    },
    {
      "title": "Background and motivation",

• some other sphinx -- fails to determine sources on https://www.hedtags.org/hed-resources/ (see https://github.com/magland/qp/pull/18/files#r2525286258):

❯ ./doc_pages_index.py https://www.hedtags.org/hed-resources/ | xclip -i
Detecting documentation type for https://www.hedtags.org/hed-resources/
✓ Detected: Sphinx documentation
Fetching Sphinx search index from: https://www.hedtags.org/hed-resources/searchindex.js
Found 28 documents
Checking for _sources/ directory...
✗ No _sources/ directory found
  Consider providing --source-repo for source file URLs

so if not discoverable -- should indeed provide source-repo ... TODO: mention that to copilot if I review that PR it made ;-)

• hugo ???

So we are just to decide on how we want to wrap it up. Ideally, I would like to see most of the assistants mostly to be a .yaml file definition containing the prompts and documentation sources. But I have not yet reviewed/understood what each one has custom. @magland could you guide me?

establishing then some CI to dump/update them daily should be trivial. We could also add

• not bothering to update if didn't change

[yarikoptic]

NB submitted as PR from a branch here since my personal fork got "disconnected" from this repo, I guess due to its short trip to the "dark side"

[yarikoptic]

@copilot review assistants codebase for which documentation sites it uses across assistants, create

• some centralized registry (yaml file?) listing all those resources,
• cook up github CI to run (on PR/branch changes to that registry or the code here; and daily on cron) which would create or update per each website dump in the --format figpack. Filenames should be constructed based on the TLD of the websites, e.g. be docs.datalad.org_figpack.json for https://docs.datalad.org/en/stable/ and so on.

[Copilot]

@yarikoptic I've opened a new pull request, #17, to work on those changes. Once the pull request is ready, I'll request review from you.

[magland]

@yarikoptic I've opened a new pull request, #17, to work on those changes. Once the pull request is ready, I'll request review from you.

Sounds great!