WIP: Minutes

.claude/skills/lgsf-add-scraper-type/SKILL.md

@@ -0,0 +1,46 @@
+---
+name: lgsf-add-scraper-type
+description: Add a whole new data type to LGSF - the thing councillors and minutes each are - rather than a scraper for one council. Covers the framework seams a new type has to touch.
+when_to_use: Adding a new kind of scraped data such as decisions or committees. Not for adding a scraper for a single council.
+---
+
+# Adding a scraper type
+
+A data type is a package under `lgsf/` with a model, a base scraper, CMS
+subclasses and a command — `lgsf/councillors/` and `lgsf/minutes/` are the two
+that exist.
+
+**Read `docs/adding-a-scraper-type.md` before starting.** It lists every file
+and framework seam involved.
+
+## Decide these first
+
+They are hard to change later:
+
+- **Is the latest scrape the whole truth, or a historical record?** Sets
+  `storage_mode` to `StorageMode.REPLACE` or `.ACCUMULATE`. Getting it wrong
+  on a historical type silently deletes anything outside the scrape window.
+- **Does it download files?** Those go to the document store, not the metadata
+  store.
+- **What is the natural scrape window?**
+- **What identifies one record?** It must be stable across runs — a source
+  system id, not a row position.
+
+## What a new type touches
+
+```
+lgsf/<type>/          model, scrapers, command, exceptions, tests
+lgsf/conf/__init__.py add "<type>" to settings.APPS
+lgsf/templates/       scaffolding templates
+lgsf/tests/test_runner.py  asserts the full app list
+scrapers/<CODE>/      per-council <type>.py and metadata block
+```
+
+`lgsf/metadata/models.py` needs **no** change — services are keyed by name, so
+a type works as soon as a council's `metadata.json` declares it. Do not add a
+per-service field there.
+
+## Related
+
+- `docs/adding-a-scraper-type.md` — the full procedure
+- `.claude/skills/lgsf-minutes` — a worked example of a type

.claude/skills/lgsf-councillors/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: lgsf-councillors
+description: Write and fix LGSF councillor scrapers - the councillor scraper classes, what a Councillor object needs, and how to skip rows that are not councillors.
+when_to_use: Working on a council's councillors.py, on lgsf/councillors/, or on councillor data generally.
+paths: scrapers/*/councillors.py, lgsf/councillors/**
+---
+
+# Councillor scrapers
+
+Councillor scrapers produce a set of `CouncillorBase` objects, one per
+councillor. Full reference in `docs/councillor-scrapers.md`.
+
+For running them, the dev loop and diagnosing failures, see the `lgsf-run`
+skill.
+
+## Choosing a class
+
+| Council's site | Class | Usually needs |
+| --- | --- | --- |
+| ModernGov | `ModGovCouncillorScraper` | nothing but a `base_url` in metadata |
+| CMIS | `CMISCouncillorScraper` | nothing but a `base_url` in metadata |
+| Anything else, one page | `HTMLCouncillorScraper` | CSS selectors and `get_single_councillor` |
+| Anything else, paginated | `PagedHTMLCouncillorScraper` | as above plus a next-page selector |
+| A JSON API | `JSONCouncillorScraper` | a `get_single_councillor` over the parsed feed |
+
+Scaffold one rather than writing it from scratch:
+
+```bash
+uv run python manage.py templates --council ABC \
+    --template councillor_scraper_html \
+    --context base_url https://example.gov.uk/councillors
+```
+
+## The contract
+
+`get_single_councillor` must return a councillor built with
+`self.add_councillor()`, which requires `url`, `identifier`, `name`, `party`
+and `division`. All five are asserted, so a missing one fails loudly.
+
+`identifier` must be stable across runs — an id from the source system, not a
+row number. It becomes the filename, so an unstable one silently creates
+duplicates.
+
+Where a row isn't a councillor — an inline header row, or a neighbouring
+council's members on a shared site — raise `SkipCouncillorException` and the
+loop moves on.
+
+## When a council moves CMS
+
+A name resolution failure, or a 404 on a `base_url` that used to work, usually
+means the council has moved — often onto ModernGov. Search for
+"<council name> councillors", find the live page, then read the CMS off the
+URL:
+
+| URL contains | CMS | Switch the class to |
+| --- | --- | --- |
+| `mgMemberIndex.aspx`, or any `mg*.aspx` | ModernGov | `ModGovCouncillorScraper` |
+| `cmis.uk.com`, `/cmis5/`, or a `cmis.` host | CMIS | `CMISCouncillorScraper` |
+| neither | bespoke | `HTMLCouncillorScraper`, plus selectors |
+
+Moving to ModernGov or CMIS usually shrinks the scraper body to `pass` — both
+work from `base_url` alone.
+
+Put the new URL in the council's `metadata.json` under
+`services.councillors.base_url`, **not** on the class. A `base_url` set on the
+class is overwritten from metadata at runtime, so a scraper that only sets it
+there runs against nothing.
+
+## Storage
+
+Councillors use `StorageMode.REPLACE`: each run clears what came before,
+because the current set is the whole truth and someone who has left should
+disappear. Do not change this to accumulate.
+
+## Related
+
+- `docs/councillor-scrapers.md` — classes, the councillor object, skipping
+- `docs/running-scrapers.md` — running and diagnosing

.claude/skills/lgsf-minutes/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: lgsf-minutes
+description: Write and fix LGSF minutes scrapers - committee meetings and the documents attached to them, the rolling scrape window, and how documents are stored separately from metadata.
+when_to_use: Working on a council's minutes.py, on lgsf/minutes/, on meeting or document scraping, or picking up a council that needs a minutes scraper.
+paths: scrapers/*/minutes.py, lgsf/minutes/**
+---
+
+# Minutes scrapers
+
+Minutes scrapers produce a set of `MeetingBase` objects, each with the
+documents linked from that meeting. Full reference in
+`docs/minutes-scrapers.md`.
+
+For running them and diagnosing failures, see the `lgsf-run` skill.
+
+## Always use --skip-documents while iterating
+
+```bash
+uv run python manage.py minutes --council KIR -v --skip-documents
+```
+
+Documents are found and recorded but not downloaded. A single council can be
+hundreds of megabytes otherwise.
+
+## Choosing a class
+
+| Council's site | Class | Notes |
+| --- | --- | --- |
+| ModernGov | `ModGovMinutesScraper` | `base_url` must be the directory containing `mgWebService.asmx` |
+| CMIS | `CMISMinutesScraper` | `base_url` is the meetings page; the name varies per install |
+| Anything else | `CustomHTMLMinutesScraper` | implement `get_meetings` and `get_single_meeting` |
+
+Scaffold with `--template minutes_scraper_modgov`, `minutes_scraper_cmis` or
+`minutes_scraper_custom`.
+
+## The scrape window
+
+Meetings are scraped over a rolling window set by `weeks_back` (4) and
+`weeks_forward` (1). It looks mostly backwards on purpose: minutes are
+published days or weeks after a meeting, so revisiting is how they get picked
+up once they appear.
+
+This means **a scraper finding zero meetings is not necessarily broken.** Over
+a quiet period, or a recess, there may genuinely be none. Check by asking the
+council's API or calendar directly for the same window before concluding
+anything.
+
+## Documents
+
+Metadata and files are stored separately:
+
+- The meeting JSON goes to `data/<COUNCIL>/Minutes/json/` and is meant for
+  version control. Each document it links records a `content_hash`, a
+  `storage_key` and the backend holding it — but **never a resolved URL**,
+  which would be specific to whoever ran the scrape.
+- The files go to `data/<COUNCIL>/documents/` through a pluggable document
+  backend.
+- A document already in the store is not downloaded again, tracked through
+  `_index.json`. That index is a cache, not a source of truth: entries are
+  checked against the store before being used to skip.
+
+**Store documents before saving the record's JSON.** The JSON is serialised
+from the document dicts, and storing a document is what adds the hash and
+storage key to them. The other order records documents pointing at nothing.
+
+## Storage
+
+Minutes use `StorageMode.ACCUMULATE`: a meeting that has dropped out of the
+window is still a fact, so runs add to what is stored rather than replacing
+it. Do not change this to replace.
+
+## Related
+
+- `docs/minutes-scrapers.md` — classes, window, skipping, documents
+- `docs/running-scrapers.md` — running and diagnosing

.claude/skills/lgsf-run/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: lgsf-run
+description: Run LGSF scrapers, find councils that need work, and diagnose one that is failing. Covers manage.py commands shared by every data type - councillors, minutes and any added later.
+when_to_use: Running a scraper, working out why one is broken, finding councils with no scraper or a failing one, or checking metadata against scrapers.
+---
+
+# Running LGSF scrapers
+
+LGSF scrapes UK council websites into open data. Each council has a package in
+`scrapers/CODE-slug/` holding a `metadata.json` and one Python file per data
+type. Every data type is a `manage.py` subcommand taking the same options.
+
+Read `docs/running-scrapers.md` for the full reference.
+
+## Two things to know first
+
+**Run a single council with `-v`.** Without it the command catches the
+exception, prints a summary table and exits 0, so a broken scraper looks like
+a quiet one. Across many councils `-v` will *not* stop the run.
+
+**`base_url` comes from `metadata.json`, not the scraper class.** A scraper
+that sets `base_url` on the class and nowhere else runs against nothing. Look
+at `services.<type>.base_url` in the council's `metadata.json`.
+
+## Commands
+
+```bash
+# One council, watching what it does
+uv run python manage.py <type> --council KIR -v
+
+# Everything. Councils run concurrently; one failing never stops the rest
+uv run python manage.py <type> --all-councils
+
+# What has no scraper, what failed last time it ran here
+uv run python manage.py <type> --list-missing
+uv run python manage.py <type> --list-failing
+
+# Metadata and scraper files agree?
+uv run python manage.py metadata validate --service <type>
+uv run python manage.py metadata list-cms --service <type> --csv
+```
+
+Useful options: `--workers N` (how many scrapers at once, default 4; use 1 to
+watch one), `--report` (table of what was scraped), `--tags`,
+`--skip-documents` (minutes only).
+
+## Where things land
+
+```
+data/<COUNCIL>/<Type>/json/      normalised records
+data/<COUNCIL>/<Type>/raw/       raw responses
+data/<COUNCIL>/<Type>/runlog.json  how the last run went, locally only
+data/<COUNCIL>/documents/        downloaded files
+```
+
+`--list-failing` reads those run logs first, so it mostly knows about councils
+run on this machine. Where there are no local logs it falls back to the
+production report, which only councillors has — so for other types an empty
+result means "nothing has been run here", not "nothing is failing".
+
+## Diagnosing a broken scraper
+
+Most breakage is configuration, not parsing. In order:
+
+1. Run it with `-v` and read the actual exception.
+2. Does the host resolve, and serve what you expect in a browser?
+3. Does the council's **councillors** `base_url` work where this one doesn't?
+   They are usually the same host; if they differ the councillors one is more
+   likely current.
+4. For ModernGov, `{base_url}/mgWebService.asmx/GetCommittees` should return
+   XML. A 404 usually means `base_url` includes a page name that needs
+   stripping — it must be the directory *above* `mgWebService.asmx`.
+5. For CMIS, the meetings page name varies between installs: `Meetings.aspx`,
+   `MeetingsCalendar.aspx`, `CalendarofMeetings.aspx`.
+
+If the site is down or blocking, say so rather than working around it.
+
+### What the failure usually turns out to be
+
+| Symptom | Cause | Fix |
+| --- | --- | --- |
+| Timeout on an `http://` URL | Site is HTTPS-only now | `base_url` to `https://` in `metadata.json` |
+| SSL / certificate error | Cert chain the HTTP client won't trust | `verify_requests = False` on the scraper class |
+| Name resolution failure | Council moved CMS or domain | Find the new site — see the councillors skill |
+| 404 on a URL that used to work | URL structure changed | New `base_url` in `metadata.json` |
+| 403 Forbidden | Cloudflare or another WAF | `http_lib = "playwright"`, or `use_proxy = True` |
+| `AttributeError: 'NoneType' has no attribute …` | Page structure changed | Reselect against the live page; check for a JSON endpoint first |
+| 409 Conflict from ModernGov | Server-side fault | Usually not fixable from here |
+
+## Working through a batch of failing scrapers
+
+- **One council at a time**, committed on its own as
+  `Fix ABC (Council Name): what changed`.
+- **Only touch that council's files** under `scrapers/CODE-slug/`. If a fix
+  seems to need a change in `lgsf/`, that is a framework change — raise it
+  separately rather than folding it into a scraper fix.
+- **Fixing is not writing.** A council with no scraper is a different job;
+  `--list-missing` finds those.
+- **Time-box each one.** Some breakage needs a human — anti-bot protection
+  and server-side faults especially. Record what you found and move on.
+
+## Checks before proposing a change
+
+```bash
+uv run ruff check lgsf/ && uv run ruff format --check lgsf/
+uv run pytest lgsf/ -q
+```
+
+Tests never hit the network. Fixtures are real saved responses in
+`lgsf/<type>/tests/fixtures/`.
+
+## Related
+
+- `docs/running-scrapers.md` — the full reference
+- `.claude/skills/lgsf-councillors` — councillor scraper specifics
+- `.claude/skills/lgsf-minutes` — minutes scraper specifics

.gitignore

@@ -118,3 +118,6 @@ venv.bak/
 .mypy_cache/
 .dmypy.json
 dmypy.json
+
+# Personal Claude Code settings. Skills in .claude/skills/ are shared.
+.claude/settings.local.json