Conversation
symroe
force-pushed
the
minutes
branch
2 times, most recently
from
August 21, 2026 09:39
0aec81e to
85e73cf
Compare
Storage sessions were scoped per council but not per data type, so a minutes run would delete a council's councillors output and vice versa. Sessions are now rooted at data/<COUNCIL>/<Type>/, matching the layout the GitHub backend already used. Alongside that, BaseStorage takes a storage_mode. REPLACE clears everything stored for the council and data type before writing, which is what councillors needs: the current set is the whole truth and someone who has left should disappear. ACCUMULATE adds to what is there, which is what any historical record needs, where last year's data is still a fact. REPLACE remains the default, so existing scrapers are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scraped documents are large binaries with different needs from the metadata that describes them: a meeting's JSON is a few KB of text that belongs in version control, while the agenda pack behind it can be hundreds of megabytes that belongs in an object store. lgsf/storage/documents/ stores those files, deliberately independent of the metadata backend so the two can be pointed at different places. The default LocalDocumentStorage writes to data/<COUNCIL>/documents/, beside rather than inside the per-type metadata directories, since documents are shared across scraper types and excluded from version control. Each stored document reports its sha256 hash, size, storage key and backend name, which is what keeps metadata and file linked. The resolved URL is deliberately not part of that: for local storage it is an absolute filesystem path, so recording it would put a developer's home directory into metadata destined for git and give every person a different value for the same file. Backends rebuild it from the key via url_for(). Unlike the metadata backends this is not session based. Documents are immutable once written, so exists() is a cheap and reliable way for a scraper to skip work it has already done. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three additions to ScraperBase, all needed by scrapers that fetch documents or revisit the same pages on a schedule. get_conditional() sends If-None-Match and If-Modified-Since so a server can answer 304 instead of resending a body. response_status() and response_header() normalise the differences between the four supported HTTP clients, since wreq returns a StatusCode object and bytes-valued headers where the others return ints and strings. get() was also silently discarding extra_headers under wreq, which made conditional requests impossible; headers are now merged over the emulated browser set rather than dropped. document_storage is a lazily created document backend, so scrapers that store no documents never construct one. storage_mode declares whether this scraper's data replaces or accumulates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CouncilMetadata had one field per service, plus matching branches in from_file(), to_dict() and a SERVICE_NAMES tuple. Adding a service meant four coordinated edits, each with its own failure mode: missing the to_dict() one silently dropped that service's config from every council the next time metadata was saved, and missing SERVICE_NAMES made get_service_metadata() return None so every scraper saw a base_url of None. Services are now a dict keyed by name. A new service works as soon as a council's metadata.json declares it, with no change to the model and no list of permitted names to keep in sync. get_service_metadata() always returns a ServiceData rather than None for an unknown service, so callers can read .base_url without a null check and get None for "not configured" either way. get_summary() reports every configured service rather than singling councillors out: a council has a cms_type and base_url per service, and there is no such thing as "the" one. Verified behaviour-preserving by loading all 435 scrapers/*/metadata.json through both the old and new models and comparing serialised output: identical for every file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
manage.py metadata update was unusable. Two calls named methods that do not exist, so updating any council raised AttributeError before writing anything: update_everyelectiion_data (the real one has one i, matching the method rather than the misspelled JSON key) and get_councillors_service. Separately, --council only searched the first page of results, but the EveryElection API ignores an official_identifier filter and returns all 502 organisations paginated at 100, so every council after the first hundred alphabetically reported "not found" - quietly, since that path prints rather than raises. Three places assumed councillors, which is fine with one scraper type and wrong with more. Validation looked for councillors.py and read services.councillors, so a missing or broken scraper for any other service was invisible; it now takes --service, reports councils that take no part in a service as not applicable rather than failing them, and treats a scraper file without metadata (or the reverse) as an error. Tag filtering loaded councillors scrapers whatever service was requested, so --service minutes --tags filtered on the wrong scrapers entirely. And base_url was only ever auto-filled from councillors.py, so a council could gain a service and never have its metadata completed - every scraper file is now considered, named after its service by convention. The CMS type check used a table of three councillor class names, so scrapers written against any other base class went unchecked. Base classes are named <CMS><Type>Scraper, so it now reads the CMS off the prefix and needs no entry per scraper type. That found four real metadata inaccuracies: three councils label a PagedHTML scraper as "Custom HTML", and ERY labels a JSON scraper the same way. Also derive report["valid"] from whether errors were recorded. The base_url check appended an error without clearing the flag, so a council could be reported as passing while listing errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Running every council was close to unusable. With -v the first broken council raised and ended the run; without it, every failure went into one of several hundred Rich tables scrolling past. Neither leaves you with a list of what is broken, which is the reason to run them all. Councils are independent sites, so they are now scraped concurrently (--workers, default 4) and one failing never stops the others. Each council's outcome is recorded in data/<COUNCIL>/<Type>/runlog.json, beside the data that run produced, keeping the most recent run and the scraper's console output for the ones that failed. --list-failing reads those, so after a full run it reports what is actually broken rather than what a production dashboard says about a different data type. Run logs are written straight to disk rather than through the storage session, because the runs most worth recording are the ones the session never commits: a failure resets it without writing, and a run that scrapes nothing skips the commit entirely. Running exactly one council still raises with -v: that is someone working on that scraper, or a scripted call that needs the failure to reach its exit code. One council is also never run "concurrently", so its output streams as it happens instead of being replayed at the end. Concurrency needed two things to stop being shared. run_council mutated self.options["council"], so parallel runs would overwrite each other's council and scrape the wrong site; each council now gets its own options. And they shared one recording console, which would interleave every council's log into every other council's run record; each now gets its own. Lambda is unaffected: it runs a single council through its own handler, which builds its own console and run log and never enters this loop. Run logs are skipped there, and for any backend that keeps its data elsewhere, since there is nowhere local to put them. Also generalise --report to collect from any scraper type, and show real council names in the listings - they read official_name from the top level of metadata.json, where the current format nests it under everyelectiion_data, so every council showed as "Unknown". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ports the meeting-scraping half of poteris/council-scraper into the framework, mirroring the shape of the councillors vertical: MeetingBase, BaseMinutesScraper, ModernGov and CMIS implementations, and a command. Meetings are scraped over a rolling window that looks mostly backwards, because minutes are published days or weeks after a meeting and revisiting is how they get picked up. Minutes accumulate rather than replace, since a meeting that has dropped out of the window is still a fact. Documents go to the document store, not into raw/ beside the metadata, and a _index.json records where each one went along with its hash and any HTTP validators. Documents already in the store are skipped, so a second run over the same window costs almost nothing - without it a single council re-downloaded 267 files and 104MB every run. The index is treated as a cache rather than a source of truth: every entry is checked against the store before it is used to skip, so a deleted file costs a re-download rather than losing the document. The full agenda pack is kept; categorise_document only labels documents, it doesn't decide which are worth downloading. The CMIS scraper walks a month calendar per month in the window. Its list view renders only the server's current calendar month and ignores any date parameter, so a CMIS council could otherwise never return the same window as ModernGov. The month view is reachable only in DNN's path form, which needs the site's tabid as well as the module id; both are lifted from a real meeting link, and the list view remains as a fallback. --skip-documents records documents without fetching them, which is what you want while iterating on parsing, or when checking several hundred scrapers still work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
manage.py templates was broken for every template, councillors included: it unpacked scraper_abs_path()'s return value as a tuple, but that function returns a Path, so any attempt to scaffold a scraper raised TypeError. Nothing caught it because nothing exercised the command. Adds minutes templates for ModernGov and CMIS, plus a custom starter for councils on neither, where the scraper has to be written by hand and there is no base class to inherit the work from. That template stubs both required methods and shows the shape of a meeting, including how documents are recorded. Tests drive every template end to end: that it writes a file, that the file parses as Python, that it defines a Scraper class, that base_url is substituted, and that an existing scraper is never clobbered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A services.minutes block and a minutes.py for every council whose meetings are on ModernGov or CMIS: 304 ModernGov and 32 CMIS, or 87% of current councils. The remaining 49 are on custom sites that need a hand-written scraper. Generated from the council list in poteris/council-scraper, matched to LGSF scraper directories by hostname - councillors and minutes almost always share a domain - then checked by running every one of them. That check mattered. It found four base_urls left ending in /mgCalendarMonthView.asp, which a ModernGov scraper cannot use since it needs the directory above mgWebService.asmx; nine councils whose minutes host differed from their working councillors host, four of which did not resolve at all; and two councils matched to the wrong organisation entirely, Liverpool to Merseytravel and Rugby to the Welsh Parliament. Rugby has no ModernGov to correct to and is left in the coverage gap rather than configured wrongly. 307 of the 336 scrape cleanly. Most of the rest are councils whose own sites are down, refusing connections or returning 5xx; those are listed in PLAN-minutes.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README said councillors were the only supported type, documented the councillor scraper classes in detail with no equivalent for minutes, and had grown to 445 lines of reference material. Anyone arriving had to read all of it, or the source. It is now an overview and an install guide: how the pieces fit together - a council is a directory, a data type is a package, a council's scraper is usually four lines, configuration lives in metadata rather than code - followed by the commands to get a scraper running, and an index. Everything else moved to docs/, split by what someone is actually doing: running scrapers and diagnosing a broken one, the councillor classes, the minutes classes, minutes coverage, and adding a whole data type. Splitting it by task rather than by chapter means each file can be pointed at directly. Two things in there are worth knowing before touching anything, and neither was written down. Running a single council without -v, the command catches the exception, prints a summary table and exits 0, so a broken scraper looks like a quiet one. And base_url comes from metadata.json rather than the scraper class, so a scraper that only sets it on the class runs against nothing. The fixing guidance leads with configuration rather than parsing, because that is where the failures are: running every configured council, the large majority of breakage is a wrong or stale base_url. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Committed to .claude/skills/ rather than kept personally, so anyone who
clones the repo gets them and they are reviewed like any other file. Only
.claude/settings.local.json is ignored, which is personal to whoever is
running Claude Code.
Four skills, each a short router into docs/ rather than a copy of it, so
the human documentation and the agent instructions cannot drift apart:
lgsf-run commands shared by every data type, the dev
loop, and diagnosing a broken scraper
lgsf-councillors the councillor classes and their contract
lgsf-minutes meetings, the scrape window, document storage
lgsf-add-scraper-type adding a whole data type
The two domain skills carry `paths`, so editing a council's minutes.py
loads the minutes skill and not the councillors one. That is what keeps
them pointed at different documentation without either having to know
about the other.
Each carries inline the handful of things that are expensive to learn by
trial: that a single council needs -v or its exception is swallowed, that
base_url comes from metadata.json rather than the scraper class, that
councillors replace and minutes accumulate, and that a minutes scraper
finding no meetings may be correct rather than broken.
They are prefixed lgsf- because an unprefixed `run` would shadow the
bundled /run skill.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.