Correct Title-boundary schedule, disconnect from nightly run, make EMR timeout configurable.

[tombrooks248]

Background

The title-boundary pyspark job hit its 3 hour timeout on the 2026-09-07 production run. Because build-digital-land-builder fans in from every collection trigger task in dags/dag_triggers.py, this also blocked the nightly builder and the tiles builder behind a job that only runs once a month.

The timeout was not a passive expiry: execution_timeout raised AirflowTaskTimeout, Airflow called on_kill() at taskinstance.py:461-463, and EmrServerlessStartJobOperator.on_kill() called cancel_job_run(). The Spark driver was terminated and the Postgres transaction rolled back, so entity was left holding the previous month's data.

There is no real dependency holding title-boundary in the nightly chain. Every collection DAG is created with schedule=None (dags/new_collection_generator.py:74), so the master trigger DAG is the only thing that can start one — title-boundary is in the nightly run because that is the only mechanism available, not because anything downstream needs it. The schedule_rrule plus ShortCircuitOperator machinery at dags/dag_triggers.py:86-97 exists to work around this, by making a monthly job pretend to be a daily one and skip on 30 days out of 31. While investigating that machinery we also found that the recurrence rule itself is wrong.

Part 1 — fix the schedule rule

dags/collection_config.py:52 sets schedule_rrule="FREQ=MONTHLY;BYDAY=1MO" — an RFC 5545 recurrence rule meaning "the first Monday of the month". The comment above it records the intent correctly: HM Land Registry publish the INSPIRE Index Polygons on the first Sunday, and the Monday was chosen to give a day of buffer so their data is available by the time we run.

"First Monday" is not the same thing as "the Monday after the first Sunday". When the 1st of the month falls on a Monday, the first Monday precedes the first Sunday by six days, so we collect the previous month's data and do not run again until the following month. It fails silently, because the pipeline succeeds identically whether or not there is new data.

Verified with dateutil over 30 months from 2026-01, this occurs five times, including twice consecutively in 2027:

month	we run (1st Mon)	HMLR release (1st Sun)
2026-06	Mon 01 Jun	Sun 07 Jun
2027-02	Mon 01 Feb	Sun 07 Feb
2027-03	Mon 01 Mar	Sun 07 Mar
2027-11	Mon 01 Nov	Sun 07 Nov
2028-05	Mon 01 May	Sun 07 May

The 2026-06 occurrence has already happened: we ran on 1 June, HMLR published on 7 June, and that release was not collected until 6 July, so published title boundaries were a month stale for five weeks.

The fix is to pin the run to the Monday falling between the 2nd and the 8th, which is always exactly first-Sunday plus one day. Verified over 30 months with no mismatches; the five months above all move to the 8th.

schedule_rrule="FREQ=MONTHLY;BYDAY=MO;BYMONTHDAY=2,3,4,5,6,7,8"

Note this keeps the deliberate day of buffer. Moving the run to late on the Sunday itself would close the gap but spend that buffer, and HMLR publish on a day rather than at a guaranteed time.

Part 2 — give title-boundary its own monthly trigger, and remove it from the builder fan-in

Add a small dedicated trigger DAG in dags/dag_triggers.py, alongside the two already there, and drop title-boundary from the collection_tasks loop in trigger-collection-dags-scheduled so it no longer gates build-digital-land-builder.

with DAG(
    dag_id="trigger-title-boundary-monthly",
    description="Triggers the title-boundary collection on its own monthly cadence, independently of the nightly chain",
    schedule="0 1 * * 1",  # every Monday; the rrule check below selects the right one
    start_date=datetime(2024, 1, 1),
    catchup=False,
    is_paused_upon_creation=False,
):
    if collection_selected("title-boundary", config):
        check_schedule = ShortCircuitOperator(
            task_id="check-title-boundary-schedule",
            python_callable=collection_schedule_matches,
            op_kwargs={"collection": "title-boundary"},
        )
        check_schedule >> TriggerDagRunOperator(
            task_id="trigger-title-boundary-collection-dag",
            trigger_dag_id="new-title-boundary-collection",
            wait_for_completion=False,
        )

The rrule guard from Part 1 is still needed here, because cron cannot express "nth weekday of the month". Scheduling the trigger weekly rather than daily reduces the skip rate from 30-in-31 to 3-in-4, and collection_schedule_matches picks the correct Monday.

Do not instead put a schedule on the generated collection DAG. dags/new_collection_generator.py:45-46 creates new-title-boundary-collection based on specification dataset-availability, not on config.json's collection_selection. The DAG object already exists in dev and staging and simply never fires, because the master DAG's collection_selected() check is what gates it. Giving it a schedule would start it firing monthly in every environment, unpaused, where it would fail every time on the known /tmp/workspace permission error in dev.

Why the builder is safe without it

• title-boundary is already excluded from the dataset-package task (dags/new_collection_generator.py:291), so it never produces a sqlite for the builder to ingest.
• What the builder's downstream jobs do read is the Delta tables in {env}-parquet-datasets, and those are written roughly 11 minutes into a 3 hour job (01:13:53 against a 01:02 start on 2026-09-07). Everything after that publishes consumer formats and Postgres, which the builder does not read.
• write_delta uses an atomic overwrite with replaceWhere, so Delta snapshot isolation means an overlapping builder run sees the previous complete version, never a partial one. For a monthly release, a builder run picking up last month's title-boundary data is correct behaviour rather than degradation.

What it gives up, deliberately

Standalone, title-boundary loses its ordering behind organisation-builder (run_org_builder_dag >> entry_task), so the entity pipeline will read whatever organisation.csv is in S3 rather than a guaranteed-fresh one — up to about 24 hours stale. Organisation data changes very little month to month, so this is acceptable, but it should be a recorded decision rather than something that quietly stops happening.

Part 3 — make the EMR job timeout configurable per collection

Add a timeout field to CollectionDagConfig in dags/collection_config.py, which already has the right shape for per-collection overrides, and use it for the assemble-emr-job task in dags/new_collection_generator.py. Default stays at the current 3 hours; title-boundary gets a longer value (6 hours is a reasonable starting point, given the 2026-08-25 run took 2h32m and the 2026-09-07 run was still going at 3h). Once nothing waits on the job, a longer timeout costs nothing.

The important trap: there are two separate 3 hour bounds on that task at dags/new_collection_generator.py:283-284, and they do not behave the same way.

waiter_max_attempts=180,           # x waiter_delay=60 = 3h; raises a PLAIN exception, no on_kill,
waiter_delay=60,                   # so the EMR job would be LEFT RUNNING
execution_timeout=timedelta(hours=3),  # raises AirflowTaskTimeout -> on_kill() -> cancel_job_run()

Only execution_timeout cancels the EMR job. It currently wins the race by a few seconds purely because its clock starts at task start while the waiter's starts after start_job_run returns — which is accidentally the behaviour we want. Both bounds must be derived from the same config value, with execution_timeout no greater than the waiter budget, or a future overrun will orphan a job that keeps consuming the shared EMR Serverless application.

Acceptance criteria

• schedule_rrule for title-boundary selects the Monday between the 2nd and the 8th, and a unit test asserts it is exactly one day after the first Sunday for each of the five months listed above.
• trigger-title-boundary-monthly exists, is gated by collection_selected(), and triggers new-title-boundary-collection with wait_for_completion=False.
• title-boundary no longer appears in collection_tasks in trigger-collection-dags-scheduled, so build-digital-land-builder does not wait on it.
• new-title-boundary-collection still has schedule=None and does not fire in dev or staging.
• EMR job timeout is set from CollectionDagConfig, with the existing 3 hours as the default and a longer value for title-boundary.
• Both waiter_max_attempts * waiter_delay and execution_timeout are derived from that single config value, with execution_timeout no greater than the waiter budget, so an overrun always cancels the EMR job rather than orphaning it.
• Verified in dev that the nightly chain reaches build-digital-land-builder without title-boundary in it.

Out of scope

Reducing the Postgres insert time, whether by removing unused indexes on entity or by parallelising the insert. Both are blocked on establishing where the time in the 2026-09-07 run actually went, since the log suggests it did not reach the INSERT at all.