feat: allow access tokens to enumerate groups

AGENTS.md

@@ -37,10 +37,10 @@ Upload bearer tokens are long-lived, reusable credentials, not one-time codes:
 ## Personal access token lifecycle
 
 Personal access tokens (`PersonalAccessToken`, raw prefix `gpat_`) are the
-**read-only** counterpart to upload tokens. They authenticate reads — currently the
-streaming group export (`GET /api/v1/groups/<slug>/export/`) — never uploads. They
-are a distinct model and credential from `UploadToken`; the two are never
-interchangeable.
+**read-only** counterpart to upload tokens. They authenticate the group index and
+streaming group export (`GET /api/v1/groups/` and
+`GET /api/v1/groups/<slug>/export/`) — never uploads. They are a distinct model and
+credential from `UploadToken`; the two are never interchangeable.
 
 - A user mints and revokes their own from the profile page; the raw token is shown
   exactly once and only its hash is stored.

docs/api-v1.md

@@ -7,24 +7,30 @@ JSONL evidence.
 
 ## Authentication
 
-All read APIs require a logged-in Goggles user. Upload APIs use reusable bearer
-tokens and are documented separately in the app workflow notes.
-
-The streaming group export additionally accepts a **personal access token** — a
-read-only bearer credential a user mints from their profile page (or an operator
-mints for a service account with `manage.py create_access_token "<name>" --user
-<username>`). Send it as `Authorization: Bearer gpat_…`. A personal access token
-authorizes only the streaming group export (below) — not uploads, and not the
-session-authenticated read APIs; it is revoked by the owner, by an admin, by expiry,
-or by deactivating the owning user. Upload tokens (`goggles_…`) and personal access
-tokens (`gpat_…`) are distinct credentials and never interchangeable.
+Most read APIs require a logged-in Goggles user session. The group index and
+streaming group export additionally accept a **personal access token** bearer
+credential. Send it as `Authorization: Bearer gpat_…`. On those bearer-enabled
+endpoints, missing, malformed, invalid, inactive, expired, or owner-deactivated
+credentials return `401` JSON — never an HTML login redirect.
+
+Upload APIs use reusable upload bearer tokens (`goggles_…`) and are documented
+separately in the app workflow notes. Upload tokens never authorize read APIs.
+
+Personal access tokens are read-only credentials a user mints from their profile
+page (or an operator mints for a service account with
+`manage.py create_access_token "<name>" --user <username>`). They authorize the
+group list (`GET /api/v1/groups/`) and the streaming group export
+(`GET /api/v1/groups/{slug}/export/`) — not uploads or the other
+session-authenticated projection APIs. A token is revoked by the owner, by an
+admin, by expiry, or by deactivating the owning user. Upload tokens and
+personal access tokens are distinct credentials and never interchangeable.
 
 The current internal deployment treats authenticated Goggles users as one shared
-internal tenant. Even so, endpoint implementations should route through
-object-level scope checks before returning group, account, engine, message,
-report, or evidence data. If Goggles later adds tenant or account isolation,
-unauthorized resources should use the same not-found style behavior as unknown
-resources so callers cannot enumerate data outside their scope.
+internal tenant. Even so, endpoint implementations route through a shared
+object-level readable-group scope before returning group data. If Goggles later
+adds tenant or account isolation, unauthorized groups are omitted from the list
+and exports for unknown slugs return `404`, indistinguishable from a missing
+group, so callers cannot enumerate data outside their scope.
 
 Responses must not expose bearer tokens, upload secrets, source IPs, or user
 agents. Derived projection responses carry pointer-only evidence refs. Raw event
@@ -109,6 +115,71 @@ metadata is the field-level hook for enforcing stricter access.
 - `GET /api/v1/accounts/{account_ref}/groups/`
 - `GET /api/v1/engines/{engine_id}/groups/`
 
+`GET /api/v1/groups/` accepts a logged-in session or a personal access token.
+It returns metadata for every group the reader may export. Newly-created groups
+appear on the next poll without manual slug configuration.
+
+Query parameters:
+
+- `limit`: defaults to `100`, capped at `500`
+- `cursor`: optional opaque continuation token from a previous page's
+  `pagination.next_cursor`. Omit on the first page of a poll.
+- `updated_since`: optional ISO-8601 timestamp; when set, only groups with
+  `updated_at` strictly after this value are returned. The response echoes the
+  applied filter as `updated_since`. This is a **best-effort** change hint only:
+  it does not guarantee every group that became visible since your last poll (see
+  [Polling contract](#polling-contract)).
+
+Results are ordered by `updated_at desc` with a stable internal tie-breaker. The
+first page of each poll fixes a server `polling_watermark` timestamp; every page
+in that traversal reuses the same watermark and only includes groups with
+`updated_at` at or before it. Groups that are still uncommitted, or that commit
+after the watermark is captured, can require a later full index poll. The
+watermark is **not** a
+commit-safe upper bound on `updated_at`: a group can commit after page 1 with
+`updated_at` at or before the watermark and still be invisible to both the
+remaining pages and a subsequent `updated_since=polling_watermark` poll.
+
+Paginated responses include:
+
+```json
+{
+  "pagination": {
+    "limit": 100,
+    "returned": 10,
+    "has_more": true,
+    "next_cursor": "…"
+  },
+  "polling_watermark": "2026-07-29T09:00:00+00:00"
+}
+```
+
+#### Polling contract
+
+1. Start each poll without `cursor`. Read `polling_watermark` from the first
+   response and keep it for the whole traversal. Use it only to bound that
+   traversal; do not treat it as a commit-safe cursor for change detection.
+2. Follow `pagination.next_cursor` until `has_more` is `false`. Each cursor is
+   bound to that poll's watermark and original `updated_since` filter, so only
+   `cursor` and the desired `limit` need to be sent on continuation requests.
+   Tampered or foreign cursors return `400` `{"error":"invalid cursor"}`.
+3. After completing a traversal, you may set `updated_since` to the maximum
+   `updated_at` among groups you actually received to skip unchanged groups on
+   the next incremental poll. This is an optimization only: uploads assign
+   `updated_at` before commit, so a group can appear after your traversal with
+   `updated_at` at or before your `updated_since` bound and be omitted from
+   incremental polls.
+4. For **eventual completeness**, periodically run a full index poll (omit
+   `updated_since` and `cursor`) and deduplicate by `slug`. A finite overlap
+   window alone does not guarantee discovery of arbitrarily delayed commits.
+   An empty `groups` array on an incremental poll means no group has
+   `updated_at` strictly after your `updated_since` bound; it does **not** prove
+   the index is unchanged.
+
+Projection endpoints still use numeric `offset` pagination (see
+[Common Query Parameters](#common-query-parameters)); only the group index uses
+cursor pagination.
+
 Group responses include `schema_version`, group summary fields, tab counts, and
 classification metadata indicating whether full-data audit content may be
 present.

forensics/analysis.py

@@ -239,8 +239,11 @@ def file_rows_for_group(audit_files, group):
 # ---------------------------------------------------------------------------
 
 
-def group_list_rows():
-    groups = list(AuditGroup.objects.all())
+def group_list_rows(groups=None):
+    if groups is None:
+        groups = list(AuditGroup.objects.all())
+    else:
+        groups = list(groups)
     group_file_counts = audit_file_counts_for_groups(groups)
     group_event_stats, fork_group_ids = event_stats_for_groups(groups)
     for group in groups:

forensics/group_list_cursor.py

@@ -0,0 +1,90 @@
+"""Opaque keyset cursor helpers for the group index API."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime
+
+from django.core import signing
+from django.core.signing import BadSignature
+from django.db.models import Q
+from django.utils import timezone
+from django.utils.dateparse import parse_datetime
+
+CURSOR_SALT = "goggles-groups-v1-cursor"
+
+
+class InvalidGroupListCursor(ValueError):
+    pass
+
+
+@dataclass(frozen=True)
+class GroupListCursor:
+    watermark: datetime
+    updated_at: datetime
+    group_id: int
+    updated_since: datetime | None
+
+    def keyset_filter(self) -> Q:
+        return Q(updated_at__lt=self.updated_at) | Q(
+            updated_at=self.updated_at,
+            pk__gt=self.group_id,
+        )
+
+
+def encode_group_list_cursor(
+    *,
+    watermark: datetime,
+    updated_at: datetime,
+    group_id: int,
+    updated_since: datetime | None,
+) -> str:
+    payload = {
+        "w": watermark.isoformat(),
+        "u": updated_at.isoformat(),
+        "i": group_id,
+        "a": updated_since.isoformat() if updated_since is not None else None,
+    }
+    return signing.dumps(payload, salt=CURSOR_SALT, compress=True)
+
+
+def decode_group_list_cursor(raw_cursor: str) -> GroupListCursor:
+    if not raw_cursor:
+        raise InvalidGroupListCursor("cursor is required")
+    try:
+        payload = signing.loads(raw_cursor, salt=CURSOR_SALT)
+        if not isinstance(payload, dict):
+            raise InvalidGroupListCursor("cursor payload is invalid")
+        watermark = _parse_cursor_timestamp(payload.get("w"))
+        updated_at = _parse_cursor_timestamp(payload.get("u"))
+        group_id = payload.get("i")
+        updated_since = _parse_optional_cursor_timestamp(payload.get("a"))
+        if not isinstance(group_id, int) or group_id <= 0:
+            raise InvalidGroupListCursor("cursor group id is invalid")
+        return GroupListCursor(
+            watermark=watermark,
+            updated_at=updated_at,
+            group_id=group_id,
+            updated_since=updated_since,
+        )
+    except InvalidGroupListCursor:
+        raise
+    except (BadSignature, TypeError, ValueError) as exc:
+        raise InvalidGroupListCursor("cursor is invalid") from exc
+
+
+def _parse_cursor_timestamp(value) -> datetime:
+    if not isinstance(value, str) or not value:
+        raise InvalidGroupListCursor("cursor timestamp is invalid")
+    parsed = parse_datetime(value)
+    if parsed is None:
+        raise InvalidGroupListCursor("cursor timestamp is invalid")
+    if timezone.is_naive(parsed):
+        parsed = timezone.make_aware(parsed, timezone.get_current_timezone())
+    return parsed
+
+
+def _parse_optional_cursor_timestamp(value) -> datetime | None:
+    if value is None:
+        return None
+    return _parse_cursor_timestamp(value)

forensics/migrations/0014_auditgroup_updated_at_id_index.py

@@ -0,0 +1,17 @@
+# Generated by Django 6.0.7 on 2026-07-29 08:43
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('forensics', '0013_personalaccesstoken'),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name='auditgroup',
+            index=models.Index(fields=['-updated_at', 'id'], name='forensics_a_updated_8f04c0_idx'),
+        ),
+    ]

forensics/models.py

@@ -21,6 +21,9 @@ class AuditGroup(models.Model):
 
     class Meta:
         ordering = ["-updated_at", "-created_at"]
+        indexes = [
+            models.Index(fields=["-updated_at", "id"]),
+        ]
 
     def __str__(self) -> str:
         return self.name