feat: require Python 3.14 (STL release) + unconditional video + EPUB scoping

.github/workflows/ci.yml

@@ -50,7 +50,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.10", "3.12"]
+        python-version: ["3.14"]
 
     steps:
       - uses: actions/checkout@v6

README.md

@@ -127,7 +127,7 @@ Every system ran on the same harness with the same model and budgets, scored by
 
 | Requirement | Minimum | Check | Install |
 |---|---|---|---|
-| Python | 3.10+ | `python --version` | [python.org](https://www.python.org/downloads/) |
+| Python | 3.14+ | `python --version` | [python.org](https://www.python.org/downloads/) |
 | uv *(recommended)* | any | `uv --version` | `curl -LsSf https://astral.sh/uv/install.sh \| sh` |
 | pipx *(alternative)* | any | `pipx --version` | `pip install pipx` |
 
@@ -250,7 +250,7 @@ Codex users also need `multi_agent = true` under `[features]` in `~/.codex/confi
 | `neo4j` | Neo4j push support | `uv tool install "graphifyy[neo4j]"` |
 | `falkordb` | FalkorDB push support | `uv tool install "graphifyy[falkordb]"` |
 | `svg` | SVG graph export | `uv tool install "graphifyy[svg]"` |
-| `leiden` | Leiden community detection (Python < 3.13 only) | `uv tool install "graphifyy[leiden]"` |
+| `leiden` | Leiden community detection | `uv tool install "graphifyy[leiden]"` |
 | `ollama` | Ollama local inference | `uv tool install "graphifyy[ollama]"` |
 | `openai` | OpenAI / OpenAI-compatible APIs | `uv tool install "graphifyy[openai]"` |
 | `gemini` | Google Gemini API | `uv tool install "graphifyy[gemini]"` |
@@ -337,7 +337,7 @@ To remove graphify from all platforms at once: `graphify uninstall` (add `--purg
 | MCP configs | `.mcp.json` `mcp.json` `mcp_servers.json` `claude_desktop_config.json` — extracts server nodes, package refs, env var requirements |
 | Package manifests | `apm.yml` `pyproject.toml` `go.mod` `pom.xml` — one canonical package node per package (by name) plus `depends_on` edges, so a package referenced from many manifests is a single hub |
 | Docs | `.md .mdx .qmd .html .txt .rst .yaml .yml` (markdown `[text](./other.md)` links and `[[wikilinks]]` become `references` edges between docs) |
-| Office | `.docx .xlsx` (requires `uv tool install graphifyy[office]`) |
+| Office | `.pptx` (built in); `.docx .xlsx` (require `uv tool install graphifyy[office]`) |
 | Google Workspace | `.gdoc .gsheet .gslides` (opt-in; requires `gws` auth and `--google-workspace`; Sheets need `uv tool install graphifyy[google]`) |
 | PDFs | `.pdf` |
 | Images | `.png .jpg .webp .gif` |
@@ -346,6 +346,14 @@ To remove graphify from all platforms at once: `graphify uninstall` (add `--purg
 
 Code is extracted **locally with no API calls** (AST via tree-sitter). Everything else goes through your AI assistant's model API.
 
+PowerPoint extraction preserves slide order, sections, hidden-slide state, text
+and bullets, speaker notes, comments, tables, chart data, SmartArt/native diagram
+relationships, hyperlinks, alt text, layout/master evidence, custom XML, and
+embedded package objects. Embedded images are sent through the normal vision
+path. Embedded audio and video are extracted with slide provenance and transcribed
+locally when `graphifyy[video]` is installed; external media links are recorded but
+never fetched automatically.
+
 Google Drive for desktop `.gdoc`, `.gsheet`, and `.gslides` files are shortcut
 pointers, not document content. To include native Google Docs, Sheets, and Slides
 in a headless extraction, install and authenticate the

docs/epub-scoping.md

@@ -0,0 +1,65 @@
+# EPUB Semantic Ingestion Scoping (STL 3.14+)
+
+## Goals (parallel to PPTX)
+- Full semantic extraction, **not** plain text.
+- Preserve chapter/spine order, structure, relationships, provenance.
+- Extract: text (XHTML/OPS), notes/annotations if present, tables, images (→ vision), embedded audio/video (→ transcription with content-aware cache), hyperlinks.
+- Emit: Markdown sidecars + structured manifest (similar to PPTX).
+- Bounded/safe parsing of untrusted EPUBs (ZIP + XML).
+- Support custom --out, watch, CLI, agent workflows.
+- Incremental + clustered liveness for generated transcripts.
+- Tests: adversarial (malformed, large, nested zips, bad XML), end-to-end with real EPUB fixtures.
+
+## Architecture Approach (like PPTX)
+- New `graphify/epub.py` (or `graphify/ebook.py`) — dedicated bounded parser.
+  - Use `zipfile` with strict limits (max members, sizes, compression ratio).
+  - Use `defusedxml` for OPF, NCX, XHTML (DTD/entity rejection, depth limits).
+  - Parse OPF for manifest/spine (reading order).
+  - Walk spine for chapter content.
+- Keep `detect.py` as dispatcher (add EPUB detection + routing).
+- Reuse `transcribe.py` for media (unconditional on 3.14+).
+- Reuse vision routing for images.
+- No heavy "ebooklib" runtime dep at core (optional for probes only, like python-pptx was for PPTX).
+
+## Key Differences from PPTX
+- EPUB structure: OPF (package doc), spine (linear reading order), manifest, NCX/toc.
+- XHTML content (not OOXML slides).
+- Often has embedded fonts, CSS — treat as unsupported/inert unless we decide to extract styles.
+- Media can be in the EPUB zip itself.
+- Chapters are the "slides" analog.
+
+## Safety Requirements (copy/adapt from PPTX)
+- Max source size, ZIP members, per-member uncompressed, aggregate, compression ratio.
+- Max XML depth/elements.
+- Reject traversal, DTD, entities, encrypted/duplicate members, active content (scripts in XHTML?).
+- Treat cache as untrusted.
+
+## Integration Points
+- `detect.py`: `convert_office_file` generalization or new `convert_epub`.
+- `cli.py`: support for EPUB in extract/watch.
+- `watch.py`: add .epub to watched extensions.
+- `tools/skillgen/...`: update references for agent EPUB workflows.
+- Tests: `tests/test_epub.py` (mirroring test_presentation.py).
+
+## Deliverables for separate PR
+- Core parser in new module.
+- Full provenance (chapter IDs, reading order, parent EPUB).
+- Media → transcript/vision nodes preserved in graphs.
+- Adversarial + real EPUB fixtures (public domain books + crafted bad ones).
+- Docs update + example.
+- No mixing with PPTX work.
+
+## Open Questions to Resolve in Scoping
+- Exact library choice for XHTML parsing (lxml with defused? html5lib?).
+- Do we extract CSS/embedded fonts as assets or inert?
+- TOC/NCX vs spine order (prefer spine).
+- Annotations/highlights in EPUB (some formats have them).
+- Performance for large novels (many chapters).
+
+## Next Immediate Steps (if approved)
+1. Inventory real EPUB test fixtures.
+2. Prototype minimal safe OPF + spine parser.
+3. Mirror PPTX test patterns.
+4. Coordinate with upstream if any existing thin EPUB PRs.
+
+Status: Ready to start dedicated branch after this STL 3.14 work lands.

docs/how-it-works.md

@@ -16,8 +16,11 @@ Video and audio files are transcribed with faster-whisper. To focus the transcri
 Claude runs in parallel over markdown, PDFs, images, and transcripts. Each subagent reads a batch of files and outputs a JSON fragment: nodes, edges, and any group relationships. The fragments are merged into a single graph.
 
 Before Pass 3, optional converters turn supported pointer/binary formats into
-Markdown sidecars under `graphify-out/converted/`. Office files (`.docx`,
-`.xlsx`) use the `[office]` extra. Google Workspace shortcuts (`.gdoc`,
+Markdown sidecars under `graphify-out/converted/`. PowerPoint (`.pptx`) uses a
+built-in bounded OOXML reader that also emits embedded images and media into the
+vision/transcription passes; embedded audio/video needs the `[video]` extra.
+Other Office files (`.docx`, `.xlsx`) use the `[office]` extra. Google
+Workspace shortcuts (`.gdoc`,
 `.gsheet`, `.gslides`) are opt-in with `--google-workspace` or
 `GRAPHIFY_GOOGLE_WORKSPACE=1` and require an authenticated `gws` CLI.
 

graphify/__main__.py

@@ -607,6 +607,7 @@ def _run_cli() -> None:
         print("    --token-budget N        per-chunk token cap for semantic extraction (default: 60000)")
         print("    --max-concurrency N     parallel semantic chunks in flight (default: 4; set 1 for local LLMs)")
         print("    --api-timeout S         per-request timeout in seconds for the LLM client (default: 600)")
+        print("    --whisper-model M       local faster-whisper model for video/audio (default: base)")
         print("    --out DIR, --output DIR output dir (default: <path>); writes <DIR>/graphify-out/")
         print("    --google-workspace      export .gdoc/.gsheet/.gslides shortcuts via gws before extraction")
         print("    --no-gitignore         ignore .gitignore and .git/info/exclude (prioritizes .graphifyignore)")

graphify/cli.py

@@ -2464,7 +2464,7 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph":
                 "[--model M] [--mode deep] [--out DIR|--output DIR] [--google-workspace] [--no-cluster] "
                 "[--no-gitignore] [--code-only] "
                 "[--max-workers N] [--token-budget N] [--max-concurrency N] "
-                "[--api-timeout S] [--postgres DSN] [--cargo] [--allow-partial] [--timing]",
+                "[--api-timeout S] [--whisper-model M] [--postgres DSN] [--cargo] [--allow-partial] [--timing]",
                 file=sys.stderr,
             )
             sys.exit(1)
@@ -2498,6 +2498,7 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph":
         cli_token_budget: int | None = None
         cli_max_concurrency: int | None = None
         cli_api_timeout: float | None = None
+        cli_whisper_model: str | None = None
         # Clustering tuning knobs
         cli_resolution: float = 1.0
         cli_exclude_hubs: float | None = None
@@ -2582,6 +2583,10 @@ def _parse_float(name: str, raw: str) -> float:
                 cli_api_timeout = _parse_float("--api-timeout", args[i + 1]); i += 2
             elif a.startswith("--api-timeout="):
                 cli_api_timeout = _parse_float("--api-timeout", a.split("=", 1)[1]); i += 1
+            elif a == "--whisper-model" and i + 1 < len(args):
+                cli_whisper_model = args[i + 1]; i += 2
+            elif a.startswith("--whisper-model="):
+                cli_whisper_model = a.split("=", 1)[1]; i += 1
             elif a == "--resolution" and i + 1 < len(args):
                 cli_resolution = _parse_float("--resolution", args[i + 1]); i += 2
             elif a.startswith("--resolution="):
@@ -2638,6 +2643,10 @@ def _parse_float(name: str, raw: str) -> float:
         # the skill.md pipeline.
         out_root = (out_dir.resolve() if out_dir else target)
         graphify_out = out_root / _GRAPHIFY_OUT
+        if cli_whisper_model:
+            # Set before incremental detection: transcript cache identity includes
+            # the model, so liveness must be computed under the requested model.
+            os.environ["GRAPHIFY_WHISPER_MODEL"] = cli_whisper_model
         graphify_out.mkdir(parents=True, exist_ok=True)
         # Persist corpus-shaping options so later update/watch/hook rebuilds
         # use the same file set as the initial extraction (#1886).
@@ -2698,6 +2707,7 @@ def _parse_float(name: str, raw: str) -> float:
             doc_files = []
             paper_files = []
             image_files = []
+            video_files = []
             deleted_files = []
             excluded_files = []
             graph_stale_sources = []
@@ -2711,13 +2721,15 @@ def _parse_float(name: str, raw: str) -> float:
                 google_workspace=google_workspace or None,
                 extra_excludes=_effective_excludes or None,
                 gitignore=_effective_gitignore,
+                transcript_root=graphify_out / "transcripts",
             )
             files_by_type = detection.get("files", {})
             new_by_type = detection.get("new_files", {})
             code_files = [Path(p) for p in new_by_type.get("code", [])]
             doc_files = [Path(p) for p in new_by_type.get("document", [])]
             paper_files = [Path(p) for p in new_by_type.get("paper", [])]
             image_files = [Path(p) for p in new_by_type.get("image", [])]
+            video_files = [Path(p) for p in new_by_type.get("video", [])]
             deleted_files = list(detection.get("deleted_files", []))
             excluded_files = list(detection.get("excluded_files", []))
             unchanged_total = sum(len(v) for v in detection.get("unchanged_files", {}).values())
@@ -2745,26 +2757,55 @@ def _parse_float(name: str, raw: str) -> float:
             doc_files = [Path(p) for p in files_by_type.get("document", [])]
             paper_files = [Path(p) for p in files_by_type.get("paper", [])]
             image_files = [Path(p) for p in files_by_type.get("image", [])]
+            video_files = [Path(p) for p in files_by_type.get("video", [])]
             deleted_files = []
             excluded_files = []
             graph_stale_sources = []
             unchanged_total = 0
 
+        if deep_mode and incremental_mode and not code_only:
+            # Deep mode widens all semantic sources below, so transcribe all live
+            # media and let Whisper's derivative cache decide whether work is due.
+            video_files = [Path(p) for p in files_by_type.get("video", [])]
+
+        transcript_files: list[Path] = []
+        media_files_due = list(video_files)
+        if not code_only and video_files:
+            from graphify.transcribe import transcribe_all
+
+            transcript_files = transcribe_all(
+                video_files,
+                output_dir=graphify_out / "transcripts",
+                force=force,
+            )
+            doc_files.extend(transcript_files)
+            files_by_type.setdefault("document", []).extend(
+                str(path) for path in transcript_files
+            )
+            if len(transcript_files) != len(video_files):
+                print(
+                    f"[graphify extract] warning: transcribed {len(transcript_files)} of "
+                    f"{len(video_files)} video/audio file(s); install graphifyy[video] "
+                    "and check media format errors for full semantic extraction",
+                    file=sys.stderr,
+                )
+
         semantic_files = doc_files + paper_files + image_files
         # --code-only: index code (pure local AST, no key) and skip the semantic
         # (doc/paper/image) pass entirely, so a mixed repo doesn't hard-fail when no
         # LLM backend is configured (#1734). Report what was skipped rather than
         # silently dropping it.
-        if code_only and semantic_files:
+        if code_only and (semantic_files or video_files):
             print(
-                f"[graphify extract] --code-only: skipping {len(semantic_files)} "
+                f"[graphify extract] --code-only: skipping {len(semantic_files) + len(video_files)} "
                 f"non-code file(s) ({len(doc_files)} docs, {len(paper_files)} papers, "
-                f"{len(image_files)} images) — no LLM extraction"
+                f"{len(image_files)} images, {len(video_files)} video/audio) — no LLM extraction"
             )
             semantic_files = []
             doc_files = []
             paper_files = []
             image_files = []
+            video_files = []
         if deep_mode and incremental_mode and not code_only:
             # Deep mode reads/writes its own cache namespace
             # (cache/semantic-deep/), so the manifest's changed-file gate is
@@ -2796,15 +2837,16 @@ def _parse_float(name: str, raw: str) -> float:
             _excl_note = f"; {len(excluded_files)} excluded" if excluded_files else ""
             print(
                 f"[graphify extract] {len(code_files)} code, {len(doc_files)} docs, "
-                f"{len(paper_files)} papers, {len(image_files)} images changed; "
+                f"{len(paper_files)} papers, {len(image_files)} images, "
+                f"{len(video_files)} video/audio changed; "
                 f"{unchanged_total} unchanged; {len(deleted_files)} deleted"
                 f"{_excl_note}"
             )
         else:
             print(
                 f"[graphify extract] found {len(code_files)} code, "
                 f"{len(doc_files)} docs, {len(paper_files)} papers, "
-                f"{len(image_files)} images"
+                f"{len(image_files)} images, {len(video_files)} video/audio"
             )
         # Surface files that were seen but not classified (extensionless non-shebang
         # project files like Dockerfile/Makefile, or unsupported extensions), so they
@@ -3181,6 +3223,33 @@ def _progress(idx: int, total: int, _result: dict) -> None:
         # absolute file lists.
         _manifest_files = _stamped_manifest_files(files_by_type, sem_result, target,
                                                    partial_source_files=_partial_semantic_files)
+        # A media file is semantically complete only when its exact content/
+        # model/prompt-addressed transcript produced semantic output. Merely
+        # attempting transcription must not stamp corrupt media or a missing
+        # faster-whisper installation as successful forever.
+        _stamped_documents = set(_manifest_files.get("document", []))
+        _failed_media: set[str] = set()
+        if media_files_due:
+            from graphify.transcribe import transcript_path_for as _transcript_path_for
+
+            for _media in media_files_due:
+                try:
+                    _expected_transcript = str(
+                        _transcript_path_for(
+                            _media,
+                            output_dir=graphify_out / "transcripts",
+                        )
+                    )
+                except OSError:
+                    _expected_transcript = ""
+                if _expected_transcript not in _stamped_documents:
+                    _failed_media.add(str(_media))
+            if _failed_media:
+                _manifest_files["video"] = [
+                    path
+                    for path in _manifest_files.get("video", [])
+                    if str(path) not in _failed_media
+                ]
 
         # Files dispatched this run but dropped by _stamped_manifest_files
         # above (failed chunk, LLM omission, or any future exclusion) still
@@ -3196,7 +3265,9 @@ def _progress(idx: int, total: int, _result: dict) -> None:
         _stamped_semantic = {
             f for _flist in _manifest_files.values() for f in _flist
         }
-        _cleared_semantic = {str(p) for p in semantic_files} - _stamped_semantic
+        _cleared_semantic = (
+            {str(p) for p in semantic_files} - _stamped_semantic
+        ) | _failed_media
 
         # Full-scan manifest saves prune rows for in-root files that left the
         # scan corpus but still exist on disk (#1908). The corpus must be the