feat(file-based): bump unstructured to 0.18.x with custom markdown preserved

[Aaron ("AJ") Steers (aaronsteers)]

Summary

Bumps unstructured from 0.10.27 to >=0.18.27,<0.19 (resolves 0.18.32). Custom markdown rendering (_render_markdown/_convert_to_markdown) is preserved — upstream's elements_to_md() doesn't handle heading depths, list-item bullets, or formula code-blocks.

Migrates FileType API from removed dict lookups to enum methods:

FILETYPE_TO_MIMETYPE[ft]  →  ft.mime_type
STR_TO_FILETYPE.get(m)    →  FileType.from_mime_type(m)
EXT_TO_FILETYPE.get(e)    →  FileType.from_extension(e)
detect_filetype(filename=) → detect_filetype(file_path=)

Pins onnxruntime>=1.23,<1.24 — transitive dep of unstructured[pdf]; 1.24.x ships no cp310 wheels, >=1.25 requires Python >=3.11.

Partition functions typed as Optional[Callable[..., Any]] with assert guards before call sites (fixes mypy). Defensive else in _read_file_locally raises ValueError for unsupported file types.

Scenario test expectations updated: corrupted-PDF error message changed from "No /Root object!" to "Unable to get page count" in 0.18.x, and PDF content classified as NarrativeText (not Title) so no # prefix.

Link to Devin session: https://app.devin.ai/sessions/b701e6538df6470a856666567993e205
Requested by: Aaron ("AJ") Steers (@aaronsteers)

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1782941877-bump-unstructured#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1782941877-bump-unstructured

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 123 tests  ±0   4 111 ✅ ±0   10m 44s ⏱️ + 3m 13s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit ad31932. ± Comparison against base commit cdd7014.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 126 tests  ±0   4 114 ✅ ±0   15m 18s ⏱️ + 2m 47s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit ad31932. ± Comparison against base commit cdd7014.

♻️ This comment has been updated with latest results.

[devin-ai-integration]

Devin Review found 2 potential issues.

[coderabbitai]

📝 Walkthrough

Walkthrough

The unstructured file parser now loads DOCX, PPTX, and PDF partition callables dynamically, uses runtime assertions for local parsing, and switches filetype detection and upload MIME handling to filetype utilities. Dependency constraints and related parser tests and scenarios were updated to match.

Changes

Optional partition dependency handling and markdown delegation

Layer / File(s)	Summary
Import structure and dynamic partition loading airbyte_cdk/sources/file_based/file_types/unstructured_parser.py	Imports and partition sentinels were updated, and the parser now loads DOCX, PPTX, and PDF partition callables dynamically before local parsing uses them.
Filetype detection and upload MIME handling airbyte_cdk/sources/file_based/file_types/unstructured_parser.py	Remote upload MIME values, MIME-based detection, name and content detection, extension fallback, and markdown helper documentation were updated to use filetype utilities directly.
Dependency version constraints pyproject.toml	Optional file-based parsing dependencies and extras were broadened, and onnxruntime was added to the file-based extra and deptry exclusion list.
Unit test mocking and scenario expectations unit_tests/sources/file_based/file_types/test_unstructured_parser.py, unit_tests/sources/file_based/scenarios/unstructured_scenarios.py	The parser test now monkeypatches module-level partition callables, and scenario fixtures and expected contents were updated for CSV inputs, PDF-like bytes, removed sample files, and extracted text changes.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested labels: dependencies, tests, refactor

Suggested reviewers: A reviewer familiar with the unstructured parser module and Poetry dependency changes wdyt?

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main change: upgrading unstructured to 0.18.x while keeping custom markdown behavior.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch devin/1782941877-bump-unstructured

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

pyproject.toml (1)

71-72: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Unbounded ranges for deps tied to unstructured's pinned line — intentional, or worth an upper bound too?

unstructured is capped at <0.19 (Line 80), but pdf2image, pdfminer.six, and nltk — all noted as used indirectly by unstructured — are left open-ended (>=... with no ceiling). Since pdfminer.six in particular has a track record of breaking downstream consumers on new releases, a future bump could silently pull in a version untested against unstructured==0.18.x. Would it make sense to add a matching upper bound (or at least document why it's intentionally left open) for these three, wdyt?

Example: adding upper bounds to align with the `unstructured` cap

-pdf2image = { version = ">=1.16.3", optional = true }
-"pdfminer.six" = { version = ">=20221105", optional = true }  # Used indirectly by unstructured library
+pdf2image = { version = ">=1.16.3,<2.0", optional = true }
+"pdfminer.six" = { version = ">=20221105,<20250000", optional = true }  # Used indirectly by unstructured library

Also applies to: 79-79

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyproject.toml` around lines 71 - 72, The dependency specs for pdf2image,
pdfminer.six, and nltk are unbounded while unstructured is pinned below 0.19, so
update the dependency declarations in pyproject.toml to either add matching
upper bounds for these indirect unstructured-related packages or add an explicit
note documenting why they are intentionally left open-ended; use the existing
package entries for pdf2image, pdfminer.six, and nltk as the place to make the
change.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@airbyte_cdk/sources/file_based/file_types/unstructured_parser.py`:
- Around line 423-440: The content detection path in unstructured_parser.file
type inference is dropping the URI context, which can cause remote files to be
misclassified before the extension fallback. Update the detection flow around
the file_type lookup and detect_filetype call in the file type resolution method
to pass remote_file.uri through as metadata_file_path (or equivalent URI hint)
so unsupported text-like formats can be identified correctly; keep the existing
extension fallback as the final step.

In `@unit_tests/sources/file_based/file_types/test_unstructured_parser.py`:
- Around line 243-248: The test is assigning mocks directly to the
unstructured_parser_module globals, which can leak into later tests and bypass
the missing-dependency behavior. Update the test to use monkeypatch.setattr on
unstructured_partition_pdf, unstructured_partition_docx, and
unstructured_partition_pptx so the original None sentinels are automatically
restored after the test. Keep the fix scoped to the test setup that currently
uses MagicMock and the unstructured_parser_module symbols.

---

Nitpick comments:
In `@pyproject.toml`:
- Around line 71-72: The dependency specs for pdf2image, pdfminer.six, and nltk
are unbounded while unstructured is pinned below 0.19, so update the dependency
declarations in pyproject.toml to either add matching upper bounds for these
indirect unstructured-related packages or add an explicit note documenting why
they are intentionally left open-ended; use the existing package entries for
pdf2image, pdfminer.six, and nltk as the place to make the change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a2a1a415-2efb-4519-82ae-c8f88ce55a88

📥 Commits

Reviewing files that changed from the base of the PR and between cdd7014 and 158d1d4.

⛔ Files ignored due to path filters (1)

• poetry.lock is excluded by !**/*.lock

📒 Files selected for processing (4)

• airbyte_cdk/sources/file_based/file_types/unstructured_parser.py
• pyproject.toml
• unit_tests/sources/file_based/file_types/test_unstructured_parser.py
• unit_tests/sources/file_based/scenarios/unstructured_scenarios.py

[Aaron ("AJ") Steers (aaronsteers)]

Devin, fyi, take note I retitled the PR and don't override it please. Do let me know if my "latest" annotation on this version ref is not accurate. (Ideally it is the latest.)

[devin-ai-integration]

Noted — I won't override the title. The "latest" annotation is accurate: latest on PyPI is 0.18.32, and our constraint >=0.18.27,<0.19 includes it. Poetry resolved to 0.18.32 in the lockfile.

---

Devin session

[coderabbitai]

🧹 Nitpick comments (1)

airbyte_cdk/sources/file_based/file_types/unstructured_parser.py (1)

340-369: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Missing else leaves elements unbound if filetype isn't PDF/DOCX/PPTX.

The try only wraps the if/elif chain; if none of the branches match, elements is never assigned, and line 369 raises an uncaught NameError instead of a RecordParseError. Today this can't happen since _supported_file_types() filters MD/TXT upstream, but it's a latent trap if that set is ever extended without updating this chain. wdyt about adding an explicit else that raises self._create_parse_error(...) for future-proofing?

🛡️ Proposed fix

         try:
             if filetype == FileType.PDF:
                 file_handle.seek(0)
                 with BytesIO(file_handle.read()) as file:
                     file_handle.seek(0)
                     elements = unstructured_partition_pdf(file=file, strategy=strategy)
             elif filetype == FileType.DOCX:
                 elements = unstructured_partition_docx(file=file)
             elif filetype == FileType.PPTX:
                 elements = unstructured_partition_pptx(file=file)
+            else:
+                raise ValueError(f"Unsupported file type for local parsing: {filetype}")
         except Exception as e:
             raise self._create_parse_error(remote_file, str(e))

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@airbyte_cdk/sources/file_based/file_types/unstructured_parser.py` around
lines 340 - 369, The _read_file_locally method in UnstructuredParser can leave
elements unbound when filetype is not PDF, DOCX, or PPTX, causing a NameError
instead of a parse error. Add an explicit else branch in the if/elif chain
inside _read_file_locally to raise self._create_parse_error(remote_file, ...)
for unsupported filetypes, so the method always fails with the intended
RecordParseError path.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@airbyte_cdk/sources/file_based/file_types/unstructured_parser.py`:
- Around line 340-369: The _read_file_locally method in UnstructuredParser can
leave elements unbound when filetype is not PDF, DOCX, or PPTX, causing a
NameError instead of a parse error. Add an explicit else branch in the if/elif
chain inside _read_file_locally to raise self._create_parse_error(remote_file,
...) for unsupported filetypes, so the method always fails with the intended
RecordParseError path.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: d9c3e7a5-1ee6-4913-a635-94a1c0e96ed8

📥 Commits

Reviewing files that changed from the base of the PR and between 158d1d4 and ae389e3.

⛔ Files ignored due to path filters (1)

• poetry.lock is excluded by !**/*.lock

📒 Files selected for processing (4)

• airbyte_cdk/sources/file_based/file_types/unstructured_parser.py
• pyproject.toml
• unit_tests/sources/file_based/file_types/test_unstructured_parser.py
• unit_tests/sources/file_based/scenarios/unstructured_scenarios.py

[devin-ai-integration]

☑️ Resolved in 2b58f6b. Added explicit else raising ValueError for unsupported file types so it surfaces as a RecordParseError via the existing except handler instead of an unbound NameError.

(Responding to CodeRabbit nitpick on _read_file_locally missing else branch)