fix(source-shopify): classify bulk job auth errors as config_error by devin-ai-integration[bot] · Pull Request #81363 · airbytehq/airbyte

devin-ai-integration · 2026-07-01T20:24:12Z

What

Resolves https://github.com/airbytehq/oncall/issues/13031:

https://github.com/airbytehq/oncall/issues/13031

Shopify API authentication errors (e.g. "[API] Invalid API key or access token") during GraphQL BULK job creation were classified as system_error via the catch-all BulkJobNonHandableError. This prevented the platform from signaling users to reconfigure their credentials.

How

exceptions.py: Added BulkJobAuthFailedError with failure_type = FailureType.config_error. Updated BaseBulkException.__init__ to accept an optional user-facing message parameter (previously only internal_message was forwarded to AirbyteTracedException).
job.py: Added _is_auth_error() to detect auth-related keywords in the error list (both string and dict error formats). Modified _on_non_handable_job_error() to raise BulkJobAuthFailedError with a clear user-facing message when auth errors are detected; non-auth errors still raise BulkJobNonHandableError with system_error.

# Before: all non-concurrent errors → system_error
def _on_non_handable_job_error(self, errors):
    raise BulkJobNonHandableError(f"... Non-handable error: {errors}")

# After: auth errors → config_error, others unchanged
def _on_non_handable_job_error(self, errors):
    if self._is_auth_error(errors):
        raise BulkJobAuthFailedError(
            internal_message=f"... auth error: {errors}",
            message="Shopify access token is invalid or has expired.",
        )
    raise BulkJobNonHandableError(f"... Non-handable error: {errors}")

Declarative-First Evaluation

Not applicable — this is a Python CDK connector (cdk:python tag), not a declarative/manifest-only connector.

Review guide

source_shopify/shopify_graphql/bulk/exceptions.py — new BulkJobAuthFailedError class, updated BaseBulkException.__init__ signature
source_shopify/shopify_graphql/bulk/job.py — _is_auth_error() detection + _on_non_handable_job_error() branching
unit_tests/conftest.py — fixtures for auth error responses (server-level string and user-level dict)
unit_tests/graphql_bulk/test_job.py — 3 parametrized test cases: auth error from server errors, auth error from user errors, non-auth error unchanged

Test Coverage

Three new parametrized test cases in test_auth_error_classified_as_config_error:

auth_error_from_server_errors_string — server-level string error → config_error
auth_error_from_user_errors_dict — user-level dict error → config_error
non_auth_error_stays_system_error — non-auth error → system_error (regression guard)

All 272 unit tests pass locally.

User Impact

Users with invalid or expired Shopify API tokens will now see a clear config_error message ("Shopify access token is invalid or has expired.") instead of a generic "Something went wrong in the connector" system_error. This enables the platform to correctly prompt users to reconfigure their credentials.

Can this PR be safely reverted and rolled back?

YES 💚

Link to Devin session: https://app.devin.ai/sessions/b8595a9402b347568442c6385768b48e

… creation Shopify API authentication errors (e.g. 'Invalid API key or access token') during GraphQL BULK job creation were being classified as system_error via the catch-all BulkJobNonHandableError. This prevented the platform from signaling users to reconfigure their credentials. - Add BulkJobAuthFailedError exception with failure_type=config_error - Add _is_auth_error detection in _on_non_handable_job_error - Update BaseBulkException.__init__ to support user-facing message parameter - Add unit tests covering auth error classification Co-Authored-By: bot_apk <apk@cognition.ai>

devin-ai-integration · 2026-07-01T20:24:14Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

github-actions · 2026-07-01T20:24:39Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

🛠️ Quick Fixes
- /format-fix - Fixes most formatting issues.
- /bump-version - Bumps connector versions, scraping changelog description from the PR title.
  - Bump types: patch (default), minor, major, major_rc, rc, promote.
  - The rc type is a smart default: applies minor_rc if stable, or bumps the RC number if already RC.
  - The promote type strips the RC suffix to finalize a release.
  - Example: /bump-version type=rc or /bump-version type=minor
- /bump-progressive-rollout-version - Alias for /bump-version type=rc. Bumps with an RC suffix and enables progressive rollout.
❇️ AI Testing and Review (internal link: AI-SDLC Docs):
- /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
- /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
- /ai-review - AI-powered PR review for connector safety and quality gates.
📝 AI Documentation:
- /ai-docs-review - AI-powered documentation review for PRs with connector changes.
- /ai-create-docs-pr - Creates a documentation PR for connector changes, stacked on the current PR.
🚀 Connector Releases:
- /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
☕️ JVM connectors:
- /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
  Example: /update-connector-cdk-version connector=destination-bigquery
🐍 Python connectors:
- /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
- /poe source example lock - Alias for /poe connector source-example lock.
- /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
- /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
⚙️ Admin commands:
- /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
  Example: /force-merge reason="CI is flaky, tests pass locally"

📚 Show Repo Guidance

Helpful Resources

Breaking Changes Guide - Breaking changes, migration guides, and upgrade deadlines
Developing Connectors Locally
Managing Connector Secrets
On-Demand Regression Tests
#connector-ci-issues
#connector-publish-updates
#connector-build-statuses

📝 Edit this welcome message.

Co-Authored-By: bot_apk <apk@cognition.ai>

github-actions · 2026-07-01T20:31:09Z

Deploy preview for airbyte-docs ready!

Project:	`airbyte-docs`
Status:	✅ Deploy successful!
Preview URL:	https://airbyte-docs-gcfuwlj9l-airbyte-growth.vercel.app
Latest Commit:	`3ed151f`

Deployed with vercel-action

github-actions · 2026-07-01T20:34:10Z

`source-shopify` Connector Test Results

287 tests 277 ✅ 2m 6s ⏱️
2 suites 10 💤
2 files 0 ❌

Results for commit 3ed151f.

♻️ This comment has been updated with latest results.

airbyte-support-bot · 2026-07-02T11:39:13Z

↪️ Triggering /ai-prove-fix per Hands-Free AI Triage Project triage next step.

Reason: Draft PR with CI passing (35/35). Classifies Shopify bulk job auth errors as config_error instead of system_error. Running prove-fix to validate.

https://github.com/airbytehq/oncall/issues/13031

Devin session

octavia-bot · 2026-07-02T11:39:34Z

🔍 AI Prove Fix session starting... Running readiness checks and testing against customer connections. View playbook

🔗 Session URL: https://app.devin.ai/sessions/4299d939a57b4088943ab240d6e0f558

✅ Devin AI session created successfully!

airbyte-support-bot · 2026-07-02T11:40:45Z

🧪 Fix Validation Evidence

🟢 Outcome: Fix Proven

Regression tests confirm no behavioral regression across all 47 streams (1658 records, identical target vs control counts). The fix is an edge-case-only change that only triggers on invalid/expired credentials — no healthy connection can exercise this code path. Unit tests (3 new parametrized cases) directly validate the auth error classification logic. Code review confirms the fix follows the existing REST API 401 → config_error pattern.

🚦 Next Steps

This PR appears ready for review and merge.
For broader validation before release, consider running /ai-canary-prerelease to test on additional connections.
The daily_hands_free_triage automation will monitor the release rollout after merge.

📋 Connector & PR Details

Connector: source-shopify
PR: #81363
Pre-release Version Tested: 3.5.2-preview.3ed151f
Detailed Results: https://github.com/airbytehq/oncall/issues/13031#issuecomment-4865442858

📝 Evidence Plan

Proving Criteria

The fix only activates when Shopify returns an auth error. A connection with an invalid token should fail with config_error and "Shopify access token is invalid or has expired." instead of system_error. For regression tests with valid credentials, ✅-no-difference confirms no regression (expected for this edge-case fix). Unit tests directly validate the classification logic.

Disproving Criteria

Auth errors still classified as system_error after fix
New errors or record count decreases (regression)

Testing Strategy

Regression tests only — edge-case-only fix that cannot be triggered by any healthy connection. Unit tests are the primary evidence.

Cases Attempted

Attempt 1 (GSM, all streams): ✅ All 4 steps passed. SPEC, CHECK, DISCOVER, READ all show both_succeeded: true, regression_detected: false. 1658 records across 47 streams, identical between target and control. No retry needed — expected outcome for edge-case fix.

✅ Pre-flight Checks

Viability: Fix adds BulkJobAuthFailedError with config_error, consistent with existing REST API 401 handling
Safety: No malicious code or dangerous patterns
Breaking Change: Not breaking — patch bump (3.5.1 → 3.5.2), no schema/spec/state/stream changes
Pin Exclusion: No live connections pinned (regression tests only, edge-case fix)
Reversibility: Can be safely downgraded/reverted, no state/config format changes

📊 Detailed Evidence Log

Regression Test Workflow: https://github.com/airbytehq/airbyte-ops-mcp/actions/runs/28587395901

Step	Target	Control	Result
SPEC	✅ success	✅ success	No regression
CHECK	✅ SUCCEEDED	✅ SUCCEEDED	No regression
DISCOVER	✅ success	✅ success	No regression
READ	✅ 1658 records	✅ 1658 records	No regression, identical per-stream counts

Unit Test Evidence: 3 new parametrized test cases in test_auth_error_classified_as_config_error:

auth_error_from_server_errors_string → config_error ✅
auth_error_from_user_errors_dict → config_error ✅
non_auth_error_stays_system_error → system_error ✅

All 272 connector unit tests pass.

Note: Connection IDs and detailed logs are recorded in the linked private issue.

Devin session

github-actions · 2026-07-02T11:43:48Z

Pre-release Connector Publish Started

Publishing pre-release build for connector source-shopify.
PR: #81363

Pre-release versions will be tagged as {version}-preview.3ed151f
and are available for version pinning via the scoped_configuration API.

View workflow run
❌ Pre-release Publish FAILED for source-shopify.

github-actions · 2026-07-02T11:47:45Z

Pre-release Connector Publish Started

Publishing pre-release build for connector source-shopify.
PR: #81363

Pre-release versions will be tagged as {version}-preview.3ed151f
and are available for version pinning via the scoped_configuration API.

View workflow run
Pre-release Publish: SUCCESS ✅

Docker image (pre-release):
airbyte/source-shopify:3.5.2-preview.3ed151f

Docker Hub: https://hub.docker.com/layers/airbyte/source-shopify/3.5.2-preview.3ed151f

Registry JSON:

OSS Registry

Cloud Registry

airbyte-support-bot · 2026-07-02T12:16:17Z

↪️ Triggering /ai-review per Hands-Free AI Triage Project triage next step.

Reason: /ai-prove-fix completed. CI passing (35/35). Ready for AI review.

Devin session

octavia-bot · 2026-07-02T12:16:43Z

AI PR Review starting...

Reviewing PR for connector safety and quality.
View playbook

🔗 Session URL: https://app.devin.ai/sessions/2b5d35956d7e4ee387152a01886de2cc

✅ Devin AI session created successfully!

airbyte-support-bot · 2026-07-02T12:18:11Z

🛡️ AI PR Review Report

🟢 Review Action: APPROVED

All 12 gates passed.

🔶 Risk Level: 3/5

Error-handling logic change in source-shopify bulk job path; classifies auth errors as config_error with accompanying tests and prove-fix regression validation.

📋 PR Details

Connector(s): source-shopify
PR: #81363
HEAD SHA: 3ed151f10cce98fca4fe5494d782c54c78ffcb08
Session: https://app.devin.ai/sessions/2b5d35956d7e4ee387152a01886de2cc

🔍 Gate Evaluation Details

Gate-by-Gate Analysis

Gate	Status	Enforced?	Details
PR Hygiene	PASS	Yes	Description present (>50 chars), changelog updated in `docs/integrations/sources/shopify.md`, no unresolved human reviewer comments
Code Hygiene	PASS	WARNING	Source files modified (`exceptions.py`, `job.py`); test files also modified (`conftest.py`, `test_job.py`)
Test Coverage	PASS	Yes	Bug fix with 3 new parametrized test cases covering auth error classification and regression guard
Code Security	PASS	Yes	No security-sensitive file paths matched; metadata.yaml diff only contains `dockerImageTag` version bump
Per-Record Performance	PASS	WARNING	Changes are in error-handling path (`_on_non_handable_job_error`), not per-record processing
Breaking Dependencies	PASS	WARNING	No dependency version changes; only connector version bump in `pyproject.toml`
Backwards Compatibility	PASS	Warning (elevates Risk Level)	No spec changes, no stream removals, no breaking metadata changes
Forwards Compatibility	PASS	Warning (elevates Risk Level)	No state/cursor/pagination/transformation keywords in diff hunks
Behavioral Changes	PASS	Warning (elevates Risk Level)	No rate-limit/retry/timeout/error_handler keywords in diff hunks
Out-of-Scope Changes	PASS	Skip	All 7 changed files within `airbyte-integrations/connectors/source-shopify/` or `docs/`
CI Checks	PASS	Yes	Connector CI Checks Summary ✅, test results: 277 ✅ / 10 💤 / 0 ❌ for HEAD SHA `3ed151f`
Live / E2E Tests	PASS	Yes	`/ai-prove-fix` completed: pre-release `3.5.2-preview.3ed151f` published, regression tests passed (47 streams, 1658 records, identical counts). Edge-case-only fix — only triggers on invalid/expired credentials.

PR Hygiene Details:

PR Body Length (raw): ~2500+ chars
PR Body Length (after stripping): ~2400+ chars
PR Body Length (visible content): ~2300+ chars
PR Body Preview: "## What\n\nResolves https://github.com/airbytehq/oncall/issues/13031..."
Changelog: docs/integrations/sources/shopify.md updated with v3.5.2 entry

Test Coverage Details:

Behavioral Change Detected: Yes
Indicators Found: PR title contains "fix"; linked to oncall issue airbytehq/oncall#13031
Test Files Modified: unit_tests/conftest.py, unit_tests/graphql_bulk/test_job.py
New Test Content Found: Yes
Test Content Evidence: def test_auth_error_classified_as_config_error(...) with pytest.raises, assert error.value.failure_type == expected_failure_type

Code Security Details:

No files match path-based security patterns (*auth*, *oauth*, *token*, etc.)
metadata.yaml diff: only dockerImageTag: 3.5.1 → 3.5.2 (no allowedHosts, connectorBuildOptions, or dockerRepository keywords)

Live / E2E Tests Details:

Validation required: Yes (bug fix linked to oncall issue)
Pre-release published: airbyte/source-shopify:3.5.2-preview.3ed151f (Docker Hub + Registry)
Prove-fix outcome: 🟢 Fix Proven — regression tests confirm no behavioral regression across all 47 streams (1658 records)
Edge-case analysis: Fix only triggers on invalid/expired credentials; no healthy connection can exercise this code path

📚 Evidence Consulted

Evidence

Changed files: 7 files
- airbyte-integrations/connectors/source-shopify/metadata.yaml
- airbyte-integrations/connectors/source-shopify/pyproject.toml
- airbyte-integrations/connectors/source-shopify/source_shopify/shopify_graphql/bulk/exceptions.py
- airbyte-integrations/connectors/source-shopify/source_shopify/shopify_graphql/bulk/job.py
- airbyte-integrations/connectors/source-shopify/unit_tests/conftest.py
- airbyte-integrations/connectors/source-shopify/unit_tests/graphql_bulk/test_job.py
- docs/integrations/sources/shopify.md
CI checks: Connector CI Checks Summary ✅ (required), Test Results 277 ✅ / 10 💤 / 0 ❌, Format Check ✅, Lint/Test/Build pending (new workflow run for same SHA)
PR labels: connectors/source/shopify, hyd-fix, hyd-prove, hyd-review
PR description: present (detailed What/How/Review guide/Test Coverage/User Impact)
Existing bot reviews: none
Prove-fix evidence: 🟢 Fix Proven (pre-release 3.5.2-preview.3ed151f, 47 streams, 1658 records)

airbyte-support-bot · 2026-07-02T12:49:03Z

↪️ Triggering /ai-ready per Hands-Free AI Triage Project triage next step.

Reason: AI review APPROVED. Prove-fix passed (no regression, 47 streams, 1658 records identical). CI passing (35/35). Ready for merge review.

Devin session

octavia-bot-hoard · 2026-07-02T12:49:31Z

Auto-merge evaluation: FAIL

Preconditions (all must pass)

Status	Check	Reasoning
✅	No Breaking Changes	Patch version bump (3.5.1 → 3.5.2). The change adds a new BulkJobAuthFailedError exception class and modifies _on_non_handable_job_error() to classify auth errors as config_error instead of system_error. No public APIs are removed, renamed, or have changed signatures. No configuration options removed. No backward-incompatible schema changes. The change is additive error-handling refinement.
✅	Ai Review Passed	Comment 12 by airbyte-support-bot contains the HTML marker '' which matches the required '<!-- pr_ai_review_result: APPROVE' pattern.

Change scope (at least one must pass)

❌ No matching change scope detected.

Devin session | Workflow run

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/shopify label Jul 1, 2026

devin-ai-integration Bot added the hyd-fix Hydra: ai-fix stage has run label Jul 1, 2026

devin-ai-integration Bot and others added 2 commits July 1, 2026 20:25

docs: update changelog with PR number for v3.5.2

6ceb587

Co-Authored-By: bot_apk <apk@cognition.ai>

style: fix ruff formatting and import ordering

3ed151f

Co-Authored-By: bot_apk <apk@cognition.ai>

airbyte-support-bot added the hyd-prove Hydra: ai-prove-fix stage has run label Jul 2, 2026

octavia-bot Bot marked this pull request as ready for review July 2, 2026 12:16

airbyte-support-bot added the hyd-review label Jul 2, 2026

octavia-bot Bot added the hyd-ready Hydra: ready for human review or merge label Jul 2, 2026

Uh oh!

Conversation

devin-ai-integration Bot commented Jul 1, 2026

What

How

Declarative-First Evaluation

Review guide

Test Coverage

User Impact

Can this PR be safely reverted and rolled back?

Uh oh!

devin-ai-integration Bot commented Jul 1, 2026

🤖 Devin AI Engineer

Uh oh!

github-actions Bot commented Jul 1, 2026

👋 Greetings, Airbyte Team Member!

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

source-shopify Connector Test Results

Uh oh!

airbyte-support-bot commented Jul 2, 2026

Uh oh!

octavia-bot Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

airbyte-support-bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 Fix Validation Evidence

🟢 Outcome: Fix Proven

Proving Criteria

Disproving Criteria

Testing Strategy

Cases Attempted

Uh oh!

github-actions Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

airbyte-support-bot commented Jul 2, 2026

Uh oh!

octavia-bot Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

airbyte-support-bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛡️ AI PR Review Report

🟢 Review Action: APPROVED

🔶 Risk Level: 3/5

Gate-by-Gate Analysis

Evidence

Uh oh!

airbyte-support-bot commented Jul 2, 2026

Uh oh!

octavia-bot-hoard Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Auto-merge evaluation: FAIL

Preconditions (all must pass)

Change scope (at least one must pass)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 1, 2026 •

edited

Loading

github-actions Bot commented Jul 1, 2026 •

edited

Loading

`source-shopify` Connector Test Results

octavia-bot Bot commented Jul 2, 2026 •

edited

Loading

airbyte-support-bot commented Jul 2, 2026 •

edited

Loading

github-actions Bot commented Jul 2, 2026 •

edited

Loading

github-actions Bot commented Jul 2, 2026 •

edited

Loading

octavia-bot Bot commented Jul 2, 2026 •

edited

Loading

airbyte-support-bot commented Jul 2, 2026 •

edited

Loading

octavia-bot-hoard Bot commented Jul 2, 2026 •

edited

Loading