[feat]: add domain blocklist#2274
Merged
Merged
Conversation
🦋 Changeset detectedLatest commit: 54c22df The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
seanmcguire12
changed the title
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
6 issues found across 9 files
Confidence score: 2/5
- In
packages/core/lib/v3/understudy/domainPolicy.ts, hostname matching does not canonicalize trailing-dot hosts, soexample.com.can evade block rules and weaken the core policy guarantee — normalize parsed hostnames (including trailing-dot handling) before blocklist comparison. - In
packages/core/lib/v3/understudy/context.ts, attach flow swallowsFetch.enablefailures, which can leave newly attached targets unprotected even aftersetDomainPolicyreports success — treat this as a hard failure (or retry/propagate) before merging so enforcement is consistent across sessions. - In
packages/core/lib/v3/types/public/context.ts, the new publicsetDomainPolicypath is missing requiredflowLoggerinstrumentation, reducing traceability for a security-sensitive control and making incidents harder to audit — add the standard logging instrumentation for this API before merge. - In
packages/core/lib/v3/understudy/context.ts, detach cleanup does not remove domain-policy listeners, so staleFetch.requestPausedhandlers can accumulate and cause memory/perf drift over time;packages/core/lib/v3/understudy/domainPolicy.tsalso throws rawTypeErroron non-stringblockedDomainsentries, andpackages/core/tests/integration/context-domain-policy.spec.tshides navigation failures — fix listener teardown, validate input types up front, and remove the silent test catch to de-risk rollout and debugging.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/core/lib/v3/types/public/context.ts">
<violation number="1" location="packages/core/lib/v3/types/public/context.ts:37">
P2: Custom agent: **Ensure all public methods added to the stagehand class, agent, or understudy (page, locator, etc.) interfaces are properly instrumented with the flowLogger**
New public `setDomainPolicy` method lacks flowLogger instrumentation</violation>
</file>
<file name="packages/core/lib/v3/understudy/context.ts">
<violation number="1" location="packages/core/lib/v3/understudy/context.ts:517">
P2: Domain-policy listener cleanup is ineffective on detach and leaks session event handlers. Detached sessions can accumulate stale `Fetch.requestPaused` closures in the CDP event handler registry.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Dev as Developer
participant Context as V3Context
participant DP as domainPolicy Module
participant CDP as CDP Session (Chrome)
participant Network
Note over Dev,Network: Setting a Domain Blocklist
Dev->>Context: setDomainPolicy({ blockedDomains: [...] })
Context->>DP: normalizeDomainPolicy(policy)
DP->>DP: Validate each domain pattern
alt Invalid domain (URL, path, bad wildcard)
DP-->>Context: throw StagehandInvalidArgumentError
Context-->>Dev: Error thrown
else Valid
DP-->>Context: NormalizedDomainPolicy (rules + fetch patterns)
Context->>Context: Store normalized policy
Context->>CDP: Fetch.enable(patterns)
CDP-->>Context: Acknowledged
Context->>Context: installDomainPolicyHandler() – register Fetch.requestPaused listener
Context-->>Dev: Success
end
Note over Dev,Network: Request from a Page or New Page/Target
Developer->>Context: awaitActivePage() or newPage()
Context->>CDP: (pre-resume) Fetch.enable(patterns) + install listener
CDP-->>Context: Acknowledged
Context-->>Dev: Page ready
Note over Network: A page makes a network request
CDP->>Context: Fetch.requestPaused(event)
Context->>Context: handleDomainPolicyRequestPaused()
Context->>DP: shouldBlockUrl(request.url, policy)
alt URL host matches a blocked rule
DP-->>Context: true
Context->>CDP: Fetch.failRequest(BlockedByClient)
CDP-->>Network: Request blocked (ERR_BLOCKED_BY_CLIENT)
else URL does not match
DP-->>Context: false
Context->>CDP: Fetch.continueRequest()
CDP-->>Network: Request continues normally
end
Note over Dev,Network: Clearing the Policy
Dev->>Context: setDomainPolicy(null) or { blockedDomains: [] }
Context->>DP: normalizeDomainPolicy() → null
Context->>Context: Clear stored policy
Context->>CDP: Fetch.disable()
Context->>Context: uninstallDomainPolicyHandler() – remove listener
CDP-->>Context: Acknowledged
Context-->>Dev: Success (no policy active)
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
Contributor
There was a problem hiding this comment.
1 issue found across 5 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/core/lib/v3/types/public/context.ts">
<violation number="1" location="packages/core/lib/v3/types/public/context.ts:37">
P2: Custom agent: **Ensure all public methods added to the stagehand class, agent, or understudy (page, locator, etc.) interfaces are properly instrumented with the flowLogger**
New public `setDomainPolicy` method lacks flowLogger instrumentation</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
Member
Author
Contributor
|
@seanmcguire12 I have started the AI code review. It will take a few minutes to complete. |
Contributor
There was a problem hiding this comment.
4 issues found across 9 files
Confidence score: 2/5
- In
packages/core/lib/v3/understudy/context.ts, uninstalling theFetch.pausehandler even whenFetch.disablefails can leave paused requests with no handler, creating a real deadlock risk for network traffic if merged as-is — keep the pause handler installed on disable failure and add a failure-path unit/integration check before merging. - In
packages/core/lib/v3/understudy/context.ts, the new public methodsetDomainPolicyappears to be missing requiredflowLoggerinstrumentation, which can break tracing/telemetry consistency for stagehand/agent/understudy public APIs — instrument this method to match existing public interface logging conventions before merge. packages/core/tests/unit/domain-policy.test.tsis missing explicit coverage for deep wildcard subdomain matching and case-insensitive domains, so regressions in*.tracking.example.combehavior or normalization could slip through unnoticed — add focused tests for deep subdomains and mixed-case hostnames to de-risk future changes.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/core/lib/v3/understudy/context.ts">
<violation number="1" location="packages/core/lib/v3/understudy/context.ts:443">
P1: Custom agent: **Ensure all public methods added to the stagehand class, agent, or understudy (page, locator, etc.) interfaces are properly instrumented with the flowLogger**
New public method `setDomainPolicy` is not instrumented with flowLogger</violation>
</file>
Architecture diagram
sequenceDiagram
participant User as User Code
participant Context as V3Context
participant DomainPolicy as domainPolicy.ts
participant Session as CDPSession (per target)
participant Browser as Browser (Chrome)
Note over User,Browser: Setting Domain Policy (Initial Call)
User->>Context: setDomainPolicy({ blockedDomains: ["ads.example.com"] })
Context->>DomainPolicy: normalizeDomainPolicy(policy)
DomainPolicy->>DomainPolicy: Validate domains, reject URLs/wildcards
alt Invalid input
DomainPolicy-->>Context: throw StagehandInvalidArgumentError
Context-->>User: Error
end
DomainPolicy-->>Context: NormalizedDomainPolicy (rules + fetchPatterns)
Context->>Context: Store policy (this.domainPolicy = nextPolicy)
loop For each active session
Context->>Session: Fetch.enable({ patterns: [...] })
alt Success
Context->>Session: installDomainPolicyHandler (Fetch.requestPaused)
Session-->>Context: OK
else Failure (e.g., CDP error)
Context->>Session: uninstallDomainPolicyHandler
Context-->>User: throw StagehandSetDomainPolicyError(failures)
end
end
Context-->>User: void (success)
Note over User,Browser: Request Blocking (Runtime)
Browser->>Browser: HTTP(S) request initiated
Browser->>Session: Fetch.requestPaused({ requestId, request: { url } })
Session->>Context: Handler invoked
Context->>Context: shouldBlockUrl(url, domainPolicy)
alt URL matches a blocked domain
Context->>Browser: Fetch.failRequest(requestId, errorReason="BlockedByClient")
Browser->>Browser: ERR_BLOCKED_BY_CLIENT (asset fails)
else URL does not match
Context->>Browser: Fetch.continueRequest(requestId)
Browser->>Browser: Request proceeds normally
end
Note over User,Browser: New Target / Page Created After Policy Set
User->>Context: awaitActivePage() / newPage()
Context->>Session: Target attached
Context->>Context: onAttachedToTarget()
Context->>Context: queuePreResume("Fetch.enable", { patterns })
alt Fetch.enable succeeds (pre-resume step)
Context->>Session: installDomainPolicyHandler (Fetch.requestPaused)
Context-->>User: Return Page (with blocking active)
else Fetch.enable fails
Context->>Session: uninstallDomainPolicyHandler
Context->>Browser: Target.closeTarget(targetId)
Context->>Context: Store StagehandSetDomainPolicyError in pageCreationFailures
Context-->>User: newPage() throws StagehandSetDomainPolicyError
end
Note over User,Browser: Clearing Policy
User->>Context: setDomainPolicy(null) or { blockedDomains: [] }
Context->>DomainPolicy: normalizeDomainPolicy(null/empty) returns null
Context->>Context: Clear stored policy
loop For each active session
Context->>Session: Fetch.disable()
Context->>Session: uninstallDomainPolicyHandler (removes only its listener)
end
Context-->>User: void
Note over User,Browser: Reading Current Policy
User->>Context: getDomainPolicy()
Context-->>User: DomainPolicy | null (returns copy of stored policy)
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
Contributor
There was a problem hiding this comment.
1 issue found across 9 files
Confidence score: 3/5
- In
packages/core/lib/v3/understudy/context.ts, the new publicsetDomainPolicypath appears to be missingflowLoggerinstrumentation, which can create tracing gaps for this API and make behavior/regressions harder to diagnose after release; wire this method into the sameflowLoggerpattern used by other public stagehand/agent/understudy methods before merging.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/core/lib/v3/understudy/context.ts">
<violation number="1" location="packages/core/lib/v3/understudy/context.ts:443">
P1: Custom agent: **Ensure all public methods added to the stagehand class, agent, or understudy (page, locator, etc.) interfaces are properly instrumented with the flowLogger**
New public method `setDomainPolicy` is not instrumented with flowLogger</violation>
</file>
Architecture diagram
sequenceDiagram
participant User as User Code
participant Ctx as V3Context
participant DP as DomainPolicy
participant Sess as CDPSession
participant CDP as Chrome (Fetch Domain)
Note over User,CDP: NEW: Domain Blocklist via CDP Fetch Interception
User->>Ctx: setDomainPolicy({blockedDomains:[...]})
Ctx->>DP: normalizeDomainPolicy(policy)
alt invalid domain input
DP-->>Ctx: throw StagehandInvalidArgumentError
Ctx-->>User: throw
else valid
DP-->>Ctx: NormalizedDomainPolicy (rules, patterns)
Ctx->>Ctx: store domainPolicy
loop each active session
Ctx->>Sess: installDomainPolicyHandler (attach listener)
Ctx->>Sess: send Fetch.enable({patterns})
Sess->>CDP: Fetch.enable(patterns)
alt success
CDP-->>Sess: ok
else failure
CDP-->>Sess: error
Sess-->>Ctx: error
Ctx->>Sess: uninstallDomainPolicyHandler (remove listener)
Ctx->>Ctx: record failure
end
end
alt any failures
Ctx-->>User: throw StagehandSetDomainPolicyError(failures)
else all success
Ctx-->>User: void
end
end
Note over User,CDP: When a matching request is paused (triggered by Fetch.enable)
CDP-->>Sess: Fetch.requestPaused(event)
Sess->>Ctx: handler
Ctx->>Ctx: get domainPolicy
alt shouldBlockUrl returns true
Ctx->>Sess: send Fetch.failRequest(BlockedByClient)
Sess->>CDP: Fetch.failRequest
else false
Ctx->>Sess: send Fetch.continueRequest
Sess->>CDP: Fetch.continueRequest
end
Note over User,CDP: Clearing policy
User->>Ctx: setDomainPolicy(null) or {blockedDomains:[]}
Ctx->>Ctx: clear stored policy
loop each active session
Ctx->>Sess: send Fetch.disable
Sess->>CDP: Fetch.disable
alt success
Ctx->>Sess: uninstallDomainPolicyHandler
else failure
Sess-->>Ctx: error
Ctx->>Ctx: keep listener, record failure
end
end
alt any failures
Ctx-->>User: throw StagehandSetDomainPolicyError
else all success
Ctx-->>User: void
end
Note over User,CDP: NEW: Policy auto-applied to new pages/targets
User->>Ctx: newPage() (or target attach)
Ctx->>Ctx: onAttachedToTarget
alt domainPolicy is set
Ctx->>Sess: installDomainPolicyHandler
Ctx->>Sess: send Fetch.enable({patterns})
alt success
Ctx->>Ctx: continue normal attach
else failure
Sess-->>Ctx: error
Ctx->>Sess: uninstallDomainPolicyHandler
Ctx->>Ctx: close target
Ctx->>Ctx: record pageCreationFailure
Ctx-->>User: newPage() will throw StagehandSetDomainPolicyError
end
else no policy
Ctx->>Ctx: normal attach (unchanged)
end
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
miguelg719
reviewed
Jun 26, 2026
miguelg719
reviewed
Jun 26, 2026
seanmcguire12
commented
Jun 26, 2026
miguelg719
approved these changes
Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
why
users need a way to block known domains during browser sessions
what changed
stagehand.context.setDomainPolicy({ blockedDomains })to block requests to specific domains across the whole contextFetchrequest patterns, so only matching blocklist requests are paused instead of intercepting all trafficBlockedByClient, which surfaces in chrome as a client-side network blocksetDomainPolicy(null)or{ blockedDomains: [] }; clearing disables policy interception & removes stagehand's request listenerStagehandInvalidArgumentErrorfast follow:
stagehand.context.setDomainPolicy({ allowedDomains })behavioural notes:
setDomainPolicy({ blockedDomains: [...] })applies to active context sessions & future pages/targetsads.example.comblock only that hostname*.example.comblock subdomains, but not the apex domainFetchinterception for this featuresetDomainPolicy(null)or{ blockedDomains: [] }disables the policy & removes stagehand'sFetch.requestPausedlistenertest plan
packages/core/tests/unit/domain-policy.test.tsvalidates domain normalization, exact/wildcard matching, invalid domain rejection & generatedFetch.RequestPatternvaluespackages/core/tests/unit/context-domain-policy.test.tsvalidatescontext.setDomainPolicy()enables/disablesFetch, removes only its ownFetch.requestPausedlistener on clear, fails blocked requests & continues unexpected non-blocked paused requestspackages/core/tests/integration/context-domain-policy.spec.tsvalidates blocked requests fail on an existing page & on a page created after the policy is setpackages/core/tests/unit/public-api/public-error-types.test.tsvalidatesStagehandSetDomainPolicyErroris exported as a public error typeSummary by cubic
Adds a context-wide domain blocklist that intercepts and blocks outgoing HTTP(S) requests by domain using CDP
Fetch. Includes a small API, strict validation, and clearer error messages.context.setDomainPolicy(policy | null)andcontext.getDomainPolicy().example.com,*.example.com); HTTP/HTTPS on any port; case-insensitive; trailing dots handled.nullor[]disables and removes our handler on success.StagehandInvalidArgumentError.BlockedByClient.Fetch.enablefails, we uninstall the handler for that session, close new targets, andnewPage()fails fast withStagehandSetDomainPolicyErrorthat includes per-session details and CDP error text; ifFetch.disablefails, the handler stays installed and the same error is thrown.Written for commit 54c22df. Summary will update on new commits.