Skip to content

fix: narrow pinterest bot-detection entry to avoid blocking Pinterest's in-app browser - #4636

Open
yfwmaniish wants to merge 2 commits into
PostHog:mainfrom
yfwmaniish:fix/pinterest-bot-detection-false-positive
Open

fix: narrow pinterest bot-detection entry to avoid blocking Pinterest's in-app browser#4636
yfwmaniish wants to merge 2 commits into
PostHog:mainfrom
yfwmaniish:fix/pinterest-bot-detection-false-positive

Conversation

@yfwmaniish

@yfwmaniish yfwmaniish commented Aug 25, 2026

Copy link
Copy Markdown

What

Fixes a bot-detection false positive reported in #83413: Pinterest's own in-app browser gets misclassified as the Pinterestbot crawler and silently excluded from analytics.

Root cause

DEFAULT_BLOCKED_UA_STRS (packages/core/src/utils/bot-detection.ts) contains the bare substring 'pinterest'. isBlockedUA does a case-insensitive substring match of every blocklist entry against the whole user-agent string:

const uaLower = ua.toLowerCase()
return DEFAULT_BLOCKED_UA_STRS.concat(customBlockedUserAgents).some((blockedUA) => {
    const blockedUaLower = blockedUA.toLowerCase()
    return uaLower.indexOf(blockedUaLower) !== -1
})

Pinterest's own iOS/Android app in-app browser identifies itself with a UA like:

Mozilla/5.0 (iPad; CPU OS 11_2_6 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Mobile/15D100 [Pinterest/iOS]

— a real human user, not the crawler — but it contains the substring pinterest, so it gets blocked exactly like the actual bot.

Fix

Narrowed the entry to 'pinterestbot', which still matches the crawler's primary UA ("...Pinterestbot/1.0; +http://www.pinterest.com/bot.html)") but not the in-app browser's. The crawler's other known UA variant ("Pinterest/0.2 (+https://www.pinterest.com/bot.html)") doesn't contain "pinterestbot" as a contiguous substring, but stays covered by the pre-existing generic 'bot.htm' entry a few lines down — so no bot-detection coverage is lost, only the false positive is removed.

Scope note

#83413 also reports that several other in-app browsers (Instagram, LinkedIn, TikTok, WeChat, etc.) aren't detected at all, across three separate implementations (this repo, posthog-js-lite, and a CDP transformation in the main posthog repo) — a much larger piece of work already partially underway via a draft PR (posthog-js#4510). This PR only fixes the one narrow false-positive substring-match bug described above; it isn't a fix for the broader detection gap, so it doesn't use a Fixes keyword for #83413 in the commit (it should stay open for the rest of that work).

How did you test this code?

Added a case to each existing it.each block in packages/browser/src/__tests__/utils.test.ts:

  • the crawler's Pinterestbot/1.0 UA still gets blocked
  • the in-app browser's [Pinterest/iOS] UA no longer does

Verified by executing the actual (copy-pasted, unmodified) isBlockedUA logic and the fixed DEFAULT_BLOCKED_UA_STRS array directly in plain Node against three UAs -- the crawler's two known variants (both still blocked, one via the new pinterestbot entry, one via the pre-existing bot.htm entry) and the in-app browser (no longer blocked). The actual jest suite could not be run locally (pnpm --filter @posthog/core build fails on this Windows machine because @rspack/binding-win32-x64-msvc is missing from the pnpm store -- present for every other platform, seemingly unrelated to this change), so please treat CI as the first real run of the added test cases.

🤖 Agent context

Autonomy: Fully autonomous

  • Tool: Claude Code (Anthropic), operating unsupervised on an open-ended "find and fix issues" session against the repo owner's fork. No human directed this specific fix.
  • Skills invoked: none.
  • The narrower fix (vs. attempting the full multi-repo in-app-browser detection gap from #83413) was a deliberate scope decision after finding the existing draft PR posthog-js#4510 already covers that broader work.
  • Local verification was done by direct Node execution of the extracted logic, not a full local test run, because of the Windows-specific rspack binary gap noted above -- disclosed so review treats CI as the first real run of the added tests.
  • No customer data, tickets, or internal PostHog information was used; all reasoning is from public repository source and the public linked issue.

@yfwmaniish
yfwmaniish requested a review from a team as a code owner August 25, 2026 10:19

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @yfwmaniish, thanks for this pull request! the change looks good. do you mind signing your commit(s), it's a requirement to merge to main in this repo.

also, please see RELEASING.md - this change will require a changeset 🙌

@yfwmaniish

Copy link
Copy Markdown
Author

Thanks for the review! Added a changeset (.changeset/narrow-pinterest-bot.md) bumping @posthog/core, @posthog/browser-common, posthog-js, and posthog-node -- all four consume the shared bot-detection blocklist this touches.

Re: commit signing -- I don't have a GPG/SSH signing key configured in this environment yet, so I can't sign the existing commits right now. Will follow up once that's set up on my end.

@dustinbyrne

dustinbyrne commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

ah, fully autonomous - i can rebase on your behalf for commit signing if you'd prefer

thanks for updating

@yfwmaniish

Copy link
Copy Markdown
Author

That would be great, thank you -- please go ahead.

…'s in-app browser

DEFAULT_BLOCKED_UA_STRS matched the bare substring 'pinterest', which
also matches the Pinterest app's own in-app browser UA (e.g. "...
Mobile/15D100 [Pinterest/iOS]"). isBlockedUA does a case-insensitive
substring match against the whole UA string, so real Pinterest-app
users were silently classified as bots and excluded from analytics.

Narrowed the entry to 'pinterestbot', which still matches the crawler's
primary UA ("...Pinterestbot/1.0; +http://www.pinterest.com/bot.html)")
without matching the in-app browser. The crawler's other UA variant
("Pinterest/0.2 (+https://www.pinterest.com/bot.html)") remains covered
by the existing generic 'bot.htm' entry, so no bot-detection coverage
is lost.

Addresses one specific sub-bug reported in #83413 (the pinterest
false-positive) -- not the broader per-platform UA-detection gap in
that issue (Instagram/LinkedIn/TikTok/etc.), which spans multiple
repos and implementations and needs separate, larger work.
Requested by @dustinbyrne in PR review -- RELEASING.md requires a
changeset for any change that will be published.
@dustinbyrne
dustinbyrne force-pushed the fix/pinterest-bot-detection-false-positive branch from 608d635 to d700c38 Compare August 26, 2026 16:51
@dustinbyrne

Copy link
Copy Markdown
Contributor

can you just double check / fix the author attribution? it's not going to you

3ddf4e1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants