fix: narrow pinterest bot-detection entry to avoid blocking Pinterest's in-app browser - #4636
Conversation
dustinbyrne
left a comment
There was a problem hiding this comment.
hey @yfwmaniish, thanks for this pull request! the change looks good. do you mind signing your commit(s), it's a requirement to merge to main in this repo.
also, please see RELEASING.md - this change will require a changeset 🙌
|
Thanks for the review! Added a changeset ( Re: commit signing -- I don't have a GPG/SSH signing key configured in this environment yet, so I can't sign the existing commits right now. Will follow up once that's set up on my end. |
|
ah, fully autonomous - i can rebase on your behalf for commit signing if you'd prefer thanks for updating |
|
That would be great, thank you -- please go ahead. |
…'s in-app browser
DEFAULT_BLOCKED_UA_STRS matched the bare substring 'pinterest', which
also matches the Pinterest app's own in-app browser UA (e.g. "...
Mobile/15D100 [Pinterest/iOS]"). isBlockedUA does a case-insensitive
substring match against the whole UA string, so real Pinterest-app
users were silently classified as bots and excluded from analytics.
Narrowed the entry to 'pinterestbot', which still matches the crawler's
primary UA ("...Pinterestbot/1.0; +http://www.pinterest.com/bot.html)")
without matching the in-app browser. The crawler's other UA variant
("Pinterest/0.2 (+https://www.pinterest.com/bot.html)") remains covered
by the existing generic 'bot.htm' entry, so no bot-detection coverage
is lost.
Addresses one specific sub-bug reported in #83413 (the pinterest
false-positive) -- not the broader per-platform UA-detection gap in
that issue (Instagram/LinkedIn/TikTok/etc.), which spans multiple
repos and implementations and needs separate, larger work.
Requested by @dustinbyrne in PR review -- RELEASING.md requires a changeset for any change that will be published.
608d635 to
d700c38
Compare
|
can you just double check / fix the author attribution? it's not going to you |
What
Fixes a bot-detection false positive reported in #83413: Pinterest's own in-app browser gets misclassified as the Pinterestbot crawler and silently excluded from analytics.
Root cause
DEFAULT_BLOCKED_UA_STRS(packages/core/src/utils/bot-detection.ts) contains the bare substring'pinterest'.isBlockedUAdoes a case-insensitive substring match of every blocklist entry against the whole user-agent string:Pinterest's own iOS/Android app in-app browser identifies itself with a UA like:
— a real human user, not the crawler — but it contains the substring
pinterest, so it gets blocked exactly like the actual bot.Fix
Narrowed the entry to
'pinterestbot', which still matches the crawler's primary UA ("...Pinterestbot/1.0; +http://www.pinterest.com/bot.html)") but not the in-app browser's. The crawler's other known UA variant ("Pinterest/0.2 (+https://www.pinterest.com/bot.html)") doesn't contain "pinterestbot" as a contiguous substring, but stays covered by the pre-existing generic'bot.htm'entry a few lines down — so no bot-detection coverage is lost, only the false positive is removed.Scope note
#83413 also reports that several other in-app browsers (Instagram, LinkedIn, TikTok, WeChat, etc.) aren't detected at all, across three separate implementations (this repo, posthog-js-lite, and a CDP transformation in the main posthog repo) — a much larger piece of work already partially underway via a draft PR (posthog-js#4510). This PR only fixes the one narrow false-positive substring-match bug described above; it isn't a fix for the broader detection gap, so it doesn't use a
Fixeskeyword for #83413 in the commit (it should stay open for the rest of that work).How did you test this code?
Added a case to each existing
it.eachblock inpackages/browser/src/__tests__/utils.test.ts:Pinterestbot/1.0UA still gets blocked[Pinterest/iOS]UA no longer doesVerified by executing the actual (copy-pasted, unmodified)
isBlockedUAlogic and the fixedDEFAULT_BLOCKED_UA_STRSarray directly in plain Node against three UAs -- the crawler's two known variants (both still blocked, one via the newpinterestbotentry, one via the pre-existingbot.htmentry) and the in-app browser (no longer blocked). The actual jest suite could not be run locally (pnpm --filter @posthog/core buildfails on this Windows machine because@rspack/binding-win32-x64-msvcis missing from the pnpm store -- present for every other platform, seemingly unrelated to this change), so please treat CI as the first real run of the added test cases.🤖 Agent context
Autonomy: Fully autonomous