fix: update ondemand.s regex for new webpack chunk format#411
Open
fix: update ondemand.s regex for new webpack chunk format#411
Conversation
X changed how the ondemand.s JS bundle is referenced in their HTML. The old format embedded the hash directly next to the chunk name: 'ondemand.s': 'abc123' The new format uses a numeric chunk index that maps to the hash in a separate JavaScript object: ,1234:"ondemand.s" ... ,1234:"abc123" This updates ON_DEMAND_FILE_REGEX to capture the chunk index, adds ON_DEMAND_HASH_PATTERN for the second lookup, and updates INDICES_REGEX to match the new format. The get_indices method now does a two-step lookup: find the chunk index, then resolve it to the file hash. Fixes d60#408 Fixes d60#409
Reviewer's guide (collapsed on small PRs)Reviewer's GuideUpdates the client transaction logic to handle X/Twitter’s new webpack chunk format for the ondemand.s bundle by matching the numeric chunk index, resolving it to the hash, and adapting the index extraction regex accordingly. Sequence diagram for updated get_indices ondemand.s resolutionsequenceDiagram
actor User
participant App
participant ClientTransaction
participant XHomePage as X_home_page
participant AbsTwimg as abs_twid_img
User->>App: Trigger API call (e.g. create_tweet)
App->>ClientTransaction: get_indices(home_page_response, session, headers)
ClientTransaction->>ClientTransaction: validate_response(home_page_response)
ClientTransaction->>ClientTransaction: ON_DEMAND_FILE_REGEX.search(response)
alt ondemand.s chunk index found
ClientTransaction->>ClientTransaction: Extract chunk_index (group 1)
ClientTransaction->>ClientTransaction: re.search(ON_DEMAND_HASH_PATTERN.format(chunk_index), response)
alt hash found
ClientTransaction->>ClientTransaction: Extract file_hash (group 1)
ClientTransaction->>AbsTwimg: GET https://abs.twimg.com/.../ondemand.s.{file_hash}a.js
AbsTwimg-->>ClientTransaction: ondemand.s JS bundle
ClientTransaction->>ClientTransaction: INDICES_REGEX.finditer(js_text)
ClientTransaction->>ClientTransaction: Collect key_byte_indices (group 1)
else hash not found
ClientTransaction->>ClientTransaction: key_byte_indices remains empty
end
else ondemand.s chunk index not found
ClientTransaction->>ClientTransaction: key_byte_indices remains empty
end
ClientTransaction->>ClientTransaction: if not key_byte_indices: raise Exception
ClientTransaction-->>App: key_byte_indices (list[int])
App-->>User: Continue request flow using KEY_BYTE indices
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
📝 WalkthroughWalkthroughUpdated regex patterns and parsing logic in Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The new
ON_DEMAND_HASH_PATTERNonly matches double-quoted hashes (,{}:"([0-9a-f]+)"), whereasON_DEMAND_FILE_REGEXallows both single and double quotes; consider making the hash pattern consistent so it continues to work if the HTML switches to single quotes for hashes. - Since you're now using a two-step lookup by chunk index, it may be safer to tighten both regexes with clearer boundaries (e.g., ensuring the preceding character is not a digit and the following character is not part of a longer identifier) to avoid accidentally matching unrelated
,1234:or,[N], 16sequences in other parts of the page.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The new `ON_DEMAND_HASH_PATTERN` only matches double-quoted hashes (`,{}:"([0-9a-f]+)"`), whereas `ON_DEMAND_FILE_REGEX` allows both single and double quotes; consider making the hash pattern consistent so it continues to work if the HTML switches to single quotes for hashes.
- Since you're now using a two-step lookup by chunk index, it may be safer to tighten both regexes with clearer boundaries (e.g., ensuring the preceding character is not a digit and the following character is not part of a longer identifier) to avoid accidentally matching unrelated `,1234:` or `,[N], 16` sequences in other parts of the page.
## Individual Comments
### Comment 1
<location path="twikit/x_client_transaction/transaction.py" line_range="17-21" />
<code_context>
-INDICES_REGEX = re.compile(
- r"""(\(\w{1}\[(\d{1,2})\],\s*16\))+""", flags=(re.VERBOSE | re.MULTILINE))
+ r',(\d+):["\']ondemand\.s["\']', flags=(re.VERBOSE | re.MULTILINE))
+ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"'
+INDICES_REGEX = re.compile(r'\[(\d+)\],\s*16')
</code_context>
<issue_to_address>
**issue:** Hash pattern only matches double-quoted hashes while the ondemand.s key allows both quote styles.
Because `ON_DEMAND_HASH_PATTERN` only matches double-quoted hashes, `re.search` will fail whenever the hash is single-quoted (e.g. `,123:'abcd'`), so `on_demand_file_url` is never constructed and you hit `Couldn't get KEY_BYTE indices`. To avoid this, update the hash pattern to accept both quote styles, e.g. `r',{}:["\']([0-9a-f]+)["\']'` so it stays consistent with `ON_DEMAND_FILE_REGEX`.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+17
to
21
| ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"' | ||
| INDICES_REGEX = re.compile(r'\[(\d+)\],\s*16') | ||
|
|
||
|
|
||
| class ClientTransaction: |
There was a problem hiding this comment.
issue: Hash pattern only matches double-quoted hashes while the ondemand.s key allows both quote styles.
Because ON_DEMAND_HASH_PATTERN only matches double-quoted hashes, re.search will fail whenever the hash is single-quoted (e.g. ,123:'abcd'), so on_demand_file_url is never constructed and you hit Couldn't get KEY_BYTE indices. To avoid this, update the hash pattern to accept both quote styles, e.g. r',{}:["\']([0-9a-f]+)["\']' so it stays consistent with ON_DEMAND_FILE_REGEX.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@twikit/x_client_transaction/transaction.py`:
- Around line 15-18: The ON_DEMAND_HASH_PATTERN currently only matches
double-quoted hashes which can miss single-quoted webpack outputs; update
ON_DEMAND_HASH_PATTERN to accept either single or double quotes around the hash
(similar to ON_DEMAND_FILE_REGEX) so both quote styles are handled
consistently—modify the pattern referenced as ON_DEMAND_HASH_PATTERN to use a
quote-capturing alternative like ["'] around the hash value and ensure any
backreference logic matches that capture.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 7ec057d6-1634-405a-b9e7-e7769470e479
📒 Files selected for processing (1)
twikit/x_client_transaction/transaction.py
Comment on lines
15
to
+18
| ON_DEMAND_FILE_REGEX = re.compile( | ||
| r"""['|\"]{1}ondemand\.s['|\"]{1}:\s*['|\"]{1}([\w]*)['|\"]{1}""", flags=(re.VERBOSE | re.MULTILINE)) | ||
| INDICES_REGEX = re.compile( | ||
| r"""(\(\w{1}\[(\d{1,2})\],\s*16\))+""", flags=(re.VERBOSE | re.MULTILINE)) | ||
| r',(\d+):["\']ondemand\.s["\']', flags=(re.VERBOSE | re.MULTILINE)) | ||
| ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"' | ||
| INDICES_REGEX = re.compile(r'\[(\d+)\],\s*16') |
Contributor
There was a problem hiding this comment.
Quote character inconsistency between regex patterns.
ON_DEMAND_FILE_REGEX correctly handles both single and double quotes around ondemand.s, but ON_DEMAND_HASH_PATTERN only matches double quotes around the hash value. If the webpack output uses single quotes for hashes (as it might for chunk names), the second lookup will fail silently.
Proposed fix to handle both quote types
-ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"'
+ON_DEMAND_HASH_PATTERN = r',{}:["\']([0-9a-f]+)["\']'📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ON_DEMAND_FILE_REGEX = re.compile( | |
| r"""['|\"]{1}ondemand\.s['|\"]{1}:\s*['|\"]{1}([\w]*)['|\"]{1}""", flags=(re.VERBOSE | re.MULTILINE)) | |
| INDICES_REGEX = re.compile( | |
| r"""(\(\w{1}\[(\d{1,2})\],\s*16\))+""", flags=(re.VERBOSE | re.MULTILINE)) | |
| r',(\d+):["\']ondemand\.s["\']', flags=(re.VERBOSE | re.MULTILINE)) | |
| ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"' | |
| INDICES_REGEX = re.compile(r'\[(\d+)\],\s*16') | |
| ON_DEMAND_FILE_REGEX = re.compile( | |
| r',(\d+):["\']ondemand\.s["\']', flags=(re.VERBOSE | re.MULTILINE)) | |
| ON_DEMAND_HASH_PATTERN = r',{}:["\']([0-9a-f]+)["\']' | |
| INDICES_REGEX = re.compile(r'\[(\d+)\],\s*16') |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@twikit/x_client_transaction/transaction.py` around lines 15 - 18, The
ON_DEMAND_HASH_PATTERN currently only matches double-quoted hashes which can
miss single-quoted webpack outputs; update ON_DEMAND_HASH_PATTERN to accept
either single or double quotes around the hash (similar to ON_DEMAND_FILE_REGEX)
so both quote styles are handled consistently—modify the pattern referenced as
ON_DEMAND_HASH_PATTERN to use a quote-capturing alternative like ["'] around the
hash value and ensure any backreference logic matches that capture.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
X changed how the
ondemand.sJS bundle is referenced in their HTML (around March 18, 2026). The old webpack format embedded the hash directly next to the chunk name:The new format uses a numeric chunk index that maps to the hash in a separate JavaScript object:
This causes
ON_DEMAND_FILE_REGEXto fail, which meansget_indicescan't find theKEY_BYTEindices, raisingCouldn't get KEY_BYTE indicesfor every API call.Fix
Three changes in
transaction.py:ON_DEMAND_FILE_REGEX— now captures the numeric chunk index instead of the hash directlyON_DEMAND_HASH_PATTERN(new) — uses the chunk index to find the actual hash in a second passINDICES_REGEX— updated to match the new JS format ([N], 16instead of(a[N], 16))get_indices— two-step lookup: find chunk index → resolve to file hash → fetch JS → extract indicesTesting
Tested and confirmed working as of 2026-03-26 in a tweet scheduling app that uses
set_cookies+create_tweet+upload_media.Fixes #408
Fixes #409
Summary by Sourcery
Update X client transaction logic to handle the new webpack chunk format for the ondemand.s bundle so KEY_BYTE indices can be resolved again.
Bug Fixes:
Summary by CodeRabbit