Conversation
Introduce a new version of the VoiceOfAmerica parser with improved content extraction. - Define V1_1 parser with distinct paragraph and subheadline selectors - Update filter configuration for reverse ordering in sitemaps - Enhance handling of gzip content types in URL scraper - Add comprehensive VOA test data for validation and new articles
addie9800
changed the title
VOAParser V1_1 with enhanced selectors
MaxDall
left a comment
MaxDall
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES — V1_1 still returns an empty body on VOA's featured-article layout.
us.VoiceOfAmerica: read 10 of 100 scanned (12 flagged, 8 in draw); layouts, over-capture and image attributes checked, plus the sort_key refactor and both of its call sites against the live sitemap indices. 1 blocker + 1 nit inline.
Not inline:
- 6 of the 100 scanned URLs are audio/video programme pages ("The Issue", "International Edition", "Issues in the News"): no
#article-content, no prose anywhere, just adiv#player<id>. Empty body is correct there and the defaultonly_complete=Truecrawl drops them — not a finding. - The 4 flagged articles outside the draw carry only player and "also read" chrome (
<ul class='subitems'>download sizes,<h4 class='…also-read__text'>,<ul class='items'>), the same signatures adjudicated benign inside the draw. - Layout coverage: straight news, a long feature, a multimedia feature and a one-paragraph video story. No opinion/column or listicle reached the draw — the newest URLs the sitemap index offers are from March 2025, so the pool is a narrow slice of the site.
| ) | ||
|
|
||
| class V1_1(V1): | ||
| _paragraph_selector = XPath("//div[@id='article-content']/div/p[not(strong)]") |
There was a problem hiding this comment.
Blocker — V1_1 returns an empty body on VOA's featured-article layout. There the prose sits at #article-content > div.wsw > div.fa-container > p, one level deeper than this selector's /div/p, so all 36 <p> are dropped and body comes back empty: "The state of Texas has the third-largest Asian American population in the United States, according to the U.S. census…" 1 of 100 scanned URLs; V1 misses it too, but this PR rewrites exactly this selector.
Fix: //div[@id='article-content']/div[@class='wsw']//p[not(strong) and not(@class)] (and …//p[strong] for the subheadline) — checked against the whole draw and the new fixture: identical counts everywhere V1_1 already works, 32 paragraphs + 4 subheadlines recovered here, and not(@class) keeps the ta-c / link-content-sharing chrome out. [1]
| class VOAParser(ParserProxy): | ||
| class V1(BaseParser): | ||
| _paragraph_selector = CSSSelector("#article-content > div > p") | ||
| VALID_UNTIL = datetime.date(2026, 7, 20) |
There was a problem hiding this comment.
Nit — this VALID_UNTIL marks a layout change the HTML doesn't show. On current pages V1 and V1_1 extract the same text to the character (on the new fixture both 11940 chars); V1_1 only re-labels the <p><strong> headings as subheadlines instead of paragraphs. So V1 was never "no longer able to extract articles properly" (how_to_add_a_publisher.md), and nothing observable picks 2026-07-20. The date is structurally required — two versions at date.max raise in ParserProxy — so could the PR just say what it's based on?
There was a problem hiding this comment.
Before your other comment, you were right, that I could have just added subheadline support for V1, but now it was also unable to parse the featured articles.
2 participants
Introduce a new version of the VoiceOfAmerica parser with improved content extraction.