Skip to content

feat(publisher): Introduce V1_1 for MexicoNewsDaily #970

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
addie9800 wants to merge 2 commits into
base: master
Choose a base branch
from
Open

feat(publisher): Introduce V1_1 for MexicoNewsDaily #970

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9bab9e6
feat(publisher): Introduce `V1_1` for `MexicoNewsDaily`
addie9800 Aug 18, 2026
aa692e5
fix(publisher): update `MexicoNewsDaily` V1_1 paragraph selector to i…
addie9800 Aug 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions src/fundus/publishers/mx/mexico_news_daily.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@

class MexicoNewsDailyParser(ParserProxy):
class V1(BaseParser):
VALID_UNTIL = datetime.date(2026, 7, 27)

_paragraph_selector = XPath("//div[@class='tdb-block-inner td-fix-index']/p[text()] ")

_bloat_topics = {"editors_pick"}
Expand Down Expand Up @@ -55,3 +57,6 @@ def images(self) -> List[Image]:
upper_boundary_selector=XPath("//h1"),
author_selector=re.compile(r"\((?P<credits>.*?)\)\s*$"),
)

class V1_1(V1):
_paragraph_selector = XPath("//div[@class='tts_content_wrapper_1']/p[text()]")

@MaxDall MaxDall Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker — p[text()] drops any <p> whose text sits inside <em>/<a>, so real prose is lost. “…UPDATE June 20, 4:16pm CDT: FIFA announced a fine of 10,000 Swiss francs (US $10,000) against the Mexican Football Federation…” never reaches the body [1]; a CORRECTION: paragraph goes the same way [2], and all 100 pooled articles show an uncaptured <p> (mostly the Source: … trailer). Fix: /p[normalize-space()], plus a bloat filter for the Source:/subscribe trailers, which would otherwise leak in.

83 changes: 83 additions & 0 deletions tests/resources/parser/test_data/mx/MexicoNewsDaily.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,5 +212,88 @@
"steel aluminum tariffs mexico",
"steel trade us-mexico"
]
},
"V1_1": {
"authors": [
"César Hernández"
],
"body": {
"summary": [],
"sections": [
{
"headline": [],
"paragraphs": [
"Human error caused a head-on collision by two trains in Chihuahua yesterday that killed a conductor and injured five other crew members, Ferromex officials said.",
"The trains collided about 1:30am on a stretch of track between the cities of Chihuahua and Ciudad Juárez. The conductor of the northbound train died, while five crew members from both trains were injured.",
"The collision sparked a fire that could be seen for several kilometers.",
"Investigations by the rail consortium indicated that the conductor of the southbound train erred by exceeding the authorized indicated speed limit and was running ahead of time.",
"Ferromex, with its subsidiary Ferrosur, is a private rail consortium that operates the largest railway network in Mexico with combined length of 12,070 kilometers."
]
}
]
},
"images": [
{
"versions": [
{
"url": "https://mexiconewsdaily.com/wp-content/uploads/2018/05/crash-1-150x92.jpg",
"query_width": null,
"size": {
"width": 150,
"height": 92
},
"type": "image/jpeg"
},
{
"url": "https://mexiconewsdaily.com/wp-content/uploads/2018/05/crash-1-300x184.jpg",
"query_width": null,
"size": {
"width": 300,
"height": 184
},
"type": "image/jpeg"
},
{
"url": "https://mexiconewsdaily.com/wp-content/uploads/2018/05/crash-1-696x427.jpg",
"query_width": null,
"size": {
"width": 696,
"height": 427
},
"type": "image/jpeg"
},
{
"url": "https://mexiconewsdaily.com/wp-content/uploads/2018/05/crash-1-768x471.jpg",
"query_width": null,
"size": {
"width": 768,
"height": 471
},
"type": "image/jpeg"
},
{
"url": "https://mexiconewsdaily.com/wp-content/uploads/2018/05/crash-1.jpg",
"query_width": null,
"size": {
"width": 850,
"height": 521
},
"type": "image/jpeg"
}
],
"is_cover": true,
"description": "The train wreck in Chihuahua yesterday.",
"caption": "The train wreck in Chihuahua yesterday.",
"authors": [],
"position": 804
}
],
"publishing_date": "2018-05-31 16:04:54+00:00",
"title": "One dead in Chihuahua train crash; human error blamed",
"topics": [
"ferromex",
"railway",
"train accident"
]
}
}
Binary file added tests/resources/parser/test_data/mx/MexicoNewsDaily_2026_08_18.html.gz
View file
Open in desktop
Binary file not shown.
4 changes: 4 additions & 0 deletions tests/resources/parser/test_data/mx/meta.info
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,9 @@
"MexicoNewsDaily_2025_04_16.html.gz": {
"url": "https://mexiconewsdaily.com/news/bad-idea-mexico-pushes-back-trump-universal-steel-tariffs/",
"crawl_date": "2025-04-16 14:10:08.579331"
},
"MexicoNewsDaily_2026_08_18.html.gz": {
"url": "https://mexiconewsdaily.com/news/one-dead-in-train-crash-human-error-blamed/",
"crawl_date": "2026-08-18 22:38:32.846902"
}
}