Initial work on upgrading all the dependencies - #110

Draft

rahulbot wants to merge 33 commits into

mainfrom

evaluate-dependency-upgrade

Commits on Aug 25, 2026

script to compare results in old warc vs. current code #105
rahulbot
committed
66c789b
compare extracted text as length, with wiggle room
rahulbot
committed
c6f66aa
upgrade htmldate and dateparser (no change in evaluation results)
rahulbot
committed
288ea73
upgrade URL dependencies (to test)
rahulbot
committed
294cac7
update py3langid; fix test_utm_removal unit tests to match bugfix in url_normalize
rahulbot
committed
f5ee701
update faust-cchardet; all unit tests and evaluates same against old warc
rahulbot
committed
e181446
add extractor check, also AI-suggested fix to let me run tests in PyCharm
rahulbot
committed
1b65141
upgrade to latest trafilatura for testing
rahulbot
committed
1911b80
minor unit test fix (hand-checked and its ok)
rahulbot
committed
4d78ab1
better checking and logging around eval of text_content changes
rahulbot
committed
650acbc
remove redundant id from error log
rahulbot
committed
f6fab2d
tldextract upgrade: from registered_domain -> top_domain_under_public_suffix
rahulbot
committed
8f72152
script to compare results in old warc vs. current code #105
rahulbot
committed
87a9640
better checking and logging around eval of text_content changes
rahulbot
committed
c92d6b6

Commits on Aug 26, 2026

update authors list (fix #98)
rahulbot
committed
2e300d5
extract final_url logic to it's own function for testing
rahulbot
committed
2e0845a
redo unit test fixtures with requests-mock, and fix more tests
rahulbot
committed
63b12f7

Commits on Aug 31, 2026