Initial work on upgrading all the dependencies

[rahulbot]

I've gone through one by one and upgraded each dependency to the latest. After each I ran all the unit tests, and used the new scripts/evaluate-against-warc.py to look for differences in metadata results against a recent WARC file with about 700 stories. That script uses a prior warc to compare metadata results when running the updated extraction code against the exact same URL and HTML content.

I did Trafilatura last (#105), and it creates errors that look like this:

ERROR:main:fail urn:uuid:95b7e3bf-75fa-410b-80ae-7419b31ce334: 'text_content' 1445 != 998

These indicate that there is a difference of > 20% in the length of the extracted content. Overall, the result is "Done with 138 fails and 569 passes (707 total stories)". The vast majority are failures indicating a noticeable difference in text_content length like this, however only a few are in English so it is hard to debug. ⚠️ This will require some manual review to understand if content is being extracted differently in any kind of important way across those 20% of stories ⚠️.

Besides a potential data quality difference, changes like this might impact performance. It looks like our scripts/generate_stats.py is one way to assess this. Here's the run pre and post changes. This doesn't look worrisome to me as a first comparison.

Stat	Pre-Upgrade	Post-Upgrade
total	60.78543329110835 (100.000%)	89.78205291822087 (100.000%)
fetch	55.01345703797415 (90.504%)	84.53771079541184 (94.159%)
url	0.14894583635032177 (0.245%)	0.1607392963487655 (0.179%)
pub_date	0.8124426644062623 (1.337%)	0.6352829126408324 (0.708%)
content	5.909443545155227 (9.722%)	6.567109709139913 (7.315%)
title	0.017112291418015957 (0.028%)	0.012375917751342058 (0.014%)
language	0.13541033363435417 (0.223%)	0.15343421092256904 (0.171%)

[rahulbot]

Full report, after rebasing to verify unit tests and such still work:

INFO:main:Done with 138 fails and 569 passes (707 total stories)

eval-run-1.txt