Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion src/data.jl
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,17 @@ far shorter than PubChem's throttling window (it answers `503` while overloaded)
by default a stalled connection is never timed out. Requests therefore carry
connect/read-idle timeouts and are retried with a multi-second exponential backoff on
any failure except a definitive (non-`408`/`429`) `4xx` response.

`protocol = :h1` avoids HTTP.jl's HTTP/2 client, whose receive flow-control
accounting is buggy at this package's compat floor: HTTP 2.3.1 (and through
at least 2.6.7) never returns connection-level window credit for response
bytes that arrive but are never read β€” the refund is present by 2.7.1 β€” so
one throttle-induced mid-body failure permanently stalls every later request
on the pooled connection (JuliaWeb/HTTP.jl#1371). These calls are sequential
REST GETs, where HTTP/2 multiplexing gains nothing anyway.
"""
function pubchem_get(url; kwargs...)
get_once = () -> HTTP.get(url; connect_timeout = 30, read_idle_timeout = 120, kwargs...)
get_once = () -> HTTP.get(url; connect_timeout = 30, read_idle_timeout = 120, protocol = :h1, kwargs...)
delays = ExponentialBackOff(; n = 6, first_delay = 2.0, max_delay = 60.0)
return retry(get_once; delays, check = (_, e) -> is_transient_http_failure(e))()
end
Expand Down
Loading