fix(hub): strengthen retries for bucket metadata gateway errors - #215
Open
arekborucki wants to merge 9 commits into
Open
fix(hub): strengthen retries for bucket metadata gateway errors#215arekborucki wants to merge 9 commits into
arekborucki wants to merge 9 commits into
Conversation
Hub bucket tree/HEAD calls already retried 504s, but only twice with short backoff — not enough to ride out MongoDB config-server elections that stall metadata for tens of seconds. - Raise default max retries from 2 to 4 (5 total attempts) - Use longer 1s/2s/4s/8s backoff on 502/503/504 - Add --hub-max-retries and --hub-request-timeout-secs CLI flags - Reuse cached directory listings when list_tree fails transiently after poll invalidation (read-only mounts keep serving docs) Co-authored-by: Cursor <cursoragent@cursor.com>
Revert aggressive 5-try/15s gateway backoff in favor of fail-fast within a 5s operation budget, full jitter on retries, and a short-lived circuit breaker after consecutive 502/503/504. Extend stale metadata fallback to HEAD lookups and mark stale listings without faking fresh children_loaded_at. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Only populate negative dentry cache from fresh listings (children_loaded), not stale snapshots. Escalate stale list_tree retry interval 5s→10s→30s. Default hub operation deadline 3s (fail fast vs server Mongo timeout). Gate HEAD lookup stale fallback on read-only mounts. Check HTTP deadline before each attempt, not only before backoff sleep. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Wrap each send() in tokio::time::timeout(remaining) so the 3s operation budget applies to hung requests, not just between retries. Reset circuit failure counter when the cooldown expires (half-open). Allow stale fallback for empty remote-backed directories. Fix operation_deadline_ms doc comment. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…504. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
During the 2026-08-09 docs outage,
hf-mountsidecars onhub-dochit repeated 504 responses from/api/buckets/.../treewhilemongodb-moonlanding-bucket-prodconfig servers were electing. The client already retried transient Hub errors, but without a total operation budget or stale-metadata fallback, FUSE lookups could hang long enough to trip Better Uptime (30s).This PR hardens the client side of that path:
--hub-operation-deadline-ms, default 3000ms): wall-clock budget for the whole operation. Each in-flightsend()is wrapped intokio::time::timeout(remaining)so hung requests cannot ignore the budget via the 60s/30s reqwest client timeouts. Intentionally shorter than the Hub's Mongo tree-listing deadline.list_treefails transiently but a cached remote listing exists, serve it viastale_listing_since(does not resetchildren_loaded_at). Includes empty directories (children_from_remotewithout requiring non-emptychildren).list_treeretries per directory during outageschildren_loaded()listings, not stale snapshots--hub-max-retries,--hub-operation-deadline-ms,--hub-request-timeout-secs,--hub-head-request-timeout-secsContext
Incident chain: cfg election → bucketDb metadata ~1.3s → Hub
timeoutListingsMs(~5s) → 504 → FUSE EIO →/docs/*slow/down.Server-side fixes (moon-landing deadline/retry, Mongo pacing) are complementary; this protects all hf-mount consumers (docs, Spaces volumes, etc.).
Deploy caveat
Stale metadata fallback is in-memory only. Rolling
hub-docpods (required to pick up a new hf-mount image) clears any warm listing cache — freshly restarted pods have no stale fallback until they have successfully listed the tree at least once. Plan deploy accordingly (e.g. avoid simultaneous mass restart during known bucket maintenance, or accept a brief cold-start window).Deploy path after merge
hf-mountrelease (or use commit SHA)hfMount.image.tagininfra/projects/hub/05-eks-utils/hf_csi_driver.tf(currentlyv0.9.0)hub-docpods (see caveat above)Test plan
cargo fmt --check,cargo test --lib, integration tests)hf-doc-build/docread-only, simulate 504 on tree endpoint, verify stale cache serves and negative cache is not populated from stale listinghub-docpod, confirm docs load during induced bucket metadata latency