feat: fall back to GCS mirror when GitHub source is unavailable - #34
Merged
Merged
Conversation
Add https://storage.googleapis.com/token-directory-index/index as a fallback source for index.json and token lists. The primary GitHub raw URL remains the logical identity (TokenListURL, cache keys); the mirror is only a transport fallback. - fetchFromURLs tries URLs in order, returning the first 200 OK; falls back on 429/5xx, network errors, and stalled sources - per-attempt timeout (10s) so a stalled primary still leaves budget for the mirror; default client now has a 30s timeout instead of http.DefaultClient's none - ErrSourceTimeout sentinel distinguishes slow sources from the caller's own deadline expiring - errors.Join keeps every source's failure in the returned error - tests cover 429/5xx/network-error/stall/cancel paths, request path assertions on the mirror, and the URL-rewrite prefix semantics
…tion - preferFallback: after a primary failure, token-list fetches go straight to the GCS mirror instead of paying a per-file primary timeout; index refreshes always probe the primary first so recovery is detected within one 30s index cycle. Caller cancellation does not mark the primary unavailable. - response validation: 200 responses are validated (JSON unmarshal + structural check) before acceptance, so a garbage body triggers the next source instead of failing the fetch - content-hash verification: fetched token lists are checked against the index ContentHash; a stale source is rejected and the other source is tried (index and lists are updated atomically, so a mismatch is a real staleness signal) - per-attempt timeout now applies only to multi-source failover; a single arbitrary URL honors only the caller context and client - revert default client to http.DefaultClient (no behavior change for existing users); the per-attempt budget still bounds managed fetches - tests: sticky fallback skips the primary, hash mismatch falls back, malformed primary falls back, cancellation does not poison preferFallback, single URL is exempt from the failover timeout
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
https://storage.googleapis.com/token-directory-index/indexas a fallback source forindex.jsonand token lists. The primary GitHub raw URL remains the logical identity (TokenList.TokenListURLand cache keys stay the GitHub URL even when bytes come from the mirror); the GCS bucket is purely a transport fallback.How it works
fetchFromURLstries URLs in order and returns the first200 OK. It falls back on 429/5xx, network errors, and stalled sources — an error is returned only if all URLs fail, viaerrors.Joinso no failure is lost.sourceAttemptTimeout, 10s): each source gets its own budget, so a stalled primary can't consume the caller's whole deadline and still leaves room for the mirror. The single-URL fetch is extracted intofetchOnceso the body is fully read before the attempt context is canceled (structural, not a comment).NewTokenDirectorynow defaults to a client with a 30s timeout instead ofhttp.DefaultClient(no timeout), so callers usingcontext.Background()can't hang forever. Invariant documented: client timeout must exceedlen(urls) × sourceAttemptTimeout.ErrSourceTimeoutsentinel: when an attempt times out but the caller's context is still alive, the error is remapped soerrors.Is(err, context.DeadlineExceeded)no longer masquerades as the caller's own deadline — retry logic can distinguish "source was slow" (safe to retry) from "my budget is spent".fallbackURLForusesstrings.CutPrefixso only a true prefix is rewritten (an embedded occurrence in a query string is left alone).Tests
tokendirectory_test.go(new) — 15 tests, all passing under CI flags (-p 1 -race):fetchFromURLs: primary 200 (fallback untouched), 500→fallback, 429→fallback, network-error→fallback, stalled primary→fallback (blocks on<-r.Context().Done(), exercises the real timeout path), canceled context makes zero requests, all-fail reports both URLs, no-URLs guardErrSourceTimeoutand notcontext.DeadlineExceeded/index.json,/mainnet/erc20.json), not just any pathVerified against the live mirror
index.jsonandmainnet/erc20.jsonreturn 200 from the bucketindex.jsonis byte-identical to GitHub (same sha256)FetchIndextransparently pulled the full index from GCS