Skip to content

feat: fall back to GCS mirror when GitHub source is unavailable - #34

Merged
pkieltyka merged 2 commits into
masterfrom
fallback-index-url
Aug 18, 2026
Merged

pkieltyka merged 2 commits into
masterfrom
fallback-index-url

Conversation

@pkieltyka

Copy link
Copy Markdown
Member

Summary

Adds https://storage.googleapis.com/token-directory-index/index as a fallback source for index.json and token lists. The primary GitHub raw URL remains the logical identity (TokenList.TokenListURL and cache keys stay the GitHub URL even when bytes come from the mirror); the GCS bucket is purely a transport fallback.

How it works

  • fetchFromURLs tries URLs in order and returns the first 200 OK. It falls back on 429/5xx, network errors, and stalled sources — an error is returned only if all URLs fail, via errors.Join so no failure is lost.
  • Per-attempt timeout (sourceAttemptTimeout, 10s): each source gets its own budget, so a stalled primary can't consume the caller's whole deadline and still leaves room for the mirror. The single-URL fetch is extracted into fetchOnce so the body is fully read before the attempt context is canceled (structural, not a comment).
  • Bounded default client: NewTokenDirectory now defaults to a client with a 30s timeout instead of http.DefaultClient (no timeout), so callers using context.Background() can't hang forever. Invariant documented: client timeout must exceed len(urls) × sourceAttemptTimeout.
  • ErrSourceTimeout sentinel: when an attempt times out but the caller's context is still alive, the error is remapped so errors.Is(err, context.DeadlineExceeded) no longer masquerades as the caller's own deadline — retry logic can distinguish "source was slow" (safe to retry) from "my budget is spent".
  • fallbackURLFor uses strings.CutPrefix so only a true prefix is rewritten (an embedded occurrence in a query string is left alone).

Tests

tokendirectory_test.go (new) — 15 tests, all passing under CI flags (-p 1 -race):

  • fetchFromURLs: primary 200 (fallback untouched), 500→fallback, 429→fallback, network-error→fallback, stalled primary→fallback (blocks on <-r.Context().Done(), exercises the real timeout path), canceled context makes zero requests, all-fail reports both URLs, no-URLs guard
  • stalled sources report ErrSourceTimeout and not context.DeadlineExceeded
  • request-path assertions: the mirror is hit at the correct paths (/index.json, /mainnet/erc20.json), not just any path
  • URL-rewrite prefix semantics, including the embedded-URL case
  • default client has a non-zero timeout

Verified against the live mirror

  • index.json and mainnet/erc20.json return 200 from the bucket
  • index.json is byte-identical to GitHub (same sha256)
  • Live run with the primary pointed at a dead server: FetchIndex transparently pulled the full index from GCS

Add https://storage.googleapis.com/token-directory-index/index as a
fallback source for index.json and token lists. The primary GitHub raw
URL remains the logical identity (TokenListURL, cache keys); the mirror
is only a transport fallback.

- fetchFromURLs tries URLs in order, returning the first 200 OK; falls
  back on 429/5xx, network errors, and stalled sources
- per-attempt timeout (10s) so a stalled primary still leaves budget
  for the mirror; default client now has a 30s timeout instead of
  http.DefaultClient's none
- ErrSourceTimeout sentinel distinguishes slow sources from the
  caller's own deadline expiring
- errors.Join keeps every source's failure in the returned error
- tests cover 429/5xx/network-error/stall/cancel paths, request path
  assertions on the mirror, and the URL-rewrite prefix semantics
…tion

- preferFallback: after a primary failure, token-list fetches go
  straight to the GCS mirror instead of paying a per-file primary
  timeout; index refreshes always probe the primary first so recovery
  is detected within one 30s index cycle. Caller cancellation does
  not mark the primary unavailable.
- response validation: 200 responses are validated (JSON unmarshal +
  structural check) before acceptance, so a garbage body triggers the
  next source instead of failing the fetch
- content-hash verification: fetched token lists are checked against
  the index ContentHash; a stale source is rejected and the other
  source is tried (index and lists are updated atomically, so a
  mismatch is a real staleness signal)
- per-attempt timeout now applies only to multi-source failover; a
  single arbitrary URL honors only the caller context and client
- revert default client to http.DefaultClient (no behavior change for
  existing users); the per-attempt budget still bounds managed fetches
- tests: sticky fallback skips the primary, hash mismatch falls back,
  malformed primary falls back, cancellation does not poison
  preferFallback, single URL is exempt from the failover timeout
@pkieltyka
pkieltyka merged commit afbe1af into master Aug 18, 2026
4 checks passed
@pkieltyka
pkieltyka deleted the fallback-index-url branch August 18, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant