CHANGELOG.md

# Change Log

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## [Unreleased]

### Added

- An operator can capture the trace of one request: sent with
  `X-AR-IO-Trace: 1` and the admin bearer key, the request runs in its own
  root span, which the head sampler always samples (whatever
  `OTEL_TRACING_SAMPLING_RATE_DENOMINATOR` is) and the bundled OTEL
  collector always keeps (new `force-sample-policy`), and the response
  carries `X-AR-IO-Trace-Id` to find it by. Unforced requests are sampled
  exactly as before. The trace already records every retrieval source tried, so
  one curl answers "where did this come from, and what failed first"
  (#626). Ignored without the admin key.

- `arweave_chunk_fetch_orphaned_total{result}` counts shared Arweave peer
  chunk fetches that finished after every caller waiting on them had gone
  (#885). They are left running on purpose, as cache warming, which helps a
  later caller only if the result lands within the chunk promise cache's
  TTL. On one gateway 99.3% of peer-origin chunk requests were cancelled by
  the caller, so this measures how much fetch work outlives its callers.
  That work is bounded by the number of peer attempts, not by time (the
  per-attempt timeout stops once headers arrive); this is now documented
  where it arises (`getChunkByAny`).

- `POST /ar-io/admin/invalidate-data` (`{"id": "<id>"}`) forgets what the
  gateway recorded for one ID: cached hash, stored location, trust and
  verification, in the index and in memory, so the next request resolves
  it afresh (#959 item 6). It is the way to correct a wrong record that
  later writes cannot replace, such as one already marked verified, without
  editing SQLite by hand. The cached bytes are kept (they are stored by hash
  and may belong to other IDs); caches in front of the gateway still need
  their own purge.

- `ENABLE_GRAPHQL_METERING` (default off) puts `/graphql` under the rate
  limiter and x402, at a flat `GRAPHQL_OPERATION_COST_BYTES` per operation
  (default 10 KiB, 10 tokens; a batch pays per operation), charged before
  the query runs. Until now the most expensive route a gateway serves was
  the one it could not meter. Only the client's IP bucket is charged: the
  per-path resource bucket would be a single allowance shared by every
  GraphQL client. `/ar-io/info` reports the cost and the price.
- `CONTIGUOUS_DATA_CACHE_INDEX_MAX_BYTES_PER_SWEEP` (default 0, off) caps
  the bytes one contiguous-cache eviction sweep frees, so eviction is paced
  by bytes per interval instead of arriving in bursts. On turbo-gateway.com's
  btrfs cache HDD one 10,000-file sweep freed 258 GB over 30 minutes and
  stalled the disk (p99 latency 232 s for two minutes). The budget is ignored
  while free space is below `CONTIGUOUS_DATA_CACHE_MIN_FREE_BYTES`. Docker
  Compose now also passes `CONTIGUOUS_DATA_CACHE_INDEX_UNLINK_CONCURRENCY`
  and `CONTIGUOUS_DATA_CACHE_INDEX_MAX_BATCHES_PER_SWEEP`, which it did not.
- `ar-io-node index-l2-import --target clickhouse` and its worker
  (`L2_BAND_IMPORT_ENABLED`, off by default) load installed `parquet-l2`
  bands, such as `ardrive-l2`, into a gateway's ClickHouse, so its GraphQL
  answers for their data items without unbundling them. Each band is
  verified first (bound to the publisher's signed publication, its roots
  found in the L1 index, a sample of items read on chain) and refused whole
  if a check fails. One placement is kept per item, items ClickHouse
  already holds are left alone and disagreements counted
  (`l2_band_import_conflicts_total`), and nothing above ClickHouse's top
  height is written. Jobs are queued under `ETL_STAGING_PATH/l2-import` and
  loaded by `clickhouse-auto-import`, one at a time between its own batches
  (`CLICKHOUSE_AUTO_IMPORT_L2_POLL_INTERVAL`), with `clickhouse-import`'s
  new `CLICKHOUSE_IMPORT_KEEP_EXISTING`, which inserts only keys ClickHouse
  lacks, so a row the gateway exported meanwhile is never replaced. A ledger,
  `parquet_l2_imports` in `bundles.db` (new migration), records each band;
  `--forget <band>` removes what an import wrote. One 3.2M-item band of
  `ardrive-l2` verifies and queues in about two minutes and loads in about
  40 seconds. `--target sqlite` (`L2_BAND_IMPORT_TARGET=sqlite`) does the
  same for a gateway without ClickHouse, writing the items straight into
  `bundles.db` beside the running gateway, up to its stable L1 tip, in
  short transactions it can resume from; it is refused where
  `CLICKHOUSE_URL` is set. All of `ardrive-l2` is about 370 GB in SQLite.

### Changed

- AR.IO peer selection weights now drift back to their category default
  when nothing reports on a peer, halfway every
  `PEER_WEIGHT_DECAY_HALF_LIFE_MS` (default 15 min; `0` turns it off)
  (#1119). A peer marked down during an outage used to keep its low weight
  until it was picked and succeeded, which its low weight made unlikely.
  For that to reach the peers tried first, a key's home set (the hash-ring
  peers asked before any others) is now ordered by a weighted draw rather
  than strictly by weight: decay keeps weights in order, so a strict sort
  left a recovering peer behind its neighbours for good. Heavier peers still
  usually go first, and a peer near the minimum weight rarely does.
  `PEER_WEIGHT_DECAY_HALF_LIFE_MS=0` turns off both the decay and the drawn
  order, restoring the previous peer selection exactly.
- `/chunk/<offset>` and `/chunk/<offset>/data` now answer **503** with
  `Retry-After: 5` and `Cache-Control: no-store`, not 404, when a chunk
  could not be fetched in time: the serve deadline
  (`CHUNK_SERVE_DEADLINE_MS`), an upstream timeout, or an internal
  cancellation (#917). A 404 for an offset reads as permanent, so clients
  and peers cached it, peer weighting took it for "lacks the data", and the
  observer's reference-gateway client stopped at it instead of trying the
  network. Under load nearly every failure was this kind: 99% of requests
  at 500 concurrent, with a working gateway. A boundary lookup skipped
  because the chain fallback was at its concurrency limit is also a 503
  (`boundary_unavailable`), not a 404. A 404 now means no source found the
  chunk; one case is still ambiguous: when every peer fetch timed out, the
  peer cascade does not report why, and that ends in a 404 (`fetch_failed`).
  Dashboards will show these 503s as 5xx. Serve-deadline 503s are counted,
  as before, by `chunk_serve_deadline_exceeded_total`; all of them carry
  their reason in the `chunk.retrieval.error` span attribute; they are
  logged at debug rather than warn so a busy gateway does not flood its
  logs.

- The default `GRAPHQL_ROOT_TX_GATEWAYS_URLS` is now
  `{ "https://arweave.net": 1 }`, replacing
  `https://arweave-search.goldsky.com`, which now answers every request with
  a 307 to `https://arweave.net/graphql`: nodes on the default were already
  querying arweave.net, through an extra hop. Behaviour is otherwise
  unchanged. arweave.net rate-limits some addresses on its gateway endpoints
  (429, `retry-after: 299`); `{}` turns the source off, and a gateway with its
  own indexer can list that instead. The variable is now documented in
  `docs/envs.md`.
### Fixed

- Data fetched from a trusted gateway could be a JSON wrapper of the item
  instead of the item. arweave.net (now HyperBEAM) answers a request whose
  `Accept` includes `application/json` with a JSON document of the data
  (`raw-id`, `body`, `commitments`, `hashpath`, ...), for full and range
  requests alike, and axios sent its default `application/json,
  text/plain, */*` because the gateway set no `Accept`. arweave.net is in
  the default `TRUSTED_GATEWAYS_URLS`, so a gateway whose first upstream
  missed cached the wrapper and served it as the item, with the wrong
  content type and digest (observers report `dataHashDigest` and
  `contentType` mismatches). Byte fetches from trusted gateways, AR.IO
  peers and HTTP range sources now send `Accept: */*`. Copies already
  cached keep serving the wrapper until purged: they start with
  `{"accept-ranges"`.
- Two gateways that listed each other in `GATEWAYS_GQL_URLS` called each
  other's `/graphql` without end, until every hop timed out (#680). Fanned
  out queries now carry `X-AR-IO-Hops`, and a request that has already
  passed through `GATEWAYS_GQL_MAX_HOPS` gateways (default 2) is answered
  from the local index only.
- The Iceberg metadata scripts declared `root_parent_offset` as `int`; it
  is a byte offset into a root bundle and passes 2^31 in bundles over
  2 GiB, so it is now `long`, like the other offsets (#907). A nested
  bundle unbundled without a known offset in its root, whose items are
  recorded at `root_parent_offset` 0, is now logged and counted
  (`bundles_unbundle_unknown_parent_offset_total`) so its producers can be
  found.
- `POST /ar-io/admin/queue-bundle` queued every ID as its own root, so for a
  bundle that is itself a data item each of its items was recorded with the
  bundle as root and offsets relative to the bundle (#959 item 5, whose
  writer was unknown). Serving then rejected those locations, since the
  "root" is not an L1 transaction, and items failed intermittently. The
  route now looks up a nested bundle's root and payload offset first, and
  answers 409 when the root is known but the offset is not. Rows written
  this way before can be cleared with `POST /ar-io/admin/invalidate-data`.
- A data item stored in two bundles could be given a location made of one
  copy's root and the other copy's offsets: data attributes took the root
  from the location recorded at fetch time and filled missing offsets from
  the bundles index, even when the two named different roots (#959). The
  index now fills in only when it names the same root. Separately, an ID
  found only through its data root (an L1 transaction whose bytes are also
  a data item's) borrowed that other item's root and offsets; it no longer
  gets any location that way, and the ID's own row now always wins over one
  found through the data root.
- AR.IO peer weighting no longer treats a busy peer like one without the
  data: a 429 or 503 from a peer costs it a fifth of a failure's weight
  (#1118). Chunk peers were also penalised twice for every refusal (once
  where it was detected and again in the request's catch); each now counts
  once.
- Chunk-metadata anchoring asked only the first peer of its pool, so when
  that peer failed every lookup fell through to the slower chain search.
  It now asks the first peer not backed off: a peer that is unreachable,
  refuses (429, 402, 401, 5xx; a 404 is a healthy peer without the chunk)
  or contradicts the chain is skipped for 30 s, doubling to 10 min, and the
  next peer takes its probes. The pool now keeps every entry in priority
  order (previously `GATEWAYS_ROOT_TX_URLS` in object order, or only the
  first `TRUSTED_GATEWAYS_URLS` entry), and has its own setting,
  `CHUNK_METADATA_ANCHOR_PEER_URLS`, so anchoring can be pointed elsewhere
  or turned off (`{}`) without changing root-TX lookups.
- A gateway that started while the AR.IO gateway registry was unreachable
  had no peers until the next hourly refresh, so every peer retrieval failed
  for up to an hour. The peer list is now saved after each successful
  registry read (`AR_IO_PEER_LIST_CACHE_PATH`, default
  `data/peers/ar-io-peers.json`; Docker Compose mounts it from
  `PEERS_DATA_PATH`, default `./data/peers`) and loaded at startup, and
  while the list is empty a failed read is retried after 10 s, backing off
  to 5 min. A registry read that returns no gateways at all is treated as a
  failure rather than applied as an empty list.
- `X-AR-IO-Stable` was `false` on every response, and stable data never got
  `CACHE_STABLE_MAX_AGE` (30 days, `immutable`): since 2023 the gateway
  compared SQLite's `1` with `true`, which never matched. A transaction at
  height 1,500,000 on vilenarios.com, long past the fork depth, was served
  `X-AR-IO-Stable: false` with a two-hour `max-age`. Stable data is now
  reported stable, and directly addressed stable data (`/<id>`, `/raw/<id>`)
  that is also verified and served from the gateway's own cache gets
  `CACHE_STABLE_MAX_AGE`, `immutable`, for the first time; other stable data
  keeps the unstable tiers, and ArNS responses keep their own TTL.
  A data item the bundles flush has not yet promoted is also stable once
  its root L1 transaction is, rather than waiting for the flush (an item
  whose bundle has no `stable_block_transactions` row was never promoted).
  This is the same test the flush applies before promoting it, so no
  placement is trusted more than before. With no local record of the data,
  `X-AR-IO-Stable` and `X-AR-IO-Verified` are now sent as `false` instead
  of being left out.

- GraphQL no longer lists a data item's tags twice when the item was
  indexed from two bundles (or imported from an L2 band and later
  unbundled). The stable flush keeps the first stored row of an item, but
  wrote the tags of every placement beside it; it now writes tags only
  where it stores the item. Tag rows already duplicated are not removed.
- GraphQL no longer serves, or caches, a data item's signature or owner key
  read from a wrong offset. A signature is used only when its SHA-256 is the
  id, and an owner key only when its SHA-256 is the owner address; anything
  else counts as `attribute_fetch_total{outcome="invalid"}` and is logged
  with the offset that produced it. Cached entries that fail either check
  are removed. Rows with item-relative owner and signature offsets had been
  served wrong signatures, and, through the owner cache keyed by address,
  wrong owner keys for every item of the same owner.

- GraphQL results merged from upstream gateways (`GATEWAYS_GQL_URLS`) lost
  fields the query had asked for, because the selection forwarded upstream
  left out what the gateway reads back. Amounts read 0 AR when a query
  selected `quantity { ar }` or `fee { ar }` without `winston`; `parent`
  was null without `bundledIn`; and a field selected twice under different
  aliases (`a: tags { name } b: tags { value }`) kept only the first
  selection's sub-fields, which failed the whole query for tags. The
  forwarded selection now always asks for `winston`, merges every selection
  of a field, and the parent is read from `parent` or `bundledIn`. Stored
  data was never wrong, so nothing needs reindexing.

## [Release 87] - 2026-10-26

This is a **recommended release** focused on **index builds that fit
ordinary gateways, chain-bound L1 witnesses, and data correctness**. Key
highlights include index-export's memory held near its work by holding
glibc's allocator to two arenas (a 12M-item `parquet-l2` band now builds
at 3.3 GB instead of 16.8 GB) and DuckDB running one thread by default,
avoiding a DuckDB 1.4 crash in large spilled aggregates; a `parquet-l2`
band size cap that keeps dense ranges as sub-range bands; `parquet-l1`
layout `l1-4`, whose witness files bind format-1 and pre-2.0 blocks to the
chain, built by index-export; `ar-io-node index-census` to find items the
indexer indexed but ClickHouse never received, and a ClickHouse import fix
so the prune no longer drops unexported items; GraphQL pages that no
longer lose or repeat transactions where pending and mined rows meet; and
security updates to `multer`, `undici`, `express`/`body-parser`,
`form-data` and `ajv`. Gateways that subscribe to `parquet-l1` need this
release to take bands published as `l1-4`.

### Security

- Dependency updates for advisories in the HTTP and fetch stack, none
  imported by gateway code directly: `multer` 2.5.0 (DoS via a pending
  async file, orphaned disk writes), `undici` 7.30.1 (DoS on excessive
  errors, unbounded `EventSource` memory, SRI bypass), `express` 4.22.3
  with `body-parser` 1.20.9 (DoS via trailing bytes) and `body-parser`
  2.4.0 under `@apollo/server`, `form-data` 3.0.5 (CRLF injection in
  multipart field names), `ajv` 8.20.0 (ReDoS with `$data`), `fast-uri`
  3.1.9 and `stream-json` 3.7.0. `yarn audit` of production dependencies
  shows no high or critical advisory new since Release 86; those left
  are in build tooling (`node-gyp`) and SDK response validation (`zod`).

### Added

- **`ar-io-node index-census`: check ClickHouse holds every item the
  indexer indexed.** Read-only on `bundles.db` and ClickHouse: it compares
  each root's matched item count with the distinct ids ClickHouse holds
  under it, then lists the missing ids, so lost rows can be re-queued.
  On turbo-gateway.com it found 1.41 million items unbundled April–June
  2026 missing from ClickHouse on both nodes. See
  [docs/cli.md](docs/cli.md#index-census).
- **`parquet-l2` band size cap (`INDEX_EXPORT_L2_MAX_BAND_ITEMS`, default
  10,000,000).** A 100,000-height range is one band only while it holds at
  most that many items; above it the range stays as its twenty
  5,000-height bands. A settled range is counted at the source before it
  is frozen whole; twenty sub-range bands are rolled into one only while
  their items sum to the cap or less; and a band already over the cap is
  split into its sub-range bands, each built, compared with the old band's
  rows at its own heights and header-checked as a re-settle is, then all
  published together, each superseding the old band (subscribers keep it
  until every part is installed). On turbo-gateway.com, `ardrive-l2`'s
  1,800,000–1,899,999 band holds 69 million items, and its build peaked at
  13 GB of native memory, killed twice in a 16 GB container; its densest
  sub-range holds 12 million. Sub-range bands are already valid, so
  subscribers, verifiers and readers need no change. See
  [docs/index-export.md](docs/index-export.md#band-size-cap).
- **`parquet-l1` layout `l1-4`: witness files that bind every block of
  the chain**, so a full mainnet band set verifies and imports with no
  `--allow-*` flag. A band adds `witness_v1_data_roots.parquet` (the v1
  data root of each format-1 transaction with data) and, below the 2.0
  fork, `witness_fork_2_0.parquet` (each block's transition `tx_root` and
  the 2.0 seed fold after it), described in `band.json` and added in place
  without changing the band's id. `index-l1-verify --bands-dir` and
  `index-l1-import` recompute `tx_root` with the witnessed roots (a wrong
  one fails `tx_root`, never falls back to unbound), recompute each pre-2.0
  block's transition `tx_root` from its transactions, and carry the 2.0
  seed fold block to block until it reaches block 422,250's
  `hash_list_merkle` or a fold pinned in this build (heights 99,999,
  199,999, 299,999, 399,999, 422,249; the first four recomputed by a test
  from Arweave's own 2.0 checkpoint, the last equal to block 422,250's
  `hash_list_merkle`). Every other pre-2.0 header field, and the
  transaction ids, are bound through each block's "2.0 hash"
  (`witness_fork_2_0.h2`), recomputed from the band's fields and folded to
  pins of Arweave's `genesis_data/hash_list_1_0`: Arweave-checkpointed
  rather than chain-bound, and said so. A pre-2.0 block is bound only when
  both folds reach their pins. Under `l1-4` a format-1 transaction's
  `data_root` column must be NULL. A block whose witness is missing stays
  in its `unbound` class. New `witnesses` and `hash_list_1_0` checks; the
  result gains `foldBound` and `format1Txs`.
- Every column a band carries has a declared binding
  (`column-bindings.ts`, shown in docs/cli.md), checked by a test. Bands'
  `content_type` and `content_encoding` are recomputed from their tags,
  and `offset` (NULL, or where the data ends in the weave) from the weave,
  failing the new `canonical` check otherwise. The block sample now draws
  pre-2.0 heights too, attested by `--anchor-min` machines
  (`blocksAttested`): what binds a pre-2.0 block's transaction order.
- An `l1-4` band must hold every witness (a `witness_fork_2_0` row per
  pre-2.0 height, a v1 root per format-1 transaction with data); any of its
  blocks left unbound fails `witnesses`, whatever `--allow-*` flag is given
  (`--allow-unbound`, `--allow-unverified-format1` and
  `L1_BAND_IMPORT_ALLOW_UNVERIFIED` accept bands without witnesses only).
  The `INDEX_EXPORT_L1_WITNESS_MAX_MISSING` knob is gone.
- A pre-2.0 range ending at a pinned height is anchored there by its
  witnessed folds (`anchor.pinned`), so pre-2.0 bands verify and import
  with no `--allow-unanchored`.
- `--sample-blocks 0`, and an undrawn format-1 sample (no sources, or every
  count 0), need `--allow-unsampled` in `index-l1-verify` as in
  `index-l1-import`.
- Bands store one spelling per value: difficulties as `N.0` below the 1.8
  fork and `N` from it, no cumulative difficulty below 1.6, the reward pool
  as `N`, an unclaimed reward address as empty bytes, a wallet list always,
  no `hash_list_merkle` below 1.6 (the `canonical` check).
  The exporter re-spells what `core.db` holds into those forms, never
  changing a value, and refuses a value that isn't a whole number or a
  block without a wallet list.
- Under `l1-4` the importer writes the witnessed v1 roots and transition
  `tx_root`s over whatever `core.db` kept.
- `index-l1-verify --bands-dir` and `index-l1-import` refuse an `l1-1`
  band: its columns were its publisher's gateway's, not canonical. Both
  known publishers are on `l1-3`.
- `--sample-format1 0` over format-1 transactions with data from the 2.0
  fork now needs `--allow-unsampled`, in `index-l1-import` (whatever their
  witnesses) and `index-l1-verify` (new flag there): only the format-1 draw
  checks a transaction is format 1. Block 422,250's `hash_list_merkle` is now checked
  against the fold through 422,249 whenever a range holds both. Readers
  accept `l1-1` to `l1-4`; `--core-db` holds no witnesses and is unchanged.
  See [docs/index-publication.md](docs/index-publication.md#witness-files)
  and [docs/cli.md](docs/cli.md#what-binds-what).
- **`index-export` builds `l1-4` witnesses** for new bands and adds them
  to published `l1-3` bands in place, from v1 roots in a witness directory
  and Arweave nodes (`INDEX_EXPORT_L1_WITNESS_DIR`,
  `INDEX_EXPORT_L1_WITNESS_NODES`; unset, the core's
  `PREFERRED_CHUNK_GET_NODE_URLS`, else its default tip and data nodes,
  each asked through the node pool so a dead node rests instead of
  costing every chunk a timeout),
  keeping only what the chain confirms and judging the band with them, as
  its consumers do, before anything is published. Reported as
  `l1Witnessed` and `index_export_runs_total{kind="witness"}`. See
  [docs/index-export.md](docs/index-export.md#layout-l1-4-and-witness-files).

### Changed

- **DuckDB runs one thread by default in index-export and the index CLI.**
  The `duckdb` Node binding stops at DuckDB 1.4, whose hash aggregate can
  crash the process (SIGSEGV, exit 139) merging a large aggregate that
  spilled while running on more than one thread
  ([duckdb#23757](https://github.com/duckdb/duckdb/pull/23757), fixed in
  1.5 only). On turbo-gateway.com it killed a band split at its compare (4
  of 33 repeats of that compare crashed at two threads, none of 35 at one)
  and a full `index-l2-verify` of ardrive-l2 (which passed at one). L2
  band builds (`INDEX_EXPORT_L2_DUCKDB_THREADS`, now 1), L1 band writes,
  and the databases of `index-l1-verify`, `index-l2-verify` and
  `index-l1-import` (new `--duckdb-threads`, default 1) now run one thread.
  Results are unchanged; large steps take longer.
- `L1_BAND_IMPORT_ALLOW_UNVERIFIED` is needed only for bands without
  witnesses (`l1-3` and older); `l1-4` bands fill under the strict default.
- Wording: a pre-2.0 block hash is described per era (from the 1.6 fork a
  deep hash over the header fields and the full wallet list; below it a
  SHA-384 over the header as JSON with the full hash list and wallet list).
- better-sqlite3 is now 12.11.1, up from 9.6.0, which moves the
  gateway's bundled SQLite from 3.45.3 to 3.53.2. 9.x does not compile
  on Node.js 24, so this unblocks the move off Node.js 22 before its end
  of life (2027-04-30). The database file format is unchanged: a gateway
  can roll back to an earlier release without touching `data/sqlite`.

### Fixed

- **The ClickHouse import no longer prunes items it has not exported.**
  The prune threshold was the newest `indexed_at` of a stable data item,
  but the flush copies `indexed_at` from when an item was first indexed,
  so an item unbundled earlier (or re-queued) and flushed while a batch
  exported landed below the threshold after the export had read its
  range: it was pruned from SQLite, never reached ClickHouse, and its
  bundle stayed fully indexed, so nothing retried it. Stable items now
  carry `stable_at`, stamped by the flush itself, and the prune follows
  it. Adds a nullable column to `stable_data_items` in `bundles.db`
  (instant; migration `2026.10.21T00.00.00`).
- **On-demand data items' signature and owner offsets are stored on the
  same base as their data.** An item found through the root-tx index and
  read from its bundle had `offset` and `data_offset` stored relative to
  the root but `signature_offset` and `owner_offset` relative to the item,
  so GraphQL's `signature` and `owner` for it returned the bundle's own
  header bytes.
- **GraphQL pages no longer lose or repeat transactions where pending and
  mined rows meet** ([#947](https://github.com/ar-io/ar-io-node/issues/947)).
  Pending rows are walked by the full sort tuple, NULL heights sort the
  same in the merged query as in each leg, an ascending walk reaches the
  pending rows after the mined ones, a new data item's cursor carries its
  own id, and the second leg is read until the page's look-ahead row is
  had. In an ascending walk the ClickHouse legs no longer start over once
  a page ends on a pending row (they hold only mined rows), and pending
  rows sort and page by data item and id rather than by `indexed_at`,
  which each indexer stamps for itself, so a gateway federating several
  indexers no longer repeats pending items. A pending-row cursor issued
  before the upgrade may skip one pending row once, as the order changes
  under it. Thanks to @saravanan619 (#1096).
- The abort-signal middleware no longer assigns `req.signal`, which Node
  24 defines as a getter: on Node 24 every request would have failed with
  a 500. The gateway's own signal (aborted only when the client leaves
  before the response) is kept.
- **index-export's memory no longer grows to several times its work.**
  The service (and the CLI's verifiers) now run with glibc's allocator held
  to two arenas and returning large frees to the system
  (`MALLOC_ARENA_MAX=2`, `MALLOC_MMAP_THRESHOLD_=131072`,
  `MALLOC_TRIM_THRESHOLD_=131072`). A build runs DuckDB query after query
  in one process, and with glibc's defaults the memory each frees stayed in
  per-thread arenas: building a 12.0M-item `parquet-l2` band peaked at
  16.8 GB of anonymous memory though no single query used more than
  2.6 GB, and denser bands were OOM-killed in 16 GB. The same band now
  builds at 3.3 GB, about 15% slower, byte-identical. Override with
  `INDEX_EXPORT_MALLOC_ARENA_MAX`, `INDEX_EXPORT_MALLOC_MMAP_THRESHOLD` and
  `INDEX_EXPORT_MALLOC_TRIM_THRESHOLD`.
- **`parquet-l2` layout `l2-2`: the stats derive no longer runs a large
  band out of memory.** Each item's first value of a stat tag was taken
  with `arg_min(tag_value, tag_index)`, an aggregate whose BLOB state
  DuckDB 1.4.2 cannot spill, so the 69-million-item h1.8M band of
  `ardrive-l2` failed at the build's 2 GB limit, both in the in-place
  upgrade and in a rebuild. It is now the lowest `tag_index` per item and
  tag, joined back for the value: the same rows, and the same band derives
  in about 6 minutes at 2 GB.
- The `--publication` examples in `docs/cli.md`,
  `docs/index-publication.md`, `docs/index-verification.md`, the
  `index-l2-verify` help, and the `--publication` option description gave
  the signed index publication's URL as
  `https://<gateway>/ar-io/indexes/publication.json`, which returns 404.
  The gateway serves it at `https://<gateway>/ar-io/indexes`; the examples
  now use that. The local file `data/indexes/published/publication.json`
  is unchanged.
- `keccak` (pulled in by `@dha-team/arbundles`, and loaded by every
  SQLite worker thread) is pinned to 3.0.4. Version 3.0.2 kept one
  process-wide reference to its class constructor, so a worker thread
  loading it while another started or exited could abort the whole
  process (`Check failed: node->IsInUse()`, a segfault, or on macOS
  `libc++abi: terminating due to uncaught exception of type Napi::Error`).
- Shutdown now waits until every SQLite worker thread has exited, and no
  longer respawns a worker that dies while the gateway is stopping.

## [Release 86] - 2026-10-23

This is an **urgent, recommended release** focused on **the Node.js
Brotli remote-code-execution fix (CVE-2026-111497), index verification
that fails closed, and exact chain data**. Key highlights include the core
image moving from the end-of-life Node.js 20 to 22.23.4, and the default
observer image (`2aa908dc`, which also cranks epochs) moving to 22.23.4,
closing the Brotli decoder RCE and the TLS hostname-check bypasses from
Node's October 21 security release (**upgrade both images today**);
`index-l1-verify` and `index-l1-import` binding an L1 index to the chain
(anchor headers recomputed and agreed by two machines, every transaction
bound or sampled) and failing closed; `index-l2-verify`, which checks a
`parquet-l2` index against a signed publication and reads the roots it
names from Arweave nodes with every byte proven; a pre-2.4 block's
`reward_pool` stored exactly instead of rounded; `parquet-l2` layout
`l2-2` with a key lookup and stats per band; opting out of Index Sharing
now also stops seeding; and an upstream ArNS undername limit of 0 no
longer read as 10.

### Security

- **The core image now runs Node.js 22.23.4, up from 20.19.6.** Node's
  October 21, 2026 security release fixes an out-of-bounds write in its
  bundled Brotli decoder that permits remote code execution
  (CVE-2026-111497), plus two TLS hostname-check bypasses and a
  `dns.resolve` truncation bug. Node 20 reached end of life in April 2026
  and gets no fix. A gateway is exposed: its HTTP clients ask peers,
  trusted gateways and Arweave nodes for Brotli-compressed responses and
  decode them, and any gateway on the network can act as a peer. The
  runtime stage now copies the pinned `node` binary onto
  `distroless/cc-debian12` instead of using `distroless/nodejs22-debian12`,
  whose floating tag still carried an unpatched 22.22.0 the day after the
  fix shipped. The binary stays at `/nodejs/bin/node`, so compose files,
  `docker-entrypoint.sh` and the `tools/` scripts are unchanged. Operators
  who stopped Brotli decoding with a `NODE_OPTIONS=--require` shim can
  drop it once they run this image. Running from source now wants Node 22
  (`.nvmrc`).

- **The observer moves to Node.js 22.23.4 for the same fixes.**
  `OBSERVER_IMAGE_TAG` now pins an observer image built on
  `node:22.23.4-bookworm-slim`, up from 22.13.0. The observer fetches from
  every gateway on the network, and on a gateway that cranks epochs the
  same process signs with the operator wallet.

### Added

- **`index-l1-verify` and `index-l1-import` options to bind and to accept.**
  `index-l1-verify` takes `--sample-blocks`, `--sample-txs`, `--seed`,
  `--concurrency`, `--read-timeout`, `--allow-unanchored`,
  `--allow-unbound`, `--allow-unverified-format1`, `--spill-reserve-gb`, and
  with `--bands-dir` `--publication`, `--publisher-key`, `--registry-url`,
  `--index` and `--allow-expired`; its result has one shape in both views
  (`anchor`, `unbound`, `sample`, `skipped`, `accepted`, `assurance`).
  Both take `--allow-incomplete` (blocks whose transactions or owners'
  keys the index lacks, no longer under `--allow-unbound`) and
  `--sample-format1` (format-1 transactions drawn from blocks whose
  `tx_root` cannot be recomputed, default 200).
  `index-l1-import` takes `--anchor-from`, `--anchor-min`, the `--allow-*`
  flags, `--allow-unsampled`, the `--sample-*` options, `--seed`,
  `--concurrency`, `--read-timeout`, `--spill-max` and
  `--spill-reserve-gb`, and reports per band what bound it, what it
  accepted, what is unbound or skipped, the sample and the assurance. New
  env vars `L1_BAND_IMPORT_ALLOW_UNVERIFIED` (default false; not
  `--allow-incomplete`), `L1_BAND_IMPORT_SAMPLE_FROM` (default the trusted
  node and trusted gateways) and `L1_BAND_IMPORT_ALLOW_UNSAMPLED` (default
  false). A core migration adds `binding` and `accepted` to
  `parquet_l1_imports`. See [docs/cli.md](docs/cli.md#index-l1-verify).

- **`parquet-l2` layout `l2-2`: a key lookup and stats per band, derived
  for tag names the publisher chooses.** `lookup_key.parquet` holds, for
  each value of the index's `keyTags` (an ArDrive `Drive-Id`,
  `Parent-Folder-Id`), the heights at which items carry it and how many, so
  a reader finds a value spread over a band in a row group or two and then
  reads only the `tags` row groups at those heights, where `lookup_tag`
  gave only the first and last height and a wide drive cost the whole band.
  `stats.parquet` holds items and bytes per 1,000 heights, in all and by
  each value of the index's `statTags` (`App-Name`, `Content-Type`,
  `Entity-Type`), so a question over the whole index is answered from the
  small files alone. Both are configured per index in
  `INDEX_EXPORT_L2_INDEXES` (`keyTags`, `statTags`, none unless given) and
  recorded in `band.json`; neither enters a band's id. Every band built is
  `l2-2`, and a published `l2-1` band, or an `l2-2` band derived for other
  tag names, is upgraded in place under its id: the two files derived from
  its own tables, checked, moved in, and `band.json` replaced, before any
  build and outside the time budget (`derived` in the report,
  `index_export_runs_total{kind="derive"}`). A subscriber that does not
  know `l2-2` refuses the band and keeps its copy: upgrade subscribers
  first. `index-l2-verify` checks the two files as it checks the other
  lookups. See
  [docs/index-publication.md](docs/index-publication.md#l2-key-lookup-and-stats).
- **`index-l2-verify`: check an L2 index (a directory of `parquet-l2`
  bands, such as `ardrive-l2`) as anyone holding it can.** It needs no
  gateway and no database. Each band:
  - its id is checked against its contents, and the set for overlaps and
    holes;
  - every table is read row by row, in order, against `band.json`, and every
    owner key against its address;
  - every item's location must be laid out as ANS-104 requires, with its
    `data_size` the payload it gives;
  - every nested item must name a parent bundle that `parents` describes
    where its payload starts, and every `parents` row's payload must start
    inside it;
  - the lookups are checked against the tables;
  - a sample of items per band (`--onchain <n>`, 20 when not given) is read
    at their recorded locations from Arweave nodes and checked against their
    ids. The items are drawn uniformly over every located item of the band,
    then nested items are added until half the draw is nested, so rare
    nested items are always read; `--seed` repeats a run.
  `--spill-max` sets how much the lookup check may spill (128 GB unless
  set; the 69-million-item h1.8M band of `ardrive-l2` peaks at about 65 GB,
  past the 16 GB `index-l1-verify` allowed before, which it now shares).
  A band that its own publisher's newer band supersedes is left out. For L1
  bands, `index-l1-verify` checks the chain's own commitments instead. See
  [docs/index-verification.md](docs/index-verification.md), new, which
  covers every index kind: what each check proves, how to reproduce a band
  and how to repair one.
- **Reading roots from Arweave nodes, every byte proven**
  (`index-band-verify --arweave-nodes`, and `index-l2-verify` by default),
  so a header check trusts no gateway:
  - a root's signature must hash to its id and verify, binding its
    `data_root`, and the root must be format 2 and found in a block by two
    machines (see Fixed);
  - every chunk's proof is validated against that `data_root` under the
    consensus rules for its weave offset, and the chunk must hash to the
    proof;
  - a node serving a chunk that fails is passed over;
  - a root or chunk counts as gone (`unreadable`) only when every node that
    answered said 404. Busy nodes (429, 5xx) and chunks no node could prove
    are errors, never missing data, and a missing chunk is only that item's,
    not its root's.

### Changed

- The `CHUNK_METADATA_ANCHOR_*` settings now pass through
  `docker-compose.yaml` to core, so they can be set from `.env`. Before this,
  only a direct `yarn start` saw them.
- A lookup of an ArNS name that is not registered no longer logs a warning
  and an error. "Base name not found in ArNS names cache" is now `debug` (the
  `arns_name_cache_miss_total` counter still counts it), "Unable to resolve
  name against all resolvers" is now `info`, and "Error during parallel
  resolution" is logged only when a resolver timed out or was aborted, not
  when every resolver simply found no record. A resolver that throws still
  logs its own error.
- **Opting out of Index Sharing stops seeding.** With
  `INDEX_SWARM_SUBSCRIBE=[]` (or on the default publisher, whose default
  subscription skips itself) the `index-swarm` sidecar now takes the
  subscriber's torrents out of the engine once per start, files kept:
  installed bands stay installed and the gateway still searches them, but
  they are no longer seeded, and a download in progress stops. Torrents a
  publisher seeds from `published/` are unaffected. Subscribing again seeds
  the installed bands from their kept `.torrent` files, nothing fetched
  again. Previously the engine went on seeding installed bands after an
  opt-out.

### Fixed

- **The block importer stored a rounded `reward_pool` for blocks 585,633
  to 633,719.** Headers before the 2.4 fork (633,720) write `reward_pool`
  as a bare JSON number, and from 585,633 it passes 2^53, so `JSON.parse`
  rounded an odd value to an even neighbour (600,570 was stored as
  `...668`; the chain has `...667`). Block headers are now parsed with long
  integers kept exact, a cached header holding a rounded number is fetched
  again, and `sanityCheckBlock` refuses one. Gateways that indexed those
  heights themselves, or imported L1 bands built from them, hold the
  rounded values; `index-l1-verify` reports them as sample mismatches on
  `reward_pool`. The shared parser also stopped quoting digits inside JSON
  strings, which made such text unparseable.

- **`index-l1-verify` and `index-l1-import` could pass, or write, an L1
  index that was not the chain's; both now fail closed.** Three audits
  (2026-10-22) found ways through. Fixed:
  - The anchor compared only the top block's hash: a fabricated chain
    under the real top hash passed with two sources agreeing, a NULL hash
    at the anchor height compared nothing, the same machine under two
    names counted twice, and without `--anchor-from` the result still
    said `anchored: true`. Each source's header is now fetched whole, its
    hash recomputed from its own fields (every era from the 2.0 fork), at
    least two distinct machines must agree, and the index's row and
    transaction ids must equal it. Without an anchor the run says
    `anchored: false` and fails unless `--allow-unanchored`.
  - Blocks holding a format-1 transaction with data were skipped from
    `tx_root` and counted as passing, and pre-2.0 blocks (whose hash
    cannot be recomputed) were described as bound by the anchor. Both are
    now reported in `unbound` and fail unless `--allow-unverified-format1`
    or `--allow-unbound` accepts them. Every published `l1-3` band below
    height 2,000,000 holds such blocks.
  - Transaction ids, their order, and every transaction field were bound
    by nothing. Positions, counts, tags and owners are now checked against
    the links and wallets, and a keyed uniform sample of block headers and
    transaction signatures (`--sample-blocks`, `--sample-txs`) checks the
    rest, with the share of forged rows it would catch stated in `sample`.
  - `--bands-dir` read rows by a glob (so retired copies, second
    generations and stray directories were read, and sound sets failed),
    read only blocks and a few transaction columns, never checked
    `band.json`'s row counts and digests, column layouts, lookup file
    order, rows outside a band, band names, or a publication, silently
    dropped an unreadable edge band, clamped `--from`/`--to`, and ignored
    `--anchor-*`. Each band is now read by its own paths and checked whole,
    with `--publication` binding, and the anchor and sample run as for
    `--core-db`.
  - NULLs were read as zero or passed. They now fail `nulls`.
  - The import checked nothing against the chain: `--fill` spliced any
    band beneath held blocks, and the worker's fill did the same. Every
    band is now checked before a row is written, and bound by the block the
    gateway holds above it or by `--anchor-from`. A band refused part way
    with the gateway stopped left its rows behind and reported no hole; it
    is now cleared, and `holes` read from what the database holds. A band
    id's digest was never checked and a republished band was skipped as
    imported; both are now checked. A missing or unreadable bands
    directory made `--fill` exit 0; it is now an error, and the worker
    counts it `failed`.
  - The default `core.db` range stopped at the 2.0 fork; it is now the
    whole held range.
  - A second review round found more, all fixed. A refused import deleted
    rows the gateway held before it (its own, or a republished band's);
    it now removes only what it committed, and over held rows writes one
    transaction. A republished band outside `--from`/`--to` was imported.
    The import and fill drew no sample, so forged transaction fields
    passed; they now sample and refuse without one unless
    `--allow-unsampled`. A range imported unanchored or accepting a gap
    vouched for bands below it; the ledger now records each range's
    binding. Invented transaction ids no source holds came back
    unreadable rather than failing; one lying source could fail a run.
    An ECDSA transaction's fields, and a format-1 transaction's field
    boundaries, are not bound by its signature; they are now checked
    against two machines' answers (`txsAttested`) and the consensus rules
    (canonical decimals, target and anchor lengths, no format 1 from
    height 2,000,000). A sampled header at another height passed.
    `--allow-unbound` also accepted incomplete blocks. A format-2
    transaction relabelled format 1 escaped `tx_root` unsampled. The first
    block's link and size were never checked. A tag row filed under a
    transaction id at another height passed `tag_count`. A band whose
    `band.json` was removed silently shrank the set. `publisherBound` was
    true on a failed publication. `detects` ignored unreadable draws. The
    worker re-verified the same refused bands every interval; it now backs
    off until they change.
  - A third round found false refutations and more fail-opens, all fixed.
    Two sources answering 404 while a third that holds the item was busy
    refuted an honest index; absence now refutes only when every source
    was heard. A 404 refuted a format-1 transaction, though Arweave nodes
    now hide some real ones from `GET /tx`; it no longer does.
    `--anchor-min 1` let one machine refute a draw, or anchor the index on
    its own; `--anchor-min` now refuses anything below 2, and the sample warns when its sources are fewer. A sample that could
    not be read was reported as refuting the bands; it is now
    `refusal: "unreadable"`, and the worker retries it at the normal
    interval (`l1_band_import_runs_total{result="unreadable"}`).
    Transactions below the 2.0 fork were never sampled; they now are.
    `--sample-txs 0` (and `--sample-format1 0` over accepted format-1
    blocks) imported without `--allow-unsampled`. A draw whose owner key
    the index lacked was never asked about. A format-1 transaction past
    the 10 MiB limit, or relabelled and past the byte budget, passed as
    unreadable. A block an unfinished import wrote vouched for the band
    below it. A fill cleared rows outside the unfinished run it redid. The
    disk check ignored the write-ahead log of a band written over held
    rows. The default sample sources included gateways marked
    `trusted: false`, and a redirect could make one machine count as two.
    Fork gates on transaction rules were one block early, as the chain
    validates a block's transactions with the height of the block below.
  - A fourth round found a false refusal of real mainnet data and more,
    all fixed. RSA-PSS signatures were checked by OpenSSL, which is
    stricter than Arweave's `rsa_pss:verify` and rejected real mined
    transactions (`SnN5f1fBk3hD-_PSnFBFDze-kMBU-NFIKYWvrTEQWTo` at
    1,602,352); they are now checked by a port of Arweave's own rule (and
    its pre-2.4 legacy form), pinned. A source whose signatures did not
    verify could stop a survey early, and refuted draws were counted as
    proven (now `txsRefuted`, `format1Refuted`). Invented format-1 ids
    passed as possibly hidden; their block is now asked whether it lists
    them. The unreadable cap applied over the whole range, so an
    unreadable band passed beside readable ones; it now applies per band.
    A draw lacking its owner key was attested whatever owner the index
    named; the owner address is now compared. The format-1 byte budget is
    now 4 GB a band. The import refuses sources fewer than two machines
    up front, treats an anchor nobody answered as an outage, settles
    unfinished ledger rows that later imports covered (a range of another
    grid spanning two bands was redone for ever), and its disk check
    counts `missing_transactions` too. The worker backs off a fill whose
    sources could not be read, up to two hours.
  - A fifth round: an invented format-1 id past the byte budget was never
    checked against its block, and one in a block the chain does not have
    stayed unreadable; both are now refuted. Real pre-2.4 (legacy RSA-PSS)
    and ECDSA transactions are now in the tests.

  Gateways with `L1_BAND_IMPORT_ENABLED=true` now refuse today's `l1-3`
  bands below height 2,000,000 unless `L1_BAND_IMPORT_ALLOW_UNVERIFIED=true`.
  See [docs/cli.md](docs/cli.md#index-l1-verify).

- **`index-l2-verify` could report `ok: true` on bad data; it now fails
  closed.** Three audits found ways through. Fixed:
  - NULLs passed every row predicate: a row with no id, height, root or
    parent, a located item with no `data_size`, a nested item with no
    parent. They now fail the new `required` check, and every predicate
    treats NULL as failing.
  - Nothing checked `tags` and `wallets` against `data_items`, the items
    against the band's selection, the Parquet column types, or the order
    of lookup files (which were sorted before digesting). The new
    `consistency`, `selected` and `schema` checks do, and lookups are read
    in file order and must be in their key's order.
  - A directory entry without a readable `band.json`, or a malformed
    `.publisher.json`, was skipped silently; a band reached through a
    symlink was skipped too; and a `.publisher.json` could retire any band
    of its publisher. The first two now fail `files`, a symlinked band is
    followed and checked like any other, and a claim counts only when the
    claiming band covers the other's heights.
  - A band that vanished mid-run, or a file deleted mid-run, crashed the run
    or passed. That band now fails `complete` and the run goes on.
  - The on-chain check passed roots no block holds (one node serving a
    self-signed root), format-1 headers given a `data_root` they don't
    sign, and bands whose sampled roots were mostly made up. Roots must now
    be format 2 and placed in one block by two machines (or found in an L1
    index, `--l1-bands-dir`); a root no node has fails the band; a band
    with nothing located fails unless `--allow-unlocated`; agreement in place of a
    signature needs `--allow-agreement`; chunk proofs are checked under the
    rules of the root's block, never laxer than strict borders.
  - `--seed` did not repeat a run. Each band's draw is now seeded by
    sha256 of the run's seed and the band's id; the seed is random per run
    and reported, and `--seed` is for reproducing a run only.
  - Nothing bound the bands to their publisher: an id is a 48-bit digest
    anyone can recompute for rows of their own. `--publication`, with
    `--publisher-key` or `--registry-url`, checks every file against the
    publisher's signed publication; the result now says `publisherBound`,
    and lists every check it did not run under `skipped`.
  - Through `tools/ar-io-node`, `index-l1-*` and `index-l2-*` spill to
    `data/tmp/index-verify`, not the container layer on `/`; the spill is
    capped at 80% of the free space, a run that could not fit is refused
    before it starts, and an interrupted run removes its spill.
  - The on-chain sample was not uniform. First it drew only from about the
    first 82,000 rows of each band (DuckDB's `USING SAMPLE reservoir …
    REPEATABLE`); then, ordered by DuckDB's `hash(row, seed)`, which is a
    public function of the row XORed with one of the seed, its draws were
    clustered, so a publisher could fabricate a cluster of rows the draw
    almost never touched (caught in 11% of runs where the bound claimed
    93%). It now takes the N located rows with the smallest
    `sha256("<key>:<row position>")`, the key being sha256 of the run's
    seed, the band's id and the stratum, with nested items topped up to
    half the draw; the stated bound uses the uniform draw only.
  - A node that only failed to connect, or a machine without the
    transaction at all, no longer counts against a root: before any
    verdict of absence every silent node is asked again, any still unheard
    leaves the item unchecked, and a status 404 denies a root only from a
    machine that serves its header.
  - A band with nothing verified on chain is inconclusive, whatever
    `--max-unchecked` allows. With k items allowed unchecked, the stated
    bound treats them as possibly fabricated (a fabricated item behind
    unseeded data comes back unchecked): the share p at which
    P(at most k of N drawn are fabricated) is 5%, not the N − k draws it
    first claimed (N = 20, k = 4: 41%, where 18% was stated).
  - The on-chain check draws its sample up front and never replaces an
    item: each ends `ok`, `wrong` or `unchecked`; any wrong fails the band,
    any unchecked (after a second try) leaves it inconclusive unless
    `--max-unchecked` allows it. The pass ratio, floor and
    `--max-unreadable-share` are gone: tolerances stacked let fabricated
    items through. `assurance` says what a pass of N items means.
  - A sampled root the chain refutes (no `data_root`, format 1, a signature
    that doesn't verify, not mined, in a block at another height than its
    items) counted only as an error, tolerated up to 20%; it now fails the
    band. A node that times out no longer keeps a made-up root two machines
    deny from being `missing`; two ports of one address are one machine.
  - No single node can fail an honest band, any more than pass a bad one:
    an answer of one node that doesn't hold up is its fault, passed over
    for others and listed in `roots.nodeFaults`; a root is refuted only on
    evidence from two machines. A block is its hash and height together, a
    bound root is never `missing` (nodes holding no data answer 404 to
    `/offset`), machines are compared by all their addresses, and moved
    roots in the L1 check are listed with both positions.
  - The publication is refused once expired (`--allow-expired`), a band it
    lists in the range that the directory lacks fails, and `assurance` says
    when its key came from a registry.
  - New `coverage` check: heights asked for (`--from`/`--to`) that no
    checked band covers fail. New `located` check: a band's unlocated items
    must be exactly what its `band.json` declares and at most
    `--max-unlocated-share` (0.001) of them.
  - The spill also leaves 100 GB free, and spill folders a killed run left a
    day ago are removed.
  `index-band-verify --arweave-nodes` shares the root checks: roots must be
  format 2 and found in a block by two machines, and agreement needs
  `--allow-agreement`.

- **`parquet-l2`: a re-settle keeps every owner's key the bands it
  replaces hold.** Their `wallets` tables are a key source, tried after
  `bundles.db` and before the roots (a key verifies itself: its SHA-256 is
  the address), so a root the gateway can't serve on the day no longer
  refuses the band for a dropped key (`dropped_wallets`), as it did the
  h1.3M re-settle of `ardrive-l2`. Counted as `stats.owners.fromReplaced`.
- **`parquet-l2`: a nested item is no longer located under a parent bundle
  that `parents` does not describe.** Where every row of a bundle is
  inconsistent (offsets taken from another copy of it, ar-io-node #937),
  the bundle has no place in `parents`, yet a child held with an absolute
  offset was placed from its own row; `index-l2-verify` reported each as a
  `parents` defect (28 in the h1.3M band of `ardrive-l2`). Such a child is
  now left unlocated as `no_parent`, and a resettle locates it once the
  bundle's row is repaired.
  A re-settle that unlocates such an item is not refused for it
  (`unlocated_rows`): a location under a parent the replaced band's
  `parents` does not describe is one no reader could check, so losing it
  is reported as `corrected`, not as a loss. The pre-publish comparison
  also no longer uses correlated `EXISTS` subqueries, which DuckDB 1.4.2
  plans as a delim join over a parallel hash aggregate that failed to
  combine its thread states on the h1.3M band of `ardrive-l2` (an internal
  error, "Attempting to combine TupleDataCollection with mismatching
  types"). It is now one keyed pass: each side reduced to one row per
  (id, root), joined once into a narrow table every count and sample
  reads, and relocations found by one anti join of the located rows keyed
  by a 64-bit hash of their location. The comparison of that 6.7M-item
  band takes 12 seconds under the build's 2 GB DuckDB limit, where
  reading the two tables once per question took over an hour.
- An index-swarm subscriber can now install a Parquet band whose files have
  large footers. The footer limit was 2 MiB, and turbo-gateway.com's
  `ardrive-l2` band for heights 1,800,000–1,899,999 has a 3.7 GB
  `tags.parquet` (518M rows) with a 2.43 MB footer. Every subscriber
  downloaded the band, matched its digests, then discarded it and downloaded
  it again, about 7 GB each time. The limit is now 8 MiB. index-export also
  checks its own Parquet files against that limit before writing
  `band.json`, so a publisher can no longer publish a band its subscribers
  refuse. An over-limit footer now says so, instead of reporting that it
  "doesn't fit the file".

- **`tools/index-swarm-setup --restart` brought back a torrent engine that
  had been turned off.** It named `index-swarm-engine-init` and
  `index-swarm-engine` on the `docker compose up` command line, and Compose
  starts a service named there even when its profile is off, so a gateway
  sharing over HTTP only (engine behind an unused profile in
  `docker-compose.override.yaml`) got the engine again. Setup now names only
  the services a plain `docker compose up` enables (`docker compose config
  --services`, with `COMPOSE_PROFILES` honoured and the legacy profile names
  it passes not counted), reading `docker-compose.override.yaml` even when
  the running gateway was started before it existed, for `--restart` and for
  the commands it prints without it.

- **A chunk-metadata anchor probe kept running after every caller had given
  up on it.** Concurrent lookups of one offset share a single probe of the
  reference peer (a `HEAD /chunk/{offset}/data`, then the chain cross-checks),
  and that probe took no signal, so when every caller's request was cancelled
  it still ran to completion. Waiters are now counted: a caller that aborts
  while others wait rejects only its own lookup, and when the last one aborts
  the probe is aborted and dropped, so a later lookup of the offset starts a
  fresh one. The abort reaches the peer `HEAD` (or ranged `GET`) and the
  chain `/tx/{id}/offset` request; the chain `/tx/{id}` request does not take
  a signal yet, so one already started still runs to completion. An aborted
  probe is not remembered as a miss
  (`CHUNK_METADATA_ANCHOR_MISS_CACHE_TTL_SECONDS`).

- **An ArNS undername limit of 0 from a trusted gateway is kept, not
  replaced with the default of 10.** The resolver read the limit and index
  headers with `parseInt(header) || default`, so a name whose upstream
  limit was 0 was treated as allowing 10 undernames and served undernames
  it should not have. Only a missing or non-numeric header now falls back
  to the default (#927).

## [Release 85] - 2026-10-17

This is a **recommended release** focused on **the Arweave base layer as a
shared, verifiable index, serving compressed and mislabelled data correctly,
and chunk availability**. Key highlights include Index Sharing on by default
(a plain `docker compose up -d` now starts the `index-swarm` sidecar and its
torrent engine, subscribed to the ar.io publisher's root-TX bands; opt out
with `INDEX_SWARM_SUBSCRIBE=[]`); `parquet-l1` L1 index bands,
shared between gateways over the index swarm, an RSS feed per index and a
WebSeed, from which a gateway fills its L1 history while it runs
(`tools/index-swarm-setup --l1`) instead of indexing the chain block by
block, checked with `index-l1-verify`; opt-in `parquet-l2` bands of
ANS-104 data items, and root-TX bands on the L1 grid;
the `index-export` service, which builds a gateway's index bands, and the
`ar-io-node` CLI; fixes for gzip-encoded items served with a missing or wrong
`Content-Encoding`, and for uploads of the same bytes served with each other's
`Content-Type`; chunk fetches that learn which preferred nodes serve each
gateway, ahead of the observer's offset checks, which are back and fail
gateways by default (the default observer image moves to `c5e98ddc`); GraphQL
pages from ClickHouse no longer cut short; and peers announcing to a
publisher's tracker through the gateway's HTTPS (**upgrade the Envoy image
with core**).

### Added

- **L1 index bands (`parquet-l1`)**: the Arweave base layer (blocks,
  transactions, tags in plaintext, owners) as Parquet, one fixed height
  range per band (whole 100,000-height history bands over 5,000-height
  delta bands, so only the tip is ever rebuilt), so a new gateway can
  import its L1 index rather than index the chain block by block, and apps
  can query it with DuckDB. `index-export` builds them from `core.db` with
  `INDEX_EXPORT_KINDS` including `parquet-l1`, checking every block's
  links, `hash_list_merkle`, `tx_root`, transaction positions and owners'
  addresses before publishing; the index-swarm sidecar publishes them as
  `{"name":"parquet-l1","kind":"parquet-l1"}`, and a subscriber that names
  `parquet-l1` (`INDEX_SWARM_SUBSCRIBE` `name` may now be a list) checks
  each band's files, footers, schema and row counts before installing it.
  A subscription without `name` doesn't take them.
  - Two honest publishers of the same chain write identical bands, so they
    can check each other by digest and share one torrent and one swarm.
    Three columns a gateway fills in for itself are written one way:
    `blocks.tx_root` below the 2.0 fork is null (the block hash doesn't
    commit it, and nodes disagree), a format-1 `transactions.data_root` is
    null (the header carries none), and `content_type`/`content_encoding`
    come from the band's own tags, first by position. A band's `band.json`
    holds only what its rows determine, and its `createdAt` is the time of
    its top block. What is the publisher's own, the bands it replaces and
    its build time, sits beside the band in `.publisher.json`, which is
    never published. Checked on 2026-10-11 against turbo-gateway.com's and
    vilenarios.com's live L1 bands: all 24 are byte-identical.
  See *L1 bands* in [docs/index-publication.md](docs/index-publication.md)
  for the format and [docs/index-export.md](docs/index-export.md) for
  building them.

- **`parquet-l1` bands carry lookup files (layout `l1-3`)**, so one
  transaction, wallet or tag value can be found without scanning a band:
  `lookup_tx_id`, `lookup_wallet` and `lookup_tag`, each sorted by an
  unsigned 64-bit key in row groups of 16,384, so a reader goes from the
  footer to the one or two row groups that can hold its key. Measured on a
  100,000-height band over HTTP, in 256 KiB pieces: a transaction's height
  0.8 MB against 88 MB scanning `id`, a wallet's transactions and bytes
  stored 0.8 MB against 8 MB, the most used App-Name values 1.0 MB against
  113 MB. They add 14 to 19% to a band (59 MB on a 413 MB band, 254 MB on
  the busiest, 1.35 GB).
  - Declared in the layout beside the tables, described in `band.json` with
    rows and row digests, and left out of a band's id, which stays a digest
    of the tables' rows.
  - Subscribers check a band's files against the layout its `band.json`
    names, and lookup footers and row counts as tables'.
  - `index-l1-verify --bands-dir` checks each band's lookups against its
    tables and `band.json` (the `lookups` check; `--skip-lookups` to leave
    it out).
  - The engine (`src/lib/parquet/lookups.ts`, `keys.ts`) knows no dataset,
    and `RowDigest` moves beside it (`src/lib/parquet/digest.ts`), so another
    Parquet dataset declares lookups without new code.
  See *Lookup files* in [docs/index-publication.md](docs/index-publication.md).

- **A trusted gateway that answers 429 is skipped until its `Retry-After` has
  passed** (`GATEWAYS_THROTTLE_BACKOFF_ENABLED`, default `true`). Before, every
  request still went to it first: it waited for one of that gateway's sockets,
  often until the connection timeout, then got another 429. On
  turbo-gateway.com, arweave.net (a priority-2 fallback) answered one node's
  cold-item requests with `429` and `retry-after: 299` about 24,000 times a
  day, and 60% of those requests had waited 2-5 s for a socket, so cold items
  took ~3 s on that node against ~0.1 s on its sibling. The cooldown is the
  response's `Retry-After` (seconds or an HTTP date), capped at
  `GATEWAYS_THROTTLE_BACKOFF_MAX_MS` (300 s); without one it is
  `GATEWAYS_THROTTLE_BACKOFF_DEFAULT_MS` (30 s). Both gateway data sources
  share one set of cooldowns, and a throttled gateway is asked once per item
  rather than once per path. New metrics:
  `gateway_throttle_cooldowns_total{gateway_url}` and
  `gateway_throttle_skips_total{gateway_url}`.

- **A gateway can fill its L1 history from published bands while it
  runs** (`L1_BAND_IMPORT_ENABLED`, default `false`).
  `tools/index-swarm-setup --subscribe <publisher> --l1` subscribes to the
  publisher's `parquet-l1` bands and turns it on. Core then runs
  `ar-io-node index-l1-import --fill` every
  `L1_BAND_IMPORT_INTERVAL_SECONDS` (600) as a child process, so a crash
  in the import cannot take the gateway down. A fill writes only the
  heights `core.db` lacks below the top of `stable_blocks`, so nothing the
  gateway indexed itself is rewritten (a range an interrupted import left
  unfinished is the exception, written again whole), in short transactions
  with a pause after each, so the gateway keeps indexing and serving. It covers a
  gateway that began mid-chain, a hole inside what a gateway holds, and a
  new gateway: with the setting on, an empty `core.db` and no
  `START_HEIGHT`, the block importer starts at the chain tip instead of 0,
  and the history beneath arrives as the bands do. Only one import or fill
  runs against a `core.db` at a time. New metrics:
  `l1_band_import_runs_total{result}`, `l1_band_import_rows_total` and
  `l1_band_import_uncovered_heights`. See [docs/cli.md](docs/cli.md).

- `ar-io-node index-l1-import` fills a gateway's `core.db` from installed
  `parquet-l1` bands, so a new gateway starts from a published index rather
  than walking the chain block by block: with `--fill` while the gateway
  runs (above), or fastest with the gateway stopped. Stopped, it refuses a
  database another writer holds, one still holding unstable blocks, or one
  that has not run the import migration. Bands land in
  height order, each recorded in `parquet_l1_imports` as it commits, so a
  run that stops can be run again and carries on. A band is written to the
  ledger before its first row and completed after its last, and importing
  it clears its height range first, so a band a crash left part way is
  imported again over whatever it managed to write — progress is read from
  the ledger rather than from `stable_blocks`, which a band fills long
  before its transactions. Every write is idempotent, so a band imported
  twice leaves `core.db` exactly as importing it once does. A transaction a
  block lists that its band lacks becomes a `missing_transactions` row for
  the usual backfill. The run is refused up front if the filesystem has no
  room for it, says where it is once a minute while a band lands, and
  exits 1 with the result on stderr if a band is refused. `--cache-mib`
  sets the SQLite page cache (default 1024): a bootstrap spends its time
  maintaining indexes, and the 2 MB SQLite default makes the rate decay as
  the database fills. `--from` and `--to` fill in history beneath a
  gateway that began mid-chain, which the default upward-only run could
  not do; a band rewrites only its own range, so importing below what is
  held cannot disturb it. Because that can leave a hole between staged
  runs, and the block importer rewinds across one, the result carries
  `holes` and the run warns when any remain.
  Measured on vilenarios.com: the busiest 100,000 heights (46,606,658 rows)
  import in about half an hour and leave a 12.1 GB `core.db`; the whole
  chain is 468,501,488 rows and about 120 GB. See
  [docs/cli.md](docs/cli.md).

- `ar-io-node index-l1-verify` checks a gateway's L1 index against the
  weave size the chain commits to. Above the 2.0 fork a block's `tx_root`
  proves its transactions; below it, a pre-2.0 identity hash cannot be
  recomputed from an index at all (it commits to the full wallet list, the
  recall block's whole binary, and every transaction's data and
  signature). What is left is exact accounting: a pre-2.0 block grew the
  weave by its transactions' `data_size`, those differences telescope, and
  the first post-2.0 block commits the running total — so one trusted
  block hash pins the size of every transaction beneath it. Read-only, so
  the gateway can stay up; 422,251 blocks in about 4 seconds. It also
  rebuilds the `hash_list_merkle` recurrence — a running commitment to
  every block hash below, folded from height 0 for the fork-1.6 seed — so
  one trusted block hash pins every `indep_hash` beneath it, recomputes
  each post-fork block's `tx_root` from its transactions (1,383,754 of
  1,592,566 on the live chain; the rest hold a format-1 transaction with
  data and need `index-l1-audit`), and checks contiguity, block linkage
  and `block_size`. `--skip-tx-root` leaves out the one check that reads
  every transaction. `--bands-dir` checks a directory of bands with no
  gateway and no database, which is how a consumer verifies someone
  else's index before installing it; both views drive the same chain
  rules and a differential test holds them together. `--anchor-from` fetches that
  trusted hash from independent Arweave nodes (both sides of the fork,
  since its seed breaks the recurrence) and requires several to agree —
  one source is a single point of trust, so one reachable source fails. It does not prove which
  pre-2.0 block a transaction belonged to. See
  [docs/cli.md](docs/cli.md).

- `ar-io-node index-l1-audit` checks the blocks `index-l1-verify` has to
  skip. A format-1 transaction's `tx_root` leaf is the root of its data,
  which an index does not store, so 208,812 post-fork blocks of 1,592,566
  cannot be proved from the index alone. This fetches that data, derives
  each `data_root` and recomputes `tx_root`. It samples rather than sweeps
  (the full set is 111 GB) and reports the 95% upper bound on the error
  rate. The data source need not be trusted: wrong data can only cause a
  false alarm, never a false pass. Read-only. Measured on the full chain:
  1000 of 208,812 sampled, all 1000 matched, bounding the error rate at
  0.3%. See [docs/cli.md](docs/cli.md).

- The `index-export` service (compose profile `index-export`) builds this
  gateway's root-TX index bands from its own index once a day and puts them
  where the index-swarm sidecar signs and offers them: history bands cut at
  fixed edges, a recent band folded weekly (so rows ClickHouse has expired
  are kept) and frozen at `INDEX_EXPORT_RECENT_MAX_BLOCKS`, and a daily
  delta published only when its content changes. Several sources merge
  (`INDEX_EXPORT_SOURCES`: ClickHouse peers, SQLite, an overlay of a
  bundler's own offsets). Every band is header-checked against
  `INDEX_EXPORT_HEADER_CHECK_URL` before it publishes; a run that couldn't
  check is retried, a rejected one waits for an operator, and nothing
  published is withdrawn on failure. `index-swarm-setup --publish` sets it
  up (`--start-height`, `--header-check-url`), `index-swarm-status` reports
  it, and `prometheus.yml` scrapes it and the sidecar.
  `INDEX_EXPORT_L1_RUN_BUDGET_MINUTES` (default 240) sets how long a run may
  keep starting `parquet-l1` history bands; raise it for a bootstrap that
  would otherwise spread over several runs. Its other settings
  (`INDEX_EXPORT_*`) are in [docs/envs.md](docs/envs.md). See
  [docs/index-export.md](docs/index-export.md).

- `ar-io-node index-band-export` writes one record source's root-TX index
  records for a height range as the CSV `index-band-build` reads: this
  gateway's ClickHouse or SQLite, a peer's ClickHouse, or an overlay
  directory of coverage-named CSV files. Offsets are placed in the root only
  where proven: nested items whose `root_parent_offset` was zeroed (#907) are
  repaired when their parent proves the offsets relative, and otherwise keep
  their root without offsets. ClickHouse queries are read-only and bounded.
  The record sources are the first part of the `index-export` service; see
  [docs/cli.md](docs/cli.md).

- The `ar-io-node` CLI (`tools/ar-io-node`, [docs/cli.md](docs/cli.md)),
  in the style of the `ar.io` CLI from `@ar.io/sdk`: JSON on stdout, errors
  on stderr, exit 1 on failure. `index-band-build` builds an index band from
  CSV records, checks a sample of its headers against their root
  transactions, and publishes it for the index-swarm sidecar;
  `index-band-verify` runs that check on any band. Every other command runs
  as the `ar.io` CLI of the SDK the gateway carries. It runs in the core
  image and mounts only what a command needs.

- `INDEXES_ADVERTISE_FROM_URL` lets a node that serves an operator's
  indexes but does not sign them advertise the signing node's publication in
  `/ar-io/info`. Behind a load balancer, `/ar-io/indexes*` already goes to
  the signing node, but `/ar-io/info` is answered by whichever node the
  request reaches, so a crawler saw the `indexes` block only some of the
  time. The node fetches the document every minute, off the request path,
  and advertises its index names only if it names this node's `AR_IO_WALLET`
  as publisher and its signature verifies. A `404` or an invalid document
  withdraws them at once; failed fetches withdraw them after five minutes.
  Unset, nothing changes.

- `GET /ar-io/offsets/:id` now also answers from CDB64 indexes on local
  disk, such as bands installed by Index Sharing, when the local index can't
  place the item. It still makes no network request: remote CDB64 sources,
  and the remote partitions of the shipped indexes, are skipped. A new
  `offsets_lookup_total{source}` metric counts answers by `db`, `cdb64` or
  `none`.
- The index-swarm tracker reports who shares: `index_swarm_tracker_seeders`
  (tracked peers holding the whole band) and
  `index_swarm_tracker_seeding_hosts` (distinct addresses seeding at least one
  band, an IPv6 /64 counted once and this node's own engine left out: roughly
  how many other gateways share).
- **An RSS feed of each published index's torrents**, at
  `/ar-io/indexes/feed/<index>.xml`. One item per band offered as a torrent,
  with the `.torrent` as its enclosure and the magnet as its link, so a
  BitTorrent client such as qBittorrent follows a publisher with an
  auto-download rule and takes each new band as it is published. A view of
  the signed publication: it names the publication by SHA-256 in the channel
  and in an `X-AR-IO-Index-Feed` header, which HTTPSIG signs along with
  `Content-Digest`. Its URLs start with the new `INDEXES_PUBLIC_URL` (default
  `https://<ARNS_ROOT_HOST>`), never the request's `Host` header. Not
  metered; `public, max-age=60` with an `ETag`. Built on a format-independent
  feed layer (`src/feeds/`): a source builds a feed, a format renders it.
- **Peers announce to a publisher's tracker through the gateway's HTTPS**, at
  `https://<gateway>/ar-io/indexes/announce`, so a publishing gateway no
  longer needs port 6969 open. Envoy forwards the path to a second tracker
  listener on the sidecar (`INDEX_SWARM_TRACKER_PROXY_HOST`, default
  `index-swarm`), on `INDEX_SWARM_TRACKER_PROXY_PORT` (6970), which is
  never published and reads the peer's address from `X-Forwarded-For`.
  `tools/index-swarm-setup --publish --torrent` now writes that URL into
  `INDEX_SWARM_TRACKERS` when it knows the gateway's origin (`--public-url`,
  `INDEXES_PUBLIC_URL` or `ARNS_ROOT_HOST`). An existing tracker list is
  kept; the direct port still works. **Upgrade the Envoy image with core:**
  the route is Envoy's, and an older Envoy passes the path to core, which
  answers `404`, so every announce fails without a word. The listener
  believes `X-Forwarded-For` from `INDEX_SWARM_TRACKER_PROXY_TRUSTED_PROXIES`,
  every private range by default, like core's `TRUSTED_PROXIES`; narrow it
  where clients can reach Envoy from a private address. It refuses announces
  that bear a browser's marks (`Sec-Fetch-Site`, `Sec-Fetch-Dest`, `Origin`,
  `Referer`, or an `Accept` asking for an image or a page), so a page cannot
  list its visitors as peers.
- **`/ar-io/info` points at each index's feed.** Its `indexes` block gains
  `feeds`, mapping each index that has a band offered as a torrent to
  `/ar-io/indexes/feed/<index>.xml`, present only when the gateway serves
  feeds (it has `INDEXES_PUBLIC_URL` or `ARNS_ROOT_HOST`). A node that
  advertises another node's publication, on a fleet, advertises its feeds
  from that publication, when it has a public origin of its own; the feeds
  themselves are served by the signing node the `/ar-io/indexes*` pin
  reaches, so give it one too. So one request per registered gateway finds
  every feed.
- **A `.torrent` served by the gateway names the gateway as its WebSeed**
  (BEP 19 `url-list`, `<INDEXES_PUBLIC_URL>/ar-io/indexes/webseed/`), so a
  stock BitTorrent client following an index feed can finish a band that no
  peer is delivering, and can download a whole index, such as all of L1,
  from the feed alone. It sits outside the info dictionary, so the
  infohashes are the publisher's; the stored file is unchanged, and
  subscribers still drop every WebSeed and add the publisher's only when
  peers stall. The torrent route now answers with an `ETag` (`304` on a
  match), a `Content-Digest`, and an `X-AR-IO-Index-Torrent` header naming
  the v1 infohash, which HTTPSIG signs. Feed items name the folder a band is
  saved in (`ario:torrentName`) and the bands it replaces
  (`ario:supersedes`, only the claims the publisher honours), and each
  enclosure's length is the served torrent's. A stock client pulls a good
  share of every band from the WebSeed even while seeders could serve it,
  through the byte routes' meter when that is on;
  `INDEXES_TORRENT_WEBSEED=false` serves torrents without it. A stored
  `.torrent` whose info dictionary does not hash to the infohash it is filed
  under is not served.

- **L2 index bands (`parquet-l2`)**: a new index kind for the ANS-104 data
  items one declared selection takes (a tag name and a value prefix, with
  an optional narrower prefix excluded: `App-Name` starting `ArDrive`, say),
  with their tags, owners' keys, nested bundles and offsets in their root
  transactions, as Parquet on the L1 bands' height grid. The selection is declared in each band's
  `band.json`, bound into its id (`l2-h<from>-<to>-<publisher>-<digest>`),
  and carried in the signed publication. The index-swarm sidecar publishes
  them as `{"name":"ardrive-l2","kind":"parquet-l2"}`, and a subscriber that
  names the index checks each band's files, footers, schema, row counts, id
  and selection before installing it, refusing every band of an index whose
  bands, or `filter`, declare different selections. Opt-in, like
  `parquet-l1`. `index-export` builds them (below). See *L2 bands* in
  [docs/index-publication.md](docs/index-publication.md).
  - Every band carries three lookup files, as an `l1-3` band does, with the
    same keys: `lookup_id.parquet` (an item's id to its heights),
    `lookup_wallet.parquet` (an owner's or target's items, with heights and
    sizes) and `lookup_tag.parquet` (each tag pair, such as a `Drive-Id`,
    with its item count and first and last height), so a reader finds an
    item, a wallet or a tag pair without scanning the band. `band.json`
    describes them under `lookups`; they are not part of the band's id, and
    a subscriber checks their footers and row counts as a table's.
- **`index-export` builds L2 bands (`INDEX_EXPORT_KINDS=…,parquet-l2`)**:
  one index per selection in `INDEX_EXPORT_L2_INDEXES` (`ardrive-l2`:
  `App-Name` starting `ArDrive`, not `ArDriver`), published under
  `published/<name>/`, from this gateway's ClickHouse or a peer's
  (`INDEX_EXPORT_L2_SOURCE`), on the L1 grid with the root-TX grid's settle
  lag (`INDEX_EXPORT_L2_SETTLE_BLOCKS`), daily tip bands and weekly
  re-settles.
  - Every item takes its root's height and block position from this
    publisher's `parquet-l1` bands (else ClickHouse's L1 rows); a root found
    in neither refuses the band.
  - Items are located by the root-TX bands' placement rules, item-relative
    header offsets moved into the payload, and only when their offsets
    describe one item; absolute offsets are written parent-relative, and an
    ambiguous item is settled by reading its header from the root through
    `INDEX_EXPORT_HEADER_CHECK_URL`. Owners' keys come from ClickHouse,
    `bundles.db` or the root, kept only when they hash to the address.
  - The same rows give the same band and id whatever order they are read
    in; rows that can't be told apart refuse the band rather than be
    tie-broken. A band that replaces others never loses an item or a
    location, and every relocated item is header-checked.
  - `--once --dry-run --range <from>-<to>` builds one band of each index
    (`--keep` keeps it). New metrics: `index_export_l2_rows`,
    `index_export_l2_unlocated_rows`, `index_export_l2_owners_without_key`,
    `index_export_l2_proofs_total`. See *L2 bands* in
    [docs/index-export.md](docs/index-export.md).
  - An ambiguous item's proof reads its header as the band header check
    does: a header whose tag count field is malformed (see *Fixed*) still
    proves its reading, and is counted (`stats.malformedTags`).
  - Any band builds in memory that follows a window of the source, never
    the band: windows of at most `INDEX_EXPORT_L2_WINDOW_ITEMS` (50,000)
    items, rows sorted and digested a window at a time and appended to a
    DuckDB database in the staging directory, which writes the Parquet in
    order, with no sort of the band. DuckDB runs within
    `INDEX_EXPORT_L2_DUCKDB_MEMORY` (2GB) and `INDEX_EXPORT_L2_DUCKDB_THREADS`
    (2), spilling at most `INDEX_EXPORT_L2_DUCKDB_SPILL_MAX_GB` (256).
  - The lookup files are derived from the written tables in the same
    DuckDB, and checked before publishing: read back to their digests, and
    derived again to hold exactly the rows the tables give.
  - A root the gateway answers 404 for is unreadable for the build, asked
    about once: its ambiguous items are written unlocated (`unproven`) and
    an owner whose key could only be read there without a key, rather than
    stop the band; timeouts, 5xx and 429 still stop it, to be retried. A
    re-settle may locate such items, or key such owners, later; it never
    drops a key (`dropped_wallets`).
  - The header check reads its sample from a pool ten times as large, as
    root-TX bands do, passing over rows whose root is gone; what it read
    (`sampled`, `poolSize`, `unreadable`, `unreadableShare`) goes in the
    publisher's `.publisher.json`, never `band.json`, which holds only what
    the rows and the selection determine.
  - Re-settles are gated on a fingerprint of the rows each band's
    sub-ranges read (and the `bundles.db` keys of its owners), as root-TX
    grid bands are: `--resettle` gates, `--resettle=full` rebuilds. A band
    built with 404s is rebuilt every
    `INDEX_EXPORT_L2_RECHECK_UNREADABLE_EVERY` (4) gated re-settles.
- **Root-TX bands on the L1 grid, with a settle lag
  (`INDEX_EXPORT_ROOT_TX_LAYOUT=grid`)**: `index-export` cuts root-TX bands
  at the same heights as `parquet-l1` bands, so every publisher's bands line
  up whatever its start height, and subscribers stop downloading a
  multi-GB recent band every week. A 5,000-height `d` band freezes once its
  top is `INDEX_EXPORT_ROOT_TX_SETTLE_BLOCKS` (5,040, about 7 days) below the
  stable top, from the sources and the tip's entries; twenty of them roll
  into a 100,000-height `h` band; a `tip` above them is rebuilt daily,
  superseding the previous one, and published only when it changed. The
  default `legacy` layout is unchanged.
  - Frozen bands are re-settled every
    `INDEX_EXPORT_ROOT_TX_RESETTLE_INTERVAL_HOURS` (168; 0 for never, or
    `--once --resettle` to force one): rebuilt from their own entries and
    the sources, and replaced only when entries were added or relocated. A
    re-settle never loses an entry and header-checks only what it changes:
    every relocated entry (at most
    `INDEX_EXPORT_ROOT_TX_RESETTLE_MAX_CHANGED`, 20,000, a band), each of
    which must read and verify, and a sample of the added, so entries whose
    roots are no longer seeded don't block it. Any wrong header leaves the
    band as it is. Each is reported with its counts and counted in
    `index_export_resettles_total` and `index_export_resettle_entries`.
  - A due band is rebuilt only when its sources changed: each source's
    fingerprint of the band's heights is recorded when it settles (one
    aggregate query for ClickHouse; the files' names, sizes and SHA-256 for
    a CSV source) and compared first; equal fingerprints settle it without
    a rebuild (`unchanged`, `same_sources`,
    `index_export_resettle_gate_total`). `--once --resettle=full` rebuilds
    every one.
  - Grid bands built by hand with `index-band-build` (`h-`, `d-` and
    `tip-` ids) are taken as they are; a legacy layout's current `r` and
    daily `d` move into the new bands and the first tip supersedes them.
    Bands off the grid, or overlapping, stop the run before anything is
    built (`grid_layout`). See "Grid layout" in `docs/index-export.md`.
  - A `--once --dry-run` lists the plan in the report (`plan`).
  - `tools/index-swarm-status` takes the tip as the daily band once there
    is one.
  - `docs/index-export.md` documents the overlay files a `csv` source reads
    (names, columns, header).
  - `--once --resettle --range <from>-<to>` re-settles only the frozen
    bands inside those heights, building nothing else.

### Changed

- **Index Sharing is on by default.** The `index-swarm`,
  `index-swarm-engine-init` and `index-swarm-engine` services no longer sit
  behind compose profiles, so a plain `docker compose up -d` starts them
  (`--profile index-swarm` and `--profile index-swarm-torrent` are still
  accepted, and change nothing; `index-export` stays behind its profile,
  for publishers). Each default is taken only when its variable is unset or
  empty, so a gateway that sets them sees no change:
  - `INDEX_SWARM_SUBSCRIBE` defaults to the ar.io publisher's root-TX bands,
    `[{"publisher":"34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5","name":"root-tx-index"}]`
    (L1 bands stay opt-in); `[]` subscribes to nothing. A gateway whose
    `AR_IO_WALLET` is that publisher does not subscribe to itself: its
    default is no subscription, and the sidecar logs that at startup.
  - `INDEX_SWARM_ENGINE_AUTH`, unset, is generated by
    `index-swarm-engine-init` on its first start (`swarm:` and 48 random hex
    characters) and kept, root-owned and mode 0600, in
    `data/indexes/engine-auth`, where the sidecar reads it; it is reused on
    later starts and never logged. The engine still never runs without a
    password.
  - `CDB64_ROOT_TX_INDEX_SOURCES` defaults to
    `data/indexes/installed/root-tx-index` first, then the three shipped
    indexes, so installed bands answer first. A directory that does not
    exist yet is waited for, as before.
  - `ROOT_TX_LOOKUP_ORDER` defaults to
    `db,cdb,gateways,graphql,hyperbeam,cdb-remote` (was
    `db,gateways,graphql,hyperbeam,cdb`). A new entry, `cdb-remote`, splits
    CDB64 lookups by where the data is read from: with it in the order,
    `cdb` asks only indexes on local disk, such as installed bands, right
    after the local index, and `cdb-remote` asks the ones read over the
    network, including the three shipped indexes (partitions on Arweave),
    last. An item the bands do not hold goes to `gateways` without waiting
    on Arweave reads first. An order without `cdb-remote` behaves as before:
    `cdb` asks every CDB64 source, local and remote. Lookups are counted as
    `root_tx_lookup_total{source="cdb64"}` and `{source="cdb64-remote"}`,
    and each has its own circuit breaker, so slow remote reads cannot open
    the breaker for the installed bands (`circuit_breaker_state` labels
    `cdb64-local-root-tx-index` and `cdb64-remote-root-tx-index`; an order
    without `cdb-remote` keeps `cdb64-root-tx-index`). To stop using the
    shipped indexes, leave them out of `CDB64_ROOT_TX_INDEX_SOURCES`;
    removing `cdb-remote` from the order moves them up instead.
  - `INDEX_SWARM_MAX_DISK_BYTES` defaults to 64 GiB when
    `INDEX_SWARM_SUBSCRIBE` is also unset (the default subscription). With an
    explicit `INDEX_SWARM_SUBSCRIBE` and no budget set, there is still no
    ceiling, as before, so an existing subscriber gets no new limit. `0` is
    no ceiling, and a set value is used as given.
  - A sidecar that ran over HTTP only, with no engine password set, now gets
    the torrent engine too, since the engine runs by default and its init
    generates a password. To keep HTTP only, put the two engine services
    behind an unused profile (see "Turning it off" in `docs/index-swarm.md`).
    Two gateways on one host each publish the engine's peer port, so give
    them different `INDEX_SWARM_ENGINE_PORT` values.
  - The tracker's direct port (6969) is published on `127.0.0.1` by default
    (`INDEX_SWARM_TRACKER_HOST_IP`), so a subscriber exposes nothing it does
    not use; announces through the gateway's HTTPS are unaffected. **A
    publisher whose torrents announce to `http://<address>:6969/announce`
    must set `INDEX_SWARM_TRACKER_HOST_IP=0.0.0.0`**; the sidecar warns when
    it does not.
  - The engine's peer port, 6881 TCP and UDP, is published; opening it is
    recommended, not required (`INDEX_SWARM_ENGINE_PORT` moves it).

  Measured on a live gateway: about 22 GB of disk for the root-TX bands,
  about 90 MB of memory for the sidecar and 300 MB for the engine, upload
  capped at `INDEX_SWARM_UPLOAD_LIMIT_BYTES_PER_SEC` (10 MB/s).
  `tools/index-swarm-setup --torrent` no longer writes a password.
  `--subscribe` keeps the default subscription (never adding the gateway's
  own wallet), writes the 64 GiB budget when it turns an unset subscription
  into an explicit one (so the ceiling stays), and leaves an unset
  `ROOT_TX_LOOKUP_ORDER` unset. In a set order it puts `cdb` right after
  `db`, keeps every other entry, and adds `cdb-remote` last if missing. See "On by default" in `docs/index-swarm.md`.

- **A single-range GET of an uncached item fetches only the range.** The data,
  `/raw` and manifest-resolved routes used to fetch the whole item first,
  drop that stream and then fetch the range, so a range miss ran the
  upstream source cascade twice, and could wait on another request's
  download of the whole item (11-22 s to first byte measured on a live
  gateway). Now a GET for one satisfiable byte range of an item with an
  indexed size and no stored hash (on the data route, also a known
  non-manifest type) makes one ranged fetch and sends the same headers.
  HEAD, multipart, malformed and unsatisfiable ranges, cached items, and a
  source that returns another length or item size keep the old path.
  `uncached_range_fast_path_total{outcome}` counts `served`, `refused` and
  each `fallback_*` reason.

- **Range misses fill the cache in the background by default**:
  `BACKGROUND_CACHE_RANGE_MAX_SIZE` defaults to 10 GiB (was 0, off) and
  `BACKGROUND_CACHE_RANGE_CONCURRENCY` to 4 (was 1). The dropped whole-item
  fetch above kept downloading into the cache after the range was served,
  so it was what cached items read only by range (video, SQLite or Parquet
  over HTTP); the background fetch now does that, and a concurrent full GET
  shares it. A background fetch that receives no data for
  `STREAM_STALL_TIMEOUT_MS` is abandoned so its slot is freed. Signature and
  owner reads from a parent bundle never start one, so a GraphQL signature
  lookup does not download the whole bundle. Set the variables to keep the
  old values.

- **`tip-5.arweave.xyz` is no longer a default preferred chunk node**, for
  GET or POST. Its port 1984 was closed from two separate networks on
  2026-10-10, it served none of 60 random offsets, and `tip-1`'s own peer
  list no longer carries it. As a default it only spent chunk GET attempts
  and broadcast slots. The four remaining `tip` nodes still meet
  `CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT` (2) and
  `CHUNK_POST_MIN_SUCCESS_COUNT` (3). Operators who set
  `PREFERRED_CHUNK_GET_NODE_URLS` or `PREFERRED_CHUNK_POST_NODE_URLS`
  should drop it from their own lists too.

- **A chunk fetch now tries up to 8 Arweave peers, up from 5**
  (`ARWEAVE_PEER_CHUNK_GET_MAX_PEER_ATTEMPT_COUNT`). On the older half of
  the weave a chunk is often held by only one of the preferred `tip` nodes,
  and peers are picked by weighted random, so 5 attempts regularly ran out
  first and `/chunk/<offset>` answered 404 for a chunk the network holds.
  Measured from a US gateway: 16 of 20 random offsets served at 5 attempts,
  19 of 20 at 8 (turbo-gateway.com: 19 of 20). The observer's offset checks,
  restored in this release, fail a gateway that 404s on chunks it should
  serve. Each attempt is still capped at 500 ms, so a chunk no peer has now
  takes up to 4 s to fail instead of 2.5 s, and makes three more peer
  requests. Set the variable to keep the old value.

- **Offset (chunk-proof) checks are back in the observer, and fail gateways
  by default.** Every gateway (`OFFSET_OBSERVATION_SAMPLE_RATE`, default
  1.0; a lower rate samples a share, chosen identically by every observer
  each epoch) must serve `/chunk/<offset>` with a Merkle proof that
  validates against the chain. A checked gateway fails only if all
  `OFFSET_SAMPLE_COUNT` (default 4) offsets fail; timeouts, error
  statuses and bad proofs all count. With
  `OFFSET_OBSERVATION_ENFORCEMENT_ENABLED=true` (the default) that fails the
  observation, and the epoch result is still a 2-of-3 majority. A check the
  observer could not judge itself (its own chain lookups failed) is
  reported as `inconclusive` and never fails a gateway. This is not
  new behaviour. Enforcement was on by default from Release 54 to Release
  63, but the continuous observer that became the only service mode in
  Release 64 never ran the check, so the `OFFSET_*` settings have done
  nothing since. Reports again carry `offsetAssessments`, now with a
  `failureCategory` per failed sample (`timeout`, `network`, `http_status`,
  `invalid_chunk`, `bad_proof`, `unverifiable`). To record failures without
  failing gateways, set `OFFSET_OBSERVATION_ENFORCEMENT_ENABLED=false`; to
  turn the check off, set `OFFSET_OBSERVATION_ENABLED=false`.
  `OFFSET_SAMPLE_COUNT` is now passed through to the observer, and
  `docs/envs.md` gives the real defaults (it said 0.10 and `false`)
  (ar-io/ar-io-observer#145).

- **Default observer image bumped to `c5e98ddc`** — `OBSERVER_IMAGE_TAG` moves
  from `fe159f5a` to the `ar-io-observer` build carrying the offset checks
  above (every gateway by default, ar-io/ar-io-observer#148) and the
  reference fix under Fixed. Operators who pin
  `OBSERVER_IMAGE_TAG` in `.env` must update it there too, since that
  shadows the compose default.

- **Observer updated to `@ar.io/sdk` 4.5.2.** The observer's epoch crank
  now finds delegations to compound once per epoch instead of on every
  tick, which removes a pair of full-registry scans per tick from the
  window after each distribution (ar-io/ar-io-observer#146). A delegation
  whose compound reverts on every send is retried once and then skipped
  for the epoch, so the sweep always ends and the crank always reaches
  `create_epoch` (ar-io/ar-io-observer#147).

- **`@ar.io/sdk` updated from 4.0.0 to 4.5.0.** The hourly refresh of the
  gateway registry now reads only the registry's slot window and retries a
  failed batch (429 or 5xx) up to 3 times; before, one failed batch failed
  the whole refresh.

- **Index-swarm torrents are built with 256 KiB pieces, down from 4 MiB**, so
  a browser can verify a byte range of a band file against the signed
  `infohashV2` without the whole file: it rebuilds the file's BEP 52
  `pieces root` from the torrent's piece layer and checks each piece it
  reads. Reading a Parquet band this way fetches about a seventh of what
  4 MiB pieces would. A file's `pieces root` does not depend on the piece
  length, so only the torrents change: on its first scan after the upgrade a
  publisher rebuilds every band's torrent once and offers it under new
  infohashes, and subscribers keep their installed bands and seed the new
  torrents. Band ids and files are unchanged. See `docs/index-swarm.md`.

- `GET /ar-io/offsets/:id` answers are now signed (HTTPSIG), with a
  `Content-Digest` binding the body and an `X-AR-IO-Root-Transaction-Id`
  header, so an answer is the gateway's attributable claim. An answer from a
  CDB64 index never carries `contentType` or `dataSize`.
- The bare root of a sandbox subdomain (`GET`/`HEAD /` on
  `<52-char base32>.<ARNS_ROOT_HOST>`) now answers **451** when the ID it
  encodes is blocked and **404** otherwise, instead of falling through to
  `GET /` and returning `/ar-io/info` with a 200. Abuse reports cite that bare
  URL, so a blocked item's sandbox looked live to the reporter. Both responses
  use the not-found TTL (`CACHE_NOT_FOUND_MAX_AGE`, `must-revalidate`) rather
  than the 30-day blocked TTL, because an unblock never revalidates this URL.
  Every other path on a sandbox host is unchanged.

- **Faster whole-band reads.** `Cdb64Reader.entries()` reads in 1 MiB
  pieces rather than three reads a record (`entryBatches()` yields a
  piece's records at once), and a re-settle's comparison walks two bands in
  id order: 2.6 MB/s before, about 400 MB/s after on local disk.

### Fixed

- **Gateways missing the same chunks asked each other without end, through
  the offset lookup.** A chunk request another gateway forwards carries
  `X-AR-IO-Hops`, and the chunk sources never pass it to an AR.IO peer again
  (one hop at most). The offset lookup did not keep that limit:
  - the chunk-metadata anchor probed its reference peer (every gateway on the
    defaults shares one, `GATEWAYS_ROOT_TX_URLS`: `https://turbo-gateway.com`)
    with no `X-AR-IO-*` headers, so the peer saw a new client;
  - the tx_path check fetched a chunk from AR.IO peers without the request's
    hop count, and did so even with `ar-io-network` taken out of the
    retrieval orders.
  So a chunk that gateway A forwarded to B, and that B lacked, sent B back
  into the network, and from there back to A. On 2026-10-17 the gateways of a
  few large operators asked for offsets none of them held, and both
  turbo-gateway.com nodes stopped answering for six hours: in 32 minutes one
  made 563,600 peer chunk requests that found 7 chunks. Now:
  - a forwarded request (`X-AR-IO-Hops` of 1 or more) has its offset
    resolved from this gateway's own database only: no anchor probe, no
    AR.IO peer, and no chain search, whose few slots belong to this
    gateway's own clients. The asking gateway does its own chain lookup on a
    miss, and a database failure is reported as a failure, not a 404. A
    request that must not be forwarded to AR.IO peers skips the anchor and
    tx_path and keeps the chain search;
  - the anchor's probe sends `X-AR-IO-Hops` (and nothing else of the client's
    request), so the peer treats it as forwarded;
  - the tx_path check passes the request's hop count, and asks AR.IO peers
    only when `CHUNK_METADATA_RETRIEVAL_ORDER` includes `ar-io-network`;
  - concurrent lookups of one offset share one anchor probe, and an offset
    the anchor could not resolve is not probed again for
    `CHUNK_METADATA_ANCHOR_MISS_CACHE_TTL_SECONDS` (default 60; `0` turns it
    off);
  - a negative `X-AR-IO-Hops` is read as 0.
  This holds for every `CHUNK_PEER_ORIGIN_MODE`, including the default
  `off`: a peer's request no longer has its offset found through the anchor
  or the chain on its behalf, so `audit` no longer records
  `boundary="remote"`. `enforce` still differs from `off` by also declining
  Arweave nodes for the chunk itself.
  A gateway's probes carry the hop count once it runs this release; until
  then, the gateway it asks can stop the chain on its side by leaving
  `ar-io-network` out of `CHUNK_METADATA_RETRIEVAL_ORDER` and
  `CHUNK_DATA_RETRIEVAL_ORDER`, as turbo-gateway.com does. Outcomes are
  counted in `ario_chunk_metadata_anchor_total` as `peer_origin`,
  `miss_cached` and `probe_shared`.

- **Lookup checks no longer run large bands out of memory.** Verifying a
  `parquet-l1` or `parquet-l2` band's lookup against its tables compared the
  two with `EXCEPT ALL`, which DuckDB holds in memory, deriving the lookup
  twice. A 100,000-block ArDrive `parquet-l2` band (h1.6M) failed it under
  the default `INDEX_EXPORT_L2_DUCKDB_MEMORY` of 2 GB, after its build had
  passed. The derived rows are now digested in the lookup's order, as the
  file's are, and compared by digest; the set comparison runs only to say
  how a failing lookup differs. On a real 57.8M-tag band, the tag lookup's
  check needs about a third of the memory (passes at 512 MB; the old check
  failed at 1 GB). Verifying is slower in exchange: about 4.7 minutes for
  that band's three lookups against about 3 before, since every row is now
  read in order and digested.

- **JSON data was served as `application/json; charset=utf-8`.** Express
  appended the charset to every JSON, JavaScript and text type, so a client
  that compares the type exactly, such as Zora's metadata check, rejected
  JSON from every ar.io gateway. A JSON type is now sent exactly as it was
  uploaded; RFC 8259 defines no charset for JSON, and `fetch`, XHR and JSON
  modules read it as UTF-8 in Chromium, Firefox and WebKit. Until a Safari
  release carries WebKit's 2026-09-07 fix for JSON display, non-ASCII JSON
  opened directly in a Safari tab can show as mojibake.
  Text and JavaScript types keep the charset, since without it browsers
  decode UTF-8 text documents as windows-1252. Edge caches keep the old
  header on responses they already hold until those expire or are purged.

- **A cached range response claimed the whole item's digest.** A `206`,
  and the `416` or `400` answer to a range request, carried `Content-Digest`
  set to the SHA-256 of the whole item, a signed claim its body fails, since
  `Content-Digest` covers only the bytes sent (RFC 9530). These responses
  now carry `Repr-Digest`, the digest of the whole item, which HTTPSIG also
  covers, as the `/ar-io/indexes` routes already did. Full responses keep
  `Content-Digest`.

- **A Range header with an unknown unit was refused.** `items=0-5` got a
  `416`; RFC 9110 requires an origin server to ignore it and send the whole
  item, which it now does. The `bytes` unit is matched case-insensitively.

- **A manifest the index does not know was parsed on every request.** Its
  path resolutions were cached but never read back, so each request fetched
  and parsed the manifest again (66 ms on average on one gateway, up to 2 s).
  A resolution from trusted bytes is now reused; one from an untrusted
  source still decides only its own response.

- **A fleet node could not subscribe to its own fleet's index publication.**
  The fleet recipe in `docs/index-swarm.md` subscribes a non-signing node to
  the fleet wallet, which is that node's own `AR_IO_WALLET` (it must be, for
  `INDEXES_ADVERTISE_FROM_URL`). The sidecar resolves publishers from the
  gateway's `/ar-io/peers`, and a gateway never lists its own wallet there,
  so the publisher never resolved and every poll counted `unreachable`.
  `/ar-io/peers` now reports the gateway's own registry entry as `self`,
  apart from its peers, and the sidecar resolves its own wallet from it.

- **A write naming a different root no longer keeps the old root's offsets
  (#937 hardening).** `upsertNewDataItem` merged the root atom field by field,
  so a write for the same data item under another bundle that omitted the
  header offsets kept the previous root's `offset`, `size`, `signature_offset`
  and `owner_offset`: a header offset that frames nothing in the new bundle,
  which the read path then served. A write whose root differs from the stored
  one now replaces every root-atom field with its own values, NULLs included;
  same-root partial writes still keep what is known. No current writer sends
  such a partial write, so this is hardening: the 161 mixed rows found in
  turbo-gateway.com's ClickHouse (heights 1.30M–1.74M) were written by the
  statement before PE-9073, which did not update the header offsets at all.

- **Gateways far from the preferred chunk nodes served only about a third of
  old chunks.** A chunk GET tried nodes that advertise the chunk's range
  (`/sync_buckets`) first, then preferred nodes in list order, each capped
  at 500 ms. The advertised ranges are unreliable: the `tip` nodes serve far
  more than they claim (for 39 of 64 node and chunk pairs sampled, the node
  held the chunk without claiming its range), and with the default list a
  US gateway spent its attempts on range claimants and on `data` nodes in
  Germany that miss the 500 ms limit from there. `/chunk/<offset>` then
  answered 404, which the observer's offset checks count against the
  gateway. Preferred nodes now go first, ordered by a cost each gateway
  learns per node: an average of time-to-success where any miss counts as
  the full timeout. Each gateway settles on the preferred nodes that serve
  it well from where it runs, with no regional configuration. Simulated on
  40 random chunks surveyed from a US gateway with the default list, on
  offsets the cost was not learned from: 30% served before, 90% after (95%
  with `PREFERRED_CHUNK_GET_NODE_URLS` set to the US `tip` nodes). Live on
  gatewaypie.com (US) with every chunk setting at its default: 86% of 100
  random chunks served while it learned after a restart, then 96% of the
  next 100, against 98% from turbo-gateway.com; vilenarios.com on the old
  order and defaults had failed all 12 offsets the observer sampled. Expect
  a warm-up of roughly 100 chunk requests after each restart. New
  `peer_type="preferred"` label on `request_chunk_total`.
  `PREFERRED_CHUNK_GET_NODE_URLS` and `PREFERRED_CHUNK_POST_NODE_URLS` are
  now documented, and the documented default for
  `WEIGHTED_PEERS_TEMPERATURE_DELTA` is corrected to 2.

- **Item locations from Turbo's offsets API were never usable.** Turbo
  returns `startOffsetInRootBundle` and `rawContentLength` as JSON strings
  (Postgres bigints) although its API documents integers, and the gateway
  added them as-is: `"1102486" + 221` gave a data offset of `"1102486221"`.
  The location check refused every such location, so each first-seen item
  Turbo answered cost wasted chunk reads and two error lines before the
  bundle was searched instead. Turbo's offsets are now read as integers, and
  any malformed answer is refused and not cached. Item locations from any
  index are also refused unless every offset is an integer.

- **Byte-identical uploads with different `Content-Type` tags were all
  served as the first one a gateway saw.** For a data item the gateway had
  not indexed, the content type came from
  `contiguous_data.original_source_content_type`, which is keyed by the data
  hash and so shared by every upload of the same bytes. A path manifest whose
  body matched an earlier `text/html` upload was served as HTML and never
  resolved (seen on turbo-gateway.com for three manifests sharing the bytes
  of `7mPdjxByTDwNh2GAYT6ww_Gc-w7f-dvbfLqgjclJA6c`; an ArNS name pointing at
  one showed the raw JSON). The mislabel also spread between gateways: one
  that fetched the item from a trusted gateway stored that gateway's
  `Content-Type` as its own per-hash value. The item's own type is now
  recorded per ID (migration: `contiguous_data_ids.content_type` and
  `content_type_source`, two nullable columns), ranked by where it came
  from:
  - **`item`**, final: the item's signed `Content-Type` tag, read from its
    header by `RootParentDataSource`, or Turbo's per-item payload content
    type (S3, DynamoDB, Redis) when its record has one; the
    `application/octet-stream` served in place of a missing one is never
    recorded. `RootParentDataSource` now serves the header's
    type over a stored one; it used to read the header and keep the stored,
    possibly borrowed, type.
  - **`upstream`**: a trusted gateway's answer for that ID, written once and
    replaced only by an `item` value. Untrusted peers' types and a gateway's
    `application/octet-stream` placeholder are not recorded.

  `getDataAttributes` prefers the indexed tag, then the per-item value, then
  the per-hash value. A type a data source took from this gateway's own
  attributes is never re-recorded as per-item evidence. The in-memory
  attributes cache keeps the best-ranked type rather than the first, and
  drops an item's entry when the item or transaction is indexed, so a fix
  through `queue-bundle` or `queue-data-item` is served without a restart.
  An item already cached before this release keeps its per-hash type until it
  is indexed (for example with `POST /ar-io/admin/queue-bundle` for its
  bundle) or fetched again.

- **A response served through an ArNS name kept its old resolution after
  the name was re-pointed, for as long as the bytes stayed the same.** The
  entity tag was the content digest alone, so a cache revalidating its copy
  after the name moved to a manifest whose file at that path had the same
  bytes got a `304` and kept the old response's headers: the old
  `X-ArNS-Resolved-Id`, root offsets and signature (seen on
  turbo-gateway.com, a week-old resolution behind a 900 s TTL). On a response
  routed through an ArNS name the `ETag` is now
  `"<digest>.<X-ArNS-Resolved-Id>"`, so a `304` means the same bytes and the
  same resolution; the 304 and payment predictions use the same tag. `/raw`,
  `/{id}` and the index routes keep the plain digest. Treat the `ETag` as
  opaque and read the digest from `X-AR-IO-Digest`.

- **The webhook emitter no longer writes a target's secret to the logs.**
  `WEBHOOK_TARGET_SERVERS` can carry a credential in the URL: the content
  scanner's `WEBHOOK_SECRET` (v0.7.0) travels as `?token=`, since the emitter
  can't send headers. Every failed emission (e.g. while the scanner restarts)
  logged the full URL, token included, and the debug line did too. Logged URLs
  now keep the scheme, host, port, path and query parameter names, with values
  and userinfo replaced by `<redacted>`.

- **The observer no longer uses a gateway as its own reference.** The
  observed gateway was never excluded from reference lookups, so observing
  `turbo-gateway.com` or `ar-io.net` (the default reference hosts) compared
  each with itself and its ArNS check could not fail. The reference now
  comes from another reference host or from network consensus that
  excludes the gateway. Other gateways' references are unchanged
  (ar-io/ar-io-observer#144).

- **GraphQL pages served from ClickHouse could come back short with
  `hasNextPage: false`, silently dropping the rest of the results.** The
  query reads `(pageSize + 1) × CLICKHOUSE_GQL_DEDUPE_HEADROOM` rows before
  it folds duplicate ReplacingMergeTree versions, and where re-imported
  ranges held more versions of a row than that, the page came up short: on
  turbo-gateway.com and vilenarios.com, `first: 100` for one owner returned
  68 of 323 transactions. A short page whose read window was full now
  continues from its last row's cursor, and an id lookup re-runs with more
  headroom. A page still short after a few rounds is returned with
  `hasNextPage: true`, and rows from the other sources that sort after it
  wait for the next page. New metric:
  `clickhouse_gql_dedupe_headroom_total{leg,outcome}`.

- **ClickHouse auto-import no longer re-imports history it already holds.**
  A data item indexed late at an old height pulled each cycle's start back,
  and the cycle then re-imported every bucket from there to the tip: one
  item at height 783,731 cost 124 batches and 66 minutes, several times a
  day, and the duplicate versions it left behind caused the short GraphQL
  pages above. Buckets already imported without a gap are now skipped
  unless they hold stable data items. `clickhouse-import` now inserts only
  rows that differ from the newest stored version of their key
  (`CLICKHOUSE_IMPORT_SKIP_UNCHANGED`, default `true`), so a re-import adds
  no duplicate versions; an unchanged re-import no longer refreshes
  `expires_at`.

- **A core restart during a ClickHouse auto-import cycle failed every
  remaining batch as `debug_failed`**, leaving ClickHouse behind until the
  next cycle. The `/ar-io/admin/debug` fetch now retries with backoff, from
  5 s up to 60 s, for `CLICKHOUSE_AUTO_IMPORT_DEBUG_RETRY_WINDOW_S` (default
  600); a cycle that still can't reach it stops at that batch. Skipped
  buckets no longer wait on the fetch, and each request has a 10 s connect
  timeout and a 300 s total timeout.

- **An item tagged `Content-Encoding: gzip` whose bytes are plain was
  served with `Content-Encoding: gzip`, so every client that decodes failed
  on it.** A tag is the uploader's claim, not a property of the bytes; a
  manifest uploaded this way is on mainnet. The data routes now peek at the
  body before sending headers and, for a coding with fixed leading bytes
  (`gzip`, `zstd`), drop the header when the body does not start with them.
  A full `GET` always checks. `HEAD`, ranges and `304` answers, which do
  not send the body, check only cached bytes, so they never wait on an
  upstream; uncached, they declare the coding unchecked, and a failed read
  of cached bytes falls back to it. A copy decoded by a gateway older than #964 and cached fails
  the same check: the header is wrong for it too, so it is dropped, but such
  a copy is not the item and should be purged, so each drop is logged as a
  warning with the item's `verified` flag and counted in the new metric
  `served_content_encoding_dropped_total{encoding,verified}`. The
  `X-Arweave-Tag-Content-Encoding` header still names the tag, so a peer
  running #991 still refuses our copy as possibly decoded upstream and tries
  its next source, since from the bytes alone the two cases look the same.
  The check, here and in #991's upstream checks, now reads as many chunks as
  the coding's magic needs: it compared the first chunk alone, so a body
  whose first byte arrived by itself, `1f` for gzip, passed as gzip whatever
  followed, which let a decoded body through upstream.

- **A gateway older than #964 could poison the cache with a decoded copy
  of a gzip-compressed item.** Such gateways serve an item tagged
  `Content-Encoding: gzip` unzipped: some without the header and cut at the
  compressed length, some still declaring `Content-Encoding: gzip`. Fetched
  from a trusted gateway or an AR.IO peer, those bytes were cached and served
  as the item, with a `Content-Digest` over the wrong bytes. On vilenarios.com,
  docs.ar.io's 14.5 MiB search index came back as its first 2.5 MiB of plain
  JSON. Both sources now refuse, and try the next source for, a response
  whose body does not start with the magic bytes of the `gzip` or `zstd`
  coding it declares, and one whose `X-Arweave-Tag-Content-Encoding` names a
  coding (`gzip`, `br`, `deflate`, `zstd`) the response does not declare. New
  metric: `upstream_decoded_body_rejected_total{class,source,encoding,reason}`.
  A decoded body sent with neither the header nor tag headers still cannot be
  told apart, so do not trust gateways older than #964, and purge items
  cached before this fix.

- A 404 or 429 from a trusted gateway was logged as a warning, although the
  code meant to log those routine outcomes at debug: the check read a field
  that is only set for an unexpected 2xx, while axios reports a 4xx on its
  rejection. On turbo-gateway.com that was ~24,000 warnings a day from one
  fallback gateway's 429s.

- An ArNS name the trusted gateway does not resolve (a `404` with no
  `X-ArNS-Resolved-Id`) was logged as a warning, `Cannot read properties of
  undefined (reading 'match')`. It now logs `Name not resolved by trusted
  gateway` at info. Resolution is unchanged.

- `tools/index-swarm-setup --engine-port` and `--public-host` did nothing
  without `--torrent`: moving an existing engine to another peer port printed
  ".env already has everything" (while telling the operator to open the new
  port) and left `INDEX_SWARM_ENGINE_PORT` unchanged. Both now apply on their
  own, like `--max-disk-gib`, and `--restart` recreates the engine.
- `index_swarm_tracker_peers` counted peers of a swarm that had gone quiet,
  and bands no longer offered, until the next announce ran expiry. It now
  counts only bands still offered and peers seen within the announce TTL.
- **Gzip-encoded data fetched from a trusted gateway or AR.IO peer was served
  decompressed, still labelled `Content-Encoding: gzip`, so clients could not
  decode it.** An item uploaded gzip-compressed and tagged
  `Content-Encoding: gzip` is served by a gateway as the stored gzip bytes with
  that header. The node fetched it with axios, which decodes any response
  carrying `Content-Encoding` and drops the header, even when the request asks
  for `identity`. The node then served the decompressed body with
  `Content-Encoding: gzip` from the item's tag (browsers fail to decode it; curl
  `--compressed` exits 61), and never cached it, because the body did not match
  the indexed size, so every request fetched it again. Seen on
  turbo-gateway.com, where each node's trusted gateway is the other node, for
  gzip-encoded observer reports.
  - Upstream data fetches (`GatewaysDataSource`, `ArIODataSource`) no longer
    decode, so stored bytes pass through unchanged, and report the upstream
    `Content-Encoding` as the new `sourceContentEncoding`.
  - A response uses the item's indexed `Content-Encoding`, else the one its
    upstream reported for the bytes being served, so encoded bytes are never
    served unlabelled. A data item served as a byte range of its root bundle
    does not inherit the bundle's encoding.
  - Bytes whose upstream encoding differs from the item's indexed encoding
    (including an item whose encoding this node has not indexed) are served
    but not cached, so a later cache hit cannot serve encoded bytes without
    their header. Counted as
    `foreground_cache_skipped_total{reason="encoding_mismatch"}`.
  - Bodies already cached by a reverse proxy in front of the gateway (e.g.
    nginx `proxy_cache`) keep the wrong body until they expire or are purged.
- **Gzip-encoded data items were served without `Content-Encoding` by gateways
  that had not indexed them**, so browsers showed compressed bytes instead of
  the page. An item uploaded compressed and tagged `Content-Encoding: gzip` must
  be served as the stored bytes with that header, but the header came only from
  the index of the item's tags, which a gateway does not have for data items in
  bundles it does not unbundle (on turbo-gateway.com, every Turbo upload). The
  node now learns the encoding when it fetches the bytes:
  - from the item's own signed header, which it reads to serve an item from
    its root bundle anyway (`RootParentDataSource`);
  - from Turbo's S3 objects: the `payload-content-encoding` metadata when
    present, otherwise the item header at the start of the object, read in the
    same request as the payload (full reads only);
  - from a trusted gateway's response (see the entry above).
  The encoding is recorded with the item (a new `contiguous_data_ids.content_encoding`
  column, written once and never overwritten), so cache hits keep the header.
  It is recorded per item, not per data hash, so byte-identical uploads with
  and without the tag keep their own behaviour. An untrusted peer's encoding is
  served but not recorded or cached
  (`foreground_cache_skipped_total{reason="encoding_unverified"}`).
  - Only `gzip`, `br`, `deflate` and `zstd` are named in `Content-Encoding`;
    a list of stacked codings or an unknown value is served without the
    header, as before.
  - Remote CDB64 byte-range reads no longer decode a `Content-Encoding`
    response.
  - Items cached before this release keep serving without the header until
    their cache entries are refreshed.
- **Nested data items stored with an intermediate bundle as their root were
  served as 404s** (#959). An item inside a bundle that is itself a data item
  could be recorded with that bundle as its root and offsets measured in the
  bundle's payload. The bytes there are the item's, but since #937 every
  stored location has its header checked before use, and the check reads the
  recorded root as an L1 transaction, which it is not. The stored-root rebase
  only helps when the bundle has attributes of its own; without them the
  location was rejected, a local-first root TX lookup returned the same
  location, and the request failed. When a header cannot be read at all, the
  gateway now looks the recorded root up as a bundle, in its stored
  attributes and then the root TX index, adds the bundle's payload offset, and
  checks the header again in the enclosing root. A full read of a recovered
  location is served through signature verification, and the location is
  stored only once the payload verifies; a range read is served but not
  stored. The recovery runs at most once per location per request. On
  turbo-gateway.com about 5,500 of 5.1M stored locations per node had this
  shape; replaying 300 of them against production data, 283 were served from
  exactly the location CDB64 records for the item, and the other 17 have no
  index entry for their bundle.
- `data_item_location_check_total` gains a `reason` label on rejections:
  `header_unreadable`, `id_mismatch` or `offset_mismatch`. Before, every
  rejection looked alike, so read failures and roots that are data items read
  as wrong locations. Recovered locations are counted under sources ending in
  `_rebased`, and their payload verification under
  `data_item_signature_verification_total{source="rebased_location"}`.

- **A gzip item could lose its `Content-Encoding` while cached.** When the
  upstream that supplied an item sent neither the header nor the item's
  tags, nothing recorded the coding, and every later cache hit served the
  gzip bytes without `Content-Encoding` while still sending
  `X-Arweave-Tag-Content-Encoding: gzip`, so browsers failed to load them.
  The item's own `Content-Encoding` tag is now declared when nothing else
  names a coding, for gzip and zstd only and only when the body starts with
  the coding's magic bytes, so a copy decoded upstream is still served
  without it. Edge caches keep affected responses until they are purged.

- **Lookup file digests ordered by value**: the lookup engine digested a
  lookup's rows ordering by its own text renderings of the 64-bit columns
  (`id8`, `addr8`, `height`, `data_size`, …), so `10` < `100` < `9`, while
  the files are sorted by value: a reader digesting a lookup in file order
  could not reproduce its `rowDigest`. The order is now an explicit option
  (`numeric` or `legacy-text`). `parquet-l2` (`l2-1`) lookups use `numeric`,
  their files' own order. `parquet-l1` (`l1-3`) lookups keep `legacy-text`,
  so every published `l1-3` band's digests are unchanged and still verify;
  a future L1 layout moves to `numeric`.
- **Root-TX exports no longer place a row that can't frame its own item.**
  A row with no size, a payload that doesn't start after the item, a header
  longer than the item, or (its payload size known) an item that doesn't
  end where its payload does now keeps its root only, counted as
  `unrepaired` `layout`, top-level or nested, before any placement. Such
  rows (one at height 1,123,650: offset 1,762, size 1,151, data offset
  5,543, data size 3) were placed and failed header checks as wrong.
  ClickHouse and SQLite sources now read the item's payload size for this.
- **The header check verifies a band among the roots that can still be
  read.** History whose roots are gone network-wide no longer makes a band
  "couldn't check" (turbo-gateway.com's 1.3M band had 46% of sampled roots
  readable, and needed `--skip-header-check`). A read answered 404 is
  `unreadable`: neither passed nor wrong, left out of the 80% pass ratio,
  and its root not asked for again; 416 stays wrong, and timeouts, 429 and
  5xx still count against the ratio. Samples are drawn ten times over and
  walked until their size has been read (root-TX builds, re-settles,
  `index-band-build`, `index-band-verify`), and a band also needs 80% of
  50 verified (or of a smaller sample), which fails it only when gone roots
  use the pool up; a run's report gives `sampled` (entries read) and
  `poolSize`. Results, logs and
  `index-band-verify` report `unreadable`, the pool size and the unreadable
  share; `index_export_gate_unreadable_total` counts them. A relocated
  entry in a re-settle must still read and verify.
- **Band header check: an item with a malformed tag count is no longer a
  wrong offset.** Some early bundlers wrote the tag bytes' length into the
  header's tag count field too (an Arweave-signed item at height 789,458
  says 264 for 9 tags). Its signature hashes to its id and its header ends
  where its tag bytes say, so its location is right; the check called it
  wrong and refused the whole band (`wrong_header`), a root-TX band or a
  re-settle alike. Such an entry now passes and is counted separately
  (`malformedTags` in the check's result, its strata, a re-settle's
  report and `index-band-verify`'s output;
  `index_export_gate_malformed_tags_total`). Every other wrong header is
  still wrong, and a bundle scan still refuses such a header.

### Documentation

- **Index Sharing docs are split by reader, and CDB64 has one operator
  page.** `docs/index-swarm.md` covers running the `index-swarm` sidecar, and the new
  `docs/index-export.md` covers building root-TX and L1 bands.
  `docs/index-publication.md` now holds the L1 band format, lookup files and
  querying over HTTP. `docs/cdb64-guide.md` is merged into `docs/cdb64.md`,
  with the shipped defaults corrected (`ROOT_TX_LOOKUP_ORDER`,
  `CDB64_ROOT_TX_INDEX_SOURCES`, `CDB64_REMOTE_RETRIEVAL_ORDER`). Every
  default in `docs/envs.md` was checked against the code and
  `docker-compose.yaml` and corrected where it differed. The PlantUML
  diagrams are rendered to SVG under `docs/diagrams/`, and five stale drafts
  are removed.

- *Querying a dataset over HTTP* in `docs/index-publication.md` shows how to
  read a published dataset where it sits: a `parquet-l1` band is Parquet and
  the byte routes serve ranges, so a client needs no band files, no import
  into a database and no gateway of its own, and fetches only the footer and
  the row groups a predicate selects. DuckDB installs its own `httpfs` and
  `json` extensions on first use. The signed document is the catalog, since
  HTTP has no directory listing to glob.
  Because `Repr-Digest` is co-signable and commits to the whole file, a ranged
  read still carries a signature over the digest of the file it read from.
  Measured against a 24-band dataset covering heights 0 to 2,016,168 whose
  `transactions.parquet` files total 5.51 GB: a 100-block window across all 24
  bands in 0.1 s, and the full height span and row count in 0.6 s, over HTTPS,
  with nothing written to disk.

- The nginx guidance for `/ar-io/indexes` in `docs/index-swarm.md` named the
  wrong remedy and understated the cost. It said to set `proxy_buffering off`,
  which does not fix anything: when a location has a cache zone, nginx strips
  the client's `Range` from the upstream request and fetches the whole object
  so it can store it, before it has seen any `Cache-Control`, so no response
  header helps either. What the client sees then depends on whether the
  response was cacheable: a gateway that does not meter sends `public` and
  nginx answers the range from its cache, while a metering gateway sends
  `private`, nothing is stored, and nginx returns `200` with the whole body on
  every request. A cache zone is wrong on both branches, since the whole file
  crosses from the gateway either way and one band file can exceed 600 MB. The fix is
  `proxy_cache off` for the prefix, with no trailing slash,
  since a `proxy_pass` location whose prefix ends in `/` answers the unslashed
  form with a `301` and the document every subscriber polls lives there, with
  HTTPSig signing `@path`. The stated band size (7 to 30 MB) also predated
  `parquet-l1`, whose files can exceed 600 MB. Core and envoy answer ranges
  correctly, so the default stack was never affected; the guidance only
  mattered once an operator added a cache, and it failed quietly, because a
  `200` carrying more bytes than were asked for looks like success. Verified
  against a 541 MB band file: a 100-byte range returned all 541,753,181 bytes
  through a cache zone and `206` with it off.

## [Release 84] - 2026-09-29

This is a **recommended release** focused on **index sharing between gateways,
serving the right bytes and content type, and closing two ways to avoid rate
limits and payment**. Key highlights include the `index-swarm` sidecar, which
publishes and subscribes to signed CDB64 index bands over HTTP and optionally
BitTorrent; a check of each data item's header before its bytes are served from
a stored or indexed location, and fixes that stop items being served with their
bundle's content type; `X-Forwarded-For` trusted only from known proxies
(**upgrade the Envoy image with core**); x402 prices quoted in whole atomic
units, so requests over their limit are no longer served free when a price
could not be quoted; chunk retrieval asking Arweave nodes before AR.IO peers by
default; GraphQL on Apollo Server 5; and a background verification fix whose
migration builds a new index at startup, which can take several minutes on a
large `data.db`.

### Added

- **Soft deadline for GraphQL fan-out (`GATEWAYS_GQL_SOFT_DEADLINE_ENABLED`,
  off by default; `GATEWAYS_GQL_SOFT_DEADLINE_MS`, default 2000).** A gateway
  that fans `transactions` and `blocks` out to other gateways
  (`GATEWAYS_GQL_URLS`) waits for every upstream, so one slow upstream held
  every list query for its full request timeout. With the deadline on, once
  one source has answered with results, the merge waits at most the deadline
  for the rest and returns a partial result with an `UPSTREAM_SOFT_DEADLINE`
  warning per source it cut. An empty answer does not start the deadline:
  the local index usually answers first and is empty for exactly the data
  fan-out exists to find. Cut requests keep running, so the circuit breaker
  still records their real outcome. New metric
  `gateways_gql_soft_deadline_source_cut_total{source}`. A partial page can
  omit items that sort before ones it includes, so a client paging with
  cursors may skip them.

- **ClickHouse id lookup table for GraphQL `ids` and `bundledIn`
  (`CLICKHOUSE_GQL_ID_LOOKUP_ENABLED`, off by default)** (#946). On a large
  ClickHouse `transactions` table, `transactions(ids: [...])` with 3 or more
  ids and any `bundledIn` query failed with `TOO_MANY_ROWS`: `id_bloom` passes
  ~1% of all granules per id, and `parent_id` has no index.
  - New table `transaction_ids` (id → primary-key prefix), filled by
    `transaction_ids_mv` on every insert. `schema.sql` creates both on the next
    import; existing rows need a one-time backfill `INSERT`, documented there
    and in `docs/clickhouse-schema.md`.
  - With the flag on, the stable leg resolves ids (or `bundledIn` parents)
    there first and reads `transactions` by primary key. On 30M test rows, 3
    ids read ~20K rows instead of ~790K, 100 ids ~953K instead of ~18.5M, and
    one bundle ~9K instead of 30M.
  - Enable only after the backfill: an id missing from the table is treated as
    absent from ClickHouse. A `bundledIn` query with a parent missing from the
    table, and a failed lookup, run as before.
  - New metric `clickhouse_gql_id_lookup_total{filter, outcome}`.

- **Signed index publishing (`index-swarm` sidecar, `/ar-io/indexes`)** — a
  gateway can publish its CDB64 root-TX index bands for other gateways, and
  subscribe to theirs. Off by default (compose profile `index-swarm`).
  - The sidecar runs the core image with its own entrypoint. It signs a
    publication with the gateway's registered observer key (RFC 8785, Ed25519,
    a monotonic sequence), and resolves publishers through the gateway's
    `/ar-io/peers`, so it makes no Solana RPC calls.
  - Subscribers verify every document against the registry and every file
    against its signed SHA-256. Downloads resume, skip files already on disk,
    stop for the poll when a publisher's meter answers 402 or 429, and install
    the newest heights first. Installed bands load without a gateway restart.
  - New gateway routes: `GET /ar-io/indexes` (the signed document),
    `/ar-io/indexes/<name>/<band>/<file>` and the immutable
    `/ar-io/indexes/blob/<sha256>`. The byte routes are rate limited and
    priced with x402 like data egress, and signed with HTTPSIG. The errors these routes return (400, 402, 404,
    416, 429, 503) are `Cache-Control: no-store`, so a caching proxy never
    replays one. `/ar-io/info` advertises what is published.
  - See `docs/index-swarm.md` (operators: checklists, lookup order, running
    behind nginx) and `docs/index-publication.md` (the protocol).
  - `/ar-io/peers` gains each peer's registry fields (wallet, observer key,
    stake, status), which is what lets subscribers resolve publishers without
    RPC.
  - Built to take input from other gateways safely:
    - the signature is domain-separated (`ar-io-index-publication/v1\n` before the canonical JSON);
    - a publisher re-signs at once when its served document no longer verifies under its key (signed in the older format, or by a rotated-out observer key), instead of at the next refresh;
    - documents are bounded in size and shape;
    - band files are fetched only from the publication's origin (or `INDEX_SWARM_ALLOWED_FILE_ORIGINS`), and redirects are never followed;
    - downloads stop at their signed size while streaming, and refuse compressed bodies;
    - every CDB64 partition is walked and bounds-checked before a band installs, and the reader never trusts a length or pointer from the file;
    - replacing a band never leaves a moment when lookups to it miss;
    - one band id belongs to one publisher at a time.
    - the closed tracker lists this node's own engine under its public host instead of a Docker address, never hands private addresses to peers on the internet, and can sit behind a load balancer (`INDEX_SWARM_TRACKER_TRUSTED_PROXIES`, `INDEX_SWARM_ENGINE_PUBLIC_HOST`);
    - the tracker always lists the publisher's own engine, so a host firewall that refuses the engine's announce to itself does not hide the publisher; the default peer port is 6881, below the ephemeral range, and a short or empty engine password is refused (found by turbo-gateway's rollout);
    - seeding is bounded by default: 10 MB/s (`INDEX_SWARM_UPLOAD_LIMIT_BYTES_PER_SEC`) and 100 GB a UTC day (`INDEX_SWARM_UPLOAD_DAILY_LIMIT_BYTES`), after which the engine is throttled to 1 KiB/s until the next day, so no peer can pull terabytes from a publisher;
    - an hourly janitor removes engine torrents no state record claims (left by a state reset or a crash), after two sweeps agree;
    - a subscriber keeps a band that an offered band supersedes until that band has installed, so a replacement that takes hours to download leaves no gap in coverage;
    - a band republished with some files unchanged (a manifest edit to add `supersedes`, say) links the unchanged files from the installed copy and fetches only the rest; before, the whole band was fetched again;
    - after the first request, the gateway rechecks the publication file every 5 s off the request path, so later index requests do not wait behind a filesystem call on a saturated libuv thread pool (seen on turbo-gateway as 15–30 s responses during cache sweeps); only the first request after a start waits for the file;

- **Index bands over BitTorrent (compose profile `index-swarm-torrent`)** —
  with a torrent engine configured (`INDEX_SWARM_ENGINE_AUTH`, which points
  the sidecar at the compose engine), publishers
  also offer every band as a deterministic hybrid v1/v2 torrent and seed it,
  and subscribers fetch from peers first, turn on the publisher's metered
  WebSeed only when peers stall, fall back to HTTP on any failure, and seed
  every band they install. Off unless the engine runs; HTTP-only nodes are
  unaffected.
  - The engine is qBittorrent-nox 5.2.3, pinned by digest, on its own Docker
    network shared only with the sidecar, so it cannot reach the gateway, the
    observer or other services. It mounts the published and installed bands
    read only and writes only to `swarm/`.
  - A publisher runs a closed tracker that answers only for the bands it
    offers, and seeds from hard links to its blobs, so a band rebuilt in
    place is never served with bytes that fail their pieces.
  - A subscriber checks every `.torrent` against the signed infohashes and
    hands its engine only the info dictionary and trackers on public hosts:
    nothing outside the info dictionary is signed. Every file is hashed again
    before install. Downloads survive a restart without resetting their
    timeout.
  - New gateway routes: `/ar-io/indexes/torrents/<v1 infohash>.torrent` and the BEP 19
    WebSeed `/ar-io/indexes/webseed/<torrent name>/<file>`, metered and
    cached like the blob route.
  - See `docs/index-swarm.md#torrent-engine`.

- **`tools/index-swarm-setup` and `tools/index-swarm-status`** — set up
  index sharing in one command and check it in another. Setup edits `.env`
  idempotently (backing it up first): subscribes to a publisher, points the
  gateway at the installed bands and fixes the root-TX lookup order,
  generates the torrent engine's password, defaults the disk budget for
  installed bands to 50 GiB when none is set (`--max-disk-gib` changes it,
  also for an existing subscriber), and with `--restart` recreates only what
  the changes need, by service name, with the running gateway's own compose
  files. Status runs inside the sidecar and reports each check (publication
  accepted, bands installed and loaded by the gateway, lookups reaching them,
  disk budget, engine reachable from the internet, upload budget) with the fix
  for anything wrong. Both need only Docker. To subscribe to turbo-gateway.com, today's
  publisher:
  `./tools/index-swarm-setup --subscribe 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 --torrent --restart`,
  then open port 6881 (TCP and UDP). See `docs/index-swarm.md#quick-start`.

- **`tools/scan-bundle-offsets`** — builds CDB64 CSV input with offsets and
  item sizes for every data item in a list of root bundles, nested bundles
  included, by reading only each bundle's item index and item headers through
  a gateway's `/raw` range requests. Each item's header is decoded and its
  signature hashed to confirm the ID, and a bundle whose items do not add up to
  its size is rejected rather than indexed. Header reads are coalesced (bundles
  of small items read in 1 MiB windows, bundles of large items a few KiB per
  item), roots are scanned in parallel, and a progress file makes long runs
  resumable. An optional details CSV records each item's signature type and
  content type. An item with an unknown signature type is skipped with a
  warning instead of failing its root, rows are staged in part files so memory
  doesn't grow with a root's row count, requests are rate limited (10 per
  second by default, `--requests-per-second`) with `429` responses retried
  after their `Retry-After`, and IDs the gateway reports as data items are
  refused.

- **Item size in CDB64 root TX index values** — CDB64 values can now record
  the total data item size (`s`, header + payload) alongside the two root
  offsets. The generate tools accept it as an optional sixth CSV column,
  `data_item_size`, and `export-cdb64-root-tx-index` writes it. An offset
  without a size tells the gateway where an item starts but not where it ends,
  so every CDB64 hit still searched the root bundle's header for the item: the
  item count, the whole ID index, then the item header, each a separate range
  read of the root bundle. With the size recorded, the gateway reads the item
  header once at the recorded offset, checks that its signature hashes to the
  requested ID and that the header ends at the recorded payload offset, and
  serves the payload with the item's own content type. The payload goes
  through the same signature verification as direct offset hints, and the
  offsets are saved only once it verifies. Any mismatch, and any range
  request, falls back to the bundle search. The key is optional and ignored by
  readers that predate it, so existing indexes behave as before and new ones
  stay readable by older gateways. A size inconsistent with its offsets is
  ignored rather than invalidating the entry. Items located this way are
  counted as `root_tx_local_resolve_total{outcome="index_offsets"}`.

- **Peer-origin chunk modes (`CHUNK_PEER_ORIGIN_MODE`, default `off`).** A
  chunk request forwarded by another AR.IO gateway (`X-AR-IO-Hops` of 1 or
  more) can be answered from this gateway's own sources only. `enforce` uses
  the local cache, the local index and operator storage such as S3, makes no
  network requests for that peer, and answers 404 when none of them has the
  chunk. `audit` changes nothing and records what `enforce` would have done in
  `chunk_peer_origin_audit_total{boundary,bytes}`. `enforce` outcomes are
  counted in `chunk_serve_local_only_total{result}`. An invalid value stops
  startup. Requests from other clients are unaffected.

- **Manifest resolutions are cached and stored.** Resolved manifest paths are
  kept in an in-memory cache (`MANIFEST_RESOLUTION_CACHE_SIZE`, default 5000),
  so each asset request against the same manifest no longer re-parses it. On a
  7,784-path manifest a repeat resolution went from about 57 ms to under 1 ms.
  A manifest's index and fallback are also stored in a new `data.db` table,
  `manifest_resolutions`, so they survive restarts and can be served while the
  manifest body is briefly unavailable. Only manifest bodies from trusted
  sources are written to the table.

- **x402 payment metrics.** `x402_payment_total{outcome,target}` records how
  each payment attempt ended (for example `verify_failed`, `settle_failed`,
  `settled`), and `x402_payment_settled_usdc_total{target}` records the USDC
  settled. A payment that settles but then fails to grant access is counted as
  `outcome="topup_failed"` and its USDC is still recorded. See
  `docs/x402-and-rate-limiting.md`.

- **Chunk POST failure reasons.** `arweave_chunk_post_total` has a `reason`
  label: the peer's HTTP status when it answered, or `timeout`, `canceled` or
  `network`; successful posts carry `200` (stored) or `303` (held in the
  peer's disk pool) and `dry_run` when posting is simulated. Our own
  `CHUNK_POST_ABORT_TIMEOUT_MS` deadline is now reported as a timeout (504)
  rather than as the uploader cancelling (499).

- **ArNS metrics.** `arns_cached_resolution_fallback_on_timeout_total` counts
  requests answered from a cached (possibly stale) resolution because a fresh
  one took longer than `ARNS_CACHED_RESOLUTION_FALLBACK_TIMEOUT_MS`.
  `arns_resolution_duration_seconds` is added because `arns_resolution_time_ms`
  has always recorded seconds; the old name still works but is deprecated.

### Changed

- **Default observer image bumped to `fe159f5a`** — `OBSERVER_IMAGE_TAG` moves
  from `0e956b08` (2026-08-30, `@ar.io/sdk` 4.3.0-alpha.2) to the current
  `ar-io-observer` build on `@ar.io/sdk` 4.5.0. Observing is unchanged:
  `save_observations` did not change in the gateway registry's Wave 2 upgrade,
  so gateways on the previous image still submit correctly. The embedded epoch
  cranker (`ENABLE_EPOCH_CRANKING=true`) needs this image. Since Wave 2,
  `create_epoch`, `finalize_gone` and `compound_delegation_rewards` take new
  accounts, and the previous image's client fails on them. The cranker also
  now finalizes departed gateways in the only window the program allows it
  (between an epoch's distribution and the next epoch's creation), and claims
  delegations off leaving and delegation-disabled gateways into each
  delegate's withdrawal vault. The cranker wallet pays each vault's rent (at
  most about 0.0029 SOL, which the delegate recovers), and the sweep pauses
  while the wallet holds under 0.5 SOL. Operators who pin `OBSERVER_IMAGE_TAG`
  in `.env` must update it there too, since that shadows the compose default
  (ar-io/ar-io-observer#143).

- **GraphQL runs on Apollo Server 5** (from `apollo-server-express` 3, which
  has been end-of-life since October 2024). Queries, results, batching, GET
  requests, CSRF behavior and the invalid-cursor error are unchanged. What
  clients and operators can see:
  - GraphQL responses now carry `Cache-Control: no-store`. A CDN or proxy
    that cached GET `/graphql` responses stops caching them.
  - Malformed requests (a missing or non-JSON body, an empty batch) still get
    HTTP 400, but with a JSON `errors` body (`BAD_REQUEST`) instead of plain
    text. An unknown or missing `operationName` now reports
    `OPERATION_RESOLUTION_FAILURE` instead of `INTERNAL_SERVER_ERROR`.
  - `GET /graphql` in a browser serves the Apollo Sandbox instead of GraphQL
    Playground.
  - New metric `graphql_http_batch_size`: operations per HTTP request.

- **OpenAPI spec: current introduction, and the real version.** The spec's
  front matter (what the gateway serves, how to verify responses with the
  `X-AR-IO-*` trust headers and HTTP signatures, rate limits and x402, errors,
  authentication) replaces text from the project's first release. `/openapi.json`
  and `/api-docs` now report the running gateway's release as `info.version`,
  instead of a fixed `0.0.1`. The spec also gains a relative `servers` entry,
  so "Try it out" targets the gateway serving the page, and the current logo.

- **`root_tx_lookup_total{has_offsets}` now means both root offsets came
  back**, for every source. It used to also require `size` and `dataSize`. A
  CDB64 index never returns `dataSize` (it returns `size` only when the entry
  recorded one), so every index hit read `false`.
  A new `has_size` label records whether the item size came back. Queries
  filtering on `has_offsets="true"` will now count CDB64 hits, and any other
  source's hits that carry offsets without a data size.

- `TxChunksDataSource` now resolves a transaction's chunk-read geometry
  (`data_root`, offset and size) from the local stable transactions index
  before asking the trusted node, falling back to the chain on a miss. Every
  cold range read previously cost two trusted-node requests, which share a
  5 req/s budget with the transaction offset importer, so a busy gateway spent
  most of that budget re-reading geometry it already had indexed and starved
  the importer. A read that fails using local geometry re-checks it against
  the chain, at most once per transaction per hour, and retries only if the
  two disagree. Set `TX_CHUNKS_GEOMETRY_DB_ENABLED=false` to restore the
  previous behavior; `TX_CHUNKS_GEOMETRY_CACHE_SIZE` bounds the in-memory
  cache of resolved geometry.

- **Chunk retrieval asks Arweave nodes before AR.IO peers.** The defaults of
  `CHUNK_DATA_RETRIEVAL_ORDER` and `CHUNK_METADATA_RETRIEVAL_ORDER` are now
  `arweave-network,ar-io-network` (were `ar-io-network,arweave-network`).
  Peers only hold what they happened to cache, so asking them first was mostly
  misses. Peers stay as the fallback. This moves first requests onto Arweave
  nodes, which on a gateway that knows few nodes means mostly
  `TRUSTED_NODE_URL`. Set the old value to keep the previous order.

- **Forwarded chunk requests have a shorter deadline**
  (`CHUNK_PEER_ORIGIN_DEADLINE_MS`, default 3000). A request from another
  gateway used the general `CHUNK_SERVE_DEADLINE_MS` (12000), but the asking
  gateway gives up after one second, so later work reached no one. Requests
  from other clients are unchanged. `0` restores the general deadline. Hits
  are counted in `chunk_peer_origin_deadline_exceeded_total`.

- **The contiguous cache evictor deletes fewer files at once by default.** It
  deleted 50 files at a time and up to 50 batches per sweep, which could fill
  the libuv thread pool and stall every other file operation.
  `CONTIGUOUS_DATA_CACHE_INDEX_UNLINK_CONCURRENCY` now defaults to
  `UV_THREADPOOL_SIZE / 8` (at least 1, so 1 on a default thread pool of 4),
  and `CONTIGUOUS_DATA_CACHE_INDEX_MAX_BATCHES_PER_SWEEP` (default 50) makes
  the sweep bound configurable. A gateway whose cache grows faster than it is
  evicted can raise the concurrency.

### Fixed

- **Background data verification no longer scans the whole `data.db`
  `contiguous_data_ids` table.**
  - Marking a root transaction verified (`WHERE id = @id OR
    root_transaction_id = @id`) had no index on `root_transaction_id`, so every
    verification read the whole table while holding the single `data.db`
    writer. On a gateway with 99M rows each one took about 10 minutes, the
    writer spent half its time on them, and other `data.db` writes queued
    behind them (chunk placement confirmations waited about 3 minutes each).
    A new partial index, `contiguous_data_ids_root_transaction_id_idx`, makes
    it two index lookups.
  - Selecting the next batch to verify wrapped `verification_priority` in
    `COALESCE`, so SQLite walked every unverified row whenever fewer than 1000
    qualified. It now seeks to the qualifying range: 229 s became 0.02 s on
    the same gateway.
  - A `MIN_DATA_VERIFICATION_PRIORITY` of 0 or below still counts
    unprioritized data as priority 0 and verifies everything.
  - **Upgrade note:** the migration builds the new index at startup. On a
    72 GB `data.db` (99M rows) on a SATA SSD it took about 11.5 minutes and
    added 3.8 GiB, so the gateway starts that much later on the first boot
    after upgrading.
- An invalid `after` cursor on GraphQL `transactions` or `blocks` now returns
  `GRAPHQL_VALIDATION_FAILED` on gateways that answer from SQLite. The cursor
  was decoded inside the SQLite worker thread, which re-wraps any error as
  `Error in StandaloneSqlite worker` (`INTERNAL_SERVER_ERROR`), so only
  ClickHouse-backed gateways returned the validation error. The cursor is now
  checked before the query reaches the worker.

- **Client addresses from proxy headers could be forged** — the gateway took
  the leftmost `X-Forwarded-For` address (or `X-Real-IP`) as the client,
  whoever sent it, and exempted a request when *any* address in its headers
  was allowlisted. A client could claim an allowlisted address and skip the
  rate limit and x402 payment, or claim a new address on each request for a
  fresh rate-limit bucket (and free allowance) every time. Proxy headers are
  now believed only from proxies in the new `TRUSTED_PROXIES` (default:
  loopback, private, carrier-grade NAT and link-local addresses, where nginx
  normally sits); behind one, the client is the nearest `X-Forwarded-For` hop
  that is not a trusted proxy. Allowlists (`RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST`,
  `CHUNK_INGEST_CACHE_ALLOWLIST`) are checked against that address only.
  **Behind a CDN or a load balancer on public addresses, add its ranges to
  `TRUSTED_PROXIES`**, or every client behind it shares its address; the same
  goes for nginx on a different public host than Envoy. Envoy now appends the
  address that connected to it to `X-Forwarded-For` (`use_remote_address`,
  keeping the downstream `X-Forwarded-Proto`), so a client reaching port 3000
  directly cannot choose its address either: **upgrade the Envoy image with
  core**. The index-swarm tracker uses the same code for
  `INDEX_SWARM_TRACKER_TRUSTED_PROXIES`.

- GraphQL no longer fails with HTTP 500 ("Cannot execute GraphQL operations
  after the server has stopped") while the gateway shuts down. Apollo's own
  signal handler used to stop GraphQL while the listener was still accepting
  requests; the listener now closes first. Under load, a `docker stop` of the
  previous release answered about 27,000 requests with that error.

- `scripts/clickhouse-import` defines its functions again when sourced. The
  guard that skips the CLI returned before any function was defined, so tests
  that source it for `migrate_staging_to_final` failed with "command not
  found". `scripts/tests/parquet/test-clickhouse-ttl-rules` also placed `FINAL`
  after `WHERE`, a syntax error; it now passes end to end.

- The `tx-data` retrieval source no longer treats an unmined transaction as
  data. A node answers `202 Pending` for a transaction it has not mined, and
  that text was decoded as the transaction's data with a `NaN` size, which then
  broke x402 pricing, let the request skip rate-limit token accounting, and
  made a metrics call throw from inside the stream's `end` handler. The source
  now accepts only `200` answers, requires a whole-number `data_size`, and
  rejects data whose length differs from it, so the request moves on to the
  next source.
- The trusted-gateways root TX lookup no longer records a 1-byte payload size
  when a gateway rejects its HEAD request. The lookup then falls back to a
  `Range: bytes=0-0` GET, whose `Content-Length` is 1; that value was taken as
  the item's payload size, so the item could be served, cached and recorded as
  a single byte. The size now comes from the `Content-Range` total, and is left
  unknown when the response has none, in which case the item is located by
  searching its bundle instead.
- Payloads located by a direct offset hint (`X-AR-IO-Root-Item-Offset` +
  `X-AR-IO-Root-Item-Size`) are now checked against the data item's signature
  before they are served in full, cached, or have their offsets saved.
  Previously the hinted size was taken as given, so an incorrect size could
  frame a truncated or over-long payload under that ID. Range requests carrying
  these hints now resolve through the bundle's own index, and the two hint
  headers are no longer forwarded to other gateways; root transaction and path
  hints are unchanged. Outcomes are counted in
  `data_item_signature_verification_total`.

- Stopped GraphQL returning every tag twice for a recently uploaded data item.
  Optimistically indexed data items are written with a NULL
  `root_transaction_id`, which is part of the `new_data_item_tags` primary key.
  SQLite treats those NULLs as distinct, so the later unbundle write added a
  second tag set instead of replacing the first, and the tag lookup returned
  both. The unbundle path now removes the optimistic tag rows, and the
  optimistic path replaces its own rows rather than stacking a second copy.

- **Data items were served with their bundle's content type.** When an item's
  own content type was unknown, the gateway reported the bundle's
  `application/octet-stream`, so HTML pages downloaded instead of rendering.
  That type was then stored against the data hash and kept for the item, and
  for any byte-identical re-upload, for good. Now:
  - an item's content type is left unset rather than taken from its bundle;
  - a stored `application/octet-stream` is replaced once a real content type
    is known (never the reverse);
  - the GraphQL root TX lookup reads the `Content-Type` tag when an
    indexer's `data.type` is null;
  - an incomplete in-memory attributes entry expires instead of hiding the
    stored content type until restart.

- **Another item's bytes could be served under the requested ID** (#937). An
  item signed the same way can exist in several bundles, and a stored location
  or a root TX index result could pair one copy's root with another copy's
  offset. Before reading from such a location, the gateway now reads the item
  header there and checks that its signature matches the requested ID. A
  location that fails is not used, and index offsets are saved only after they
  pass. Outcomes are counted in `data_item_location_check_total{source,result}`.

- **Requests over their rate limit could be served free** when their x402 price
  could not be quoted. Prices were passed to x402 as a 3-decimal dollar
  string, so any price under $0.0005 became `$0.000`, which x402 rejects (at
  $0.045/GiB with a $0.0001 minimum, every item under about 12 MB), and other
  prices could come out as fractional amounts no client can pay. Prices are now computed in
  whole atomic units of the asset, and a price that still cannot be quoted
  gets a 429 instead of free access. `/ar-io/info` also advertised any
  per-byte price below 1e-10 USDC as `0`; it now shows the exact value.

- **HTTP signatures now cover every `X-ArNS-*` header.** Five
  (`X-ArNS-Basename`, `-Record`, `-Resolved-At`, `-Undername-Limit`,
  `-Record-Index`) were sent unsigned on signed ArNS responses.

- **A negative GraphQL `first` removed the page limit.** `transactions(first:
  -5)` ran an unbounded query. `first` is now clamped to at least 1.

- **The gateway could hang at startup with no HTTP listener.** An error thrown
  by a startup dependency before the listener was bound was logged and
  swallowed, leaving the indexing workers running and nothing serving (on one
  gateway, 26 hours after ClickHouse was unreachable at boot). The process now
  exits so its restart policy retries. `clickhouse` and `litestream` also get
  `restart: unless-stopped`; they were the only compose services without it,
  so they stayed down after a host reboot.

- Malformed or oversized request bodies are answered with their 400 or 413
  instead of a logged 500. On `/graphql` the reply uses GraphQL's error shape
  (`BAD_REQUEST`).

- **The chunk eviction index missed chunks cached while serving**, so the
  evictor had almost nothing to delete and the chunk cache was reclaimed only
  by the filesystem cleanup worker. Cached chunks are now indexed. The evictor
  warns when the index tracks too little of the cache to reach its target,
  and `chunk_cache_index_hook_errors_total{hook}` counts failed index writes.
  Chunks cached before the upgrade need the one-off backfill described in
  `docs/cache-cleanup.md`.

- **Chunk metadata requests to peers were doubled** when a peer refused a HEAD.
  Any non-2xx HEAD was repeated as a GET, which returns the same 404, 429 or
  5xx. The GET fallback now runs only when HEAD threw, came back without the
  chunk headers, or answered 400, 403, 405 or 501. Refusals are counted as
  `peer_refused` in `ario_chunk_metadata_anchor_total`. A chunk fetch that
  another source cancelled internally now answers 404, like a timeout, instead
  of 502.

- The header-only `Range: bytes=0-0` fallback used for root TX and chunk
  metadata lookups no longer buffers a whole response in memory when a peer
  ignores the range.

- The peer list is kept when a registry refresh fails. A failed read used to
  replace it with whatever had been fetched, often nothing, so peer retrieval
  failed until the next refresh up to an hour later.

- Manifest index precedence is deterministic: `index.id` wins over
  `index.path` whatever the JSON key order, so gateways agree on the index.
  Entries whose ID is not a valid 43-character ID are ignored.

- CDB64 directory sources: every configured directory is watched, not just
  the first; a directory missing at startup is loaded once it appears instead
  of never; and a replaced or retired index is closed only after the lookups
  already inside it finish, so they no longer miss.

- `export-sqlite-to-cdb64` and the other CDB64 generate tools no longer grow
  memory without bound (about 840 MiB per million rows), no longer hang when
  a partition stream closes during backpressure, and clean up their temporary
  files when a write fails instead of crashing.

## [Release 83] - 2026-09-01

This is a **recommended release** focused on **data-retrieval correctness,
chunk cache management, and decentralized ArNS resolution**. Key highlights
include a fix for truncated upstream bodies being cached and re-served as
complete data — the mechanism by which a one-byte fragment propagates between
gateways — plus enforcement of the blocklist on manifest-resolved paths,
index-driven eviction and backfill for the chunk data cache, coalescing of
concurrent requests for the same uncached object, and ArNS now resolving
on-demand from chain by default so each gateway is self-sufficient rather than
a client of a few trusted gateways. Gateways the registry reports as `leaving`
are no longer used as peers or spent on observation, and the bundled observer
is updated for ADR-0029 epoch rent refunds.

### Added

- **Disk-pressure watermarks for the chunk data cache** —
  `CHUNK_DATA_CACHE_LOW_WATERMARK_PERCENT`,
  `CHUNK_DATA_CACHE_HIGH_WATERMARK_PERCENT`,
  `CHUNK_DATA_CACHE_MIN_FREE_BYTES` and
  `CHUNK_DATA_CACHE_AGGRESSIVE_MIN_AGE_SECONDS` bring the chunk cache cleanup
  walk in line with the contiguous data cache, which already had them. With a
  low watermark set the walk is skipped entirely while the filesystem has
  headroom, instead of running unconditionally. This matters on large caches
  atop spinning storage: the walk is metadata-bound, so a tree with tens of
  millions of inodes cannot be traversed within
  `FS_CLEANUP_WORKER_RESTART_PAUSE_DURATION` — a pass never completes and the
  device stays saturated even with terabytes free. Previously the only escape
  was `ENABLE_CHUNK_DATA_CACHE_CLEANUP=false`, which stops reclamation
  altogether and lets the cache grow unbounded. All four default to the
  existing behavior, so nothing changes unless they are set.

- **Leader re-election on a failed foreground fetch** — a leader that fails
  releases every waiter at once, and each of them then started its own fetch in
  the same tick. That is the same total work as no coalescing at all, delivered
  as one synchronised burst instead of spread across the arrivals that produced
  it -- strictly a worse shape than the problem coalescing was added to fix.

  A waiter released by a *failure* now re-enters with one attempt spent: the
  first back through finds the ID unowned and claims it, and the rest attach to
  that new leader. `FOREGROUND_CACHE_COALESCE_MAX_ATTEMPTS` (default 2, minimum
  1) bounds the chain so a succession of dying leaders cannot park a request
  indefinitely; 1 restores the previous behavior exactly.

  Re-election is deliberately limited to failures. The in-flight promise now
  reports `cached` / `uncached` / `failed` rather than a bare boolean, because
  the old `false` conflated three endings. A leader that succeeded but declined
  to cache (size cap, concurrency cap, zero-length) would be
  followed by a new leader declined by the same policy, and a leader that timed
  out still owns its map entry, so re-attaching would wait on the fetch just
  abandoned. Both send the waiter straight to its own fetch. Re-elections are counted by
  `foreground_cache_re_elections_total`, kept separate from
  `foreground_cache_coalesced_outcome_total` so that counter still records
  exactly one terminal outcome per attached request.

- **A size floor on foreground coalescing** — `FOREGROUND_CACHE_COALESCE_MIN_SIZE`
  exempts objects known to be smaller than it, which then fetch for themselves
  exactly as they did before coalescing existed. Coalescing serves a waiter
  from the blob the leader finalizes, so it costs that waiter the whole
  download in time-to-first-byte. That is worth paying on a multi-gigabyte
  object whose duplicates are measured in gigabytes; it is a poor trade on a
  small one that duplicates cheaply and finishes fast.

  Sized against the production stampede above: objects over 1 GiB were 94.8%
  of the redundant bytes and everything under 100 MiB was 0.2%, so a 100 MiB
  floor there would have retained 99.83% of the reclaimed bytes (223.6 of
  224.0 GB) while leaving about half of all staged objects, by count,
  streaming independently.

  **Defaults to 0 — no floor — so behavior is unchanged unless it is set.**
  The size compared is the one already resolved from the attributes store at
  the point coalescing is decided; `data.size` is not available until the
  upstream fetch returns, which is after a leader has claimed the ID. An
  object of unknown size is therefore treated as eligible, so the floor can
  only narrow coalescing where an object is positively known to be small and
  can never make stampede protection weaker than leaving it unset. Exemptions
  are counted as `foreground_cache_skipped_total{reason="below_coalesce_floor"}`;
  compare it against `already_pending` to judge whether the floor is too high.

- **Bounds on foreground cache writes** — `FOREGROUND_CACHE_MAX_SIZE` and
  `FOREGROUND_CACHE_CONCURRENCY` cap how much unfinished data a burst of
  *distinct* large objects can accumulate in `contiguous/tmp`, mirroring the
  guards background range caching already had. Exceeding either serves the
  request normally and skips only the cache write. The concurrency budget is
  process-wide, shared by the on-demand and background caches, because both
  stage to the same directory.

  **Both default to unbounded, deliberately.** Coalescing alone removes the
  duplication of *identical* objects and needs no configuration, but a burst
  of *distinct* large objects is only bounded once these are set -- upgrading
  does not inherit that protection. Any finite default would silently stop a
  busy gateway caching most of what it serves, which is not a change to make
  on an operator's behalf; set them explicitly per deployment.

- **Index-driven chunk cache eviction and backfill** — the chunk data cache is
  now tracked in a `chunk_data_cache` index, so eviction reclaims by age and
  size against disk watermarks instead of walking the filesystem, and a
  backfill reconciler adopts pre-existing files into the index. Tuned by the
  `CHUNK_DATA_CACHE_INDEX_*` variables (eviction interval, batch size, target
  bytes, unlink concurrency, update-on-read) and
  `CHUNK_DATA_CACHE_AGGRESSIVE_MIN_AGE_SECONDS`.

### Changed

- **ArNS resolves on-demand (from chain) by default** — the default
  `ARNS_RESOLVER_PRIORITY_ORDER` is now `on-demand,gateway`. Each gateway reads
  the ANT record itself and only falls back to a trusted-gateway hop if that
  fails, making gateways self-sufficient rather than clients of a few trusted
  gateways. The bundled observer now references this node's own gateway
  (`ARNS_ROOT_HOST`) by default. `ARNS_COMPOSITE_LAST_RESOLVER_TIMEOUT_MS` drops
  from `30000` to `5000` in the same change: with the gateway resolver now last,
  a name that does not exist previously cost ~3s on-demand plus the full 30s
  budget before 404ing. Restore the old behaviour with
  `ARNS_RESOLVER_PRIORITY_ORDER=gateway,on-demand`.

- **Gateways leaving the network are no longer used as peers** — a gateway is
  `leaving` either because its operator withdrew it or because the network
  marked it non-responsive for 30 consecutive epochs, so it should not be
  receiving requests. Measured on turbo-gateway gw1: 334 of 646 registered
  gateways were `leaving`, and 40% of all peer errors were `ENOTFOUND` against
  hostnames that no longer resolve. Controlled by `SKIP_LEAVING_GATEWAYS`
  (default true); only an explicit `leaving` is excluded, so a registry that
  stops reporting status degrades to the previous behaviour rather than
  emptying the peer list. Counted by `ar_io_peers_skipped_leaving_total`.

- **Bundled observer updated to `0e956b08`** — adds the matching
  leaving-gateway filter on the observation path (about half of each epoch's
  observation budget was spent on gateways that always fail) and bumps
  `@ar.io/sdk` to 4.3.0-alpha.2 for ADR-0029, so `close_epoch` refunds epoch
  rent to the account that created the epoch.

### Fixed

- **Truncated upstream bodies are no longer cached** — `data.size` comes from
  the peer's `Content-Length`, so a gateway serving a fragment reports a size
  matching the bytes it sends: the existing size check passed and the fragment
  was stored as complete, then served on to the next gateway. Retrieved bodies
  are now also checked against the item's indexed ANS-104 payload size, which
  is derived from the bundle header rather than from retrieval. Rejections are
  counted by `short_reads_rejected_total`. The check stands down when those
  attributes are unavailable, so L1 transactions are unaffected.

- **Blocklisted data items are enforced on manifest paths** — an item blocked
  by ID or hash still returned 200 through any manifest or ArNS path that
  resolved to it, while `/raw/<id>` correctly returned 451. The resolved-item
  checks now also respond 503 rather than serving when the blocklist backend
  errors, so an outage cannot silently reopen the bypass.

- **x402 CDP facilitator authentication** — the CDP facilitator was not
  authenticated with its API key id, the fallback to the configured facilitator
  URL failed silently rather than surfacing, a blank credential was treated as
  configured, and the paywall client key was conflated with the facilitator
  credential. Settlement must now be confirmed before a payment is treated as
  successful.

- **Concurrent requests for one uncached object no longer stampede** — on a
  full-object cache miss `ReadThroughDataCache.getData` ran an upstream fetch
  and opened a staging file per request, with nothing checking whether a fetch
  for the same ID was already running. Seen in production as 59 concurrent
  partial copies of one 1.5 GB bundle: 1,434 open descriptors under
  `contiguous/tmp`, ~253 GB staged across only 18 distinct objects of which
  83% was redundant. It is self-amplifying — the duplicated writes saturate
  the disk, so no copy finishes, so every new request is also a miss and
  starts another copy. The first caller for an ID is now the sole owner of the
  fetch, the staging file and the tee; concurrent callers wait on it rather
  than starting their own. When it finalizes they are served from the blob it
  wrote. When it does not -- it failed, or it stalled past
  `FOREGROUND_CACHE_COALESCE_TIMEOUT_MS` (default 300000) -- there is no
  shared blob to serve, so each waiter falls back to fetching independently,
  which is the pre-existing behavior for exactly that case. Those two
  outcomes are visible as `refetched` and `timed_out` on
  `foreground_cache_coalesced_outcome_total`. Waiters hold no reference to
  the shared fetch, so one aborting can neither cancel it nor orphan its
  staging file, and the timeout keeps a stalled fetch from parking later
  requests for that ID indefinitely.

  **Operator note — the cache-miss signal for this failure is gone.** A
  coalesced request is served from the cache, so it now reports the same
  cache-hit semantics as a request arriving a moment later (`X-Cache: HIT`,
  `Content-Digest`, conditional-request eligibility) and counts as a hit
  rather than a miss. Hit-rate dashboards will move, and more importantly
  `contiguous_data_cache_miss_total` no longer rises when many requests
  converge on one uncached object -- which was the signal this failure mode
  used to produce. Use `foreground_cache_skipped_total{reason="already_pending"}`
  to detect recurrence instead: it counts exactly the requests that would
  previously have started a duplicate fetch, so a sustained rise is the
  stampede re-forming. `foreground_cache_coalesced_outcome_total{outcome}`
  breaks those down into `cache_hit` (the leader cached, waiter served from
  disk), `refetched` (the leader cached nothing) and `timed_out` (the leader
  stalled past the bound).

  The fix also **converts** part of the load rather than removing it: the
  leader writes once, then each waiter opens its own read of the finalized
  blob (measured: 49 reads for 50 concurrent requests). That is the same
  shape as N concurrent requests for an already-cached object, and far
  cheaper than N concurrent multi-GB writes, but the reads all begin at the
  moment the leader finalizes rather than being spread over time.

## [Release 82] - 2026-08-13

This is a **recommended release** focused on **data-retrieval correctness and
outbound connection health**. Key highlights include a new
`GET /ar-io/offsets/:id` endpoint that serves root transaction offsets straight
from the local index, with a matching `peers` lookup source so a miss costs the
peer one indexed read instead of a full retrieval cascade; detection and repair
of mis-rooted data items, whose stored root could be an intermediate bundle and
so sent chunk retrieval after chunks that cannot exist; keep-alive pooling and
per-host socket caps for outbound clients, removing a per-request DNS lookup
that queued behind filesystem I/O; configurable SQLite read workers with
queue-wait and slow-query instrumentation; GraphQL routing that steers
owner-filtered and L1-only queries away from ClickHouse row-cap failures; and an
opt-in SSD-resident cache index with a disk-pressure evictor for large
spinning-disk caches. It also fixes an offset-index scan that could pin a read
worker for minutes on a miss, an Envoy route timeout that cut GiB-scale
downloads mid-stream, and a Docker build context that could exhaust the daemon's
disk and take down every container on the host.

### Added

- **Root TX offsets endpoint** — new `GET /ar-io/offsets/:id` serves a data
  item's position inside its root transaction straight from the local index,
  without touching contiguous data. The previous way to ask a peer for offsets
  was `HEAD /raw/:id`, whose headers are a byproduct of a successful retrieval:
  on a cache miss the peer walked its whole `ON_DEMAND_RETRIEVAL_ORDER` cascade
  before answering. Offsets now resolve for any item the node has unbundled and
  indexed, including items with no bytes cached locally. A matching `peers`
  lookup source (`PEERS_ROOT_TX_URLS`, selectable in `ROOT_TX_LOOKUP_ORDER`)
  consumes it, with priority tiers, per-peer rate limits, and a shared LRU.
  There is no `HEAD /raw` fallback by design — compose one explicitly with
  `db,peers,gateways,...` (#837, PE-9135).
- **SSD-resident cache cleanup index and disk-pressure evictor** — an
  alternative reclaimer for the contiguous data cache that keeps eviction
  discovery off the HDD. A `contiguous_data_cache` table records
  `{hash, size, cached_at, last_access, tier}` per cached blob; an interval
  sweep driven by `statfs` evicts least-recently-accessed first, general tier
  before preferred, and drains to the low watermark. Reads refresh
  `last_access` and promote a blob's tier on a preferred-ArNS hit
  (`CONTIGUOUS_DATA_CACHE_INDEX_UPDATE_ON_READ`, default `true`; set `false`
  for FIFO). A resumable one-time backfill adopts a pre-existing cache. Off by
  default (`ENABLE_CONTIGUOUS_DATA_CACHE_INDEX`); see the new
  `CONTIGUOUS_DATA_CACHE_INDEX_*` env vars and `docs/cache-cleanup.md`
  (#823, PE-9131).
- **`PUT /ar-io/admin/unblock-data` admin endpoint** — lifts a data block
  created with `block-data`, by `id` or `hash`. Moderation previously had
  `block-data`, `block-name`, and `unblock-name` but no way to reverse a data
  block short of editing `moderation.db` by hand. Idempotent; blocked data is
  checked per request, so an unblock takes effect immediately (#821).
- **GraphQL `owner_projection` routing** — owner-filtered `transactions`
  queries can be routed through the owner-ordered projection instead of the
  height-ordered main table, where a sparse owner's rows scatter across
  millions of granules and can trip the ClickHouse row cap. Covers no-id owner
  queries (with a reactive height-windowing fallback) and `owners + ids`
  queries. Off by default; see
  `CLICKHOUSE_GQL_OWNER_PROJECTION_ROUTING_ENABLED` and
  `CLICKHOUSE_GQL_OWNER_PROJECTION_ENTITY_TYPES` (#796, #800).
- **GraphQL L1-only query routing** — `GQL_L1_ONLY_ROUTING_FILTER` classifies
  which `transactions` queries are provably confined to the base layer and
  answers them from the SQLite `stable_transactions` tag PK seek, skipping
  ClickHouse and its `max_rows_to_read` cap entirely. Uses the composable
  filter DSL restricted to its monotone subset; `not`, `isNestedBundle`, and
  `hashPartition` are rejected at startup. The filter is an operator assertion
  — routed queries intentionally exclude bundled data-item matches. Default
  `{"never": true}` (off). New `graphql_l1_only_routing_total` metric (#810).
- **Configurable SQLite read workers and queue instrumentation** — reads were
  serialized in one worker thread per pool. `CORE_SQLITE_READ_WORKER_COUNT`
  (default `1`) and `DATA_SQLITE_READ_WORKER_COUNT` (default `2`) add read
  concurrency. Queue wait and service time are now separate metrics
  (`standalone_sqlite_method_queue_wait_seconds` vs `..._service_seconds`), and
  `SQLITE_SLOW_QUERY_LOG_THRESHOLD_MS` (default `1000`) logs slow operations
  with a queue/service breakdown to identify the responsible method during a
  jam (#817).
- **Per-host outbound socket caps and socket-acquisition instrumentation** —
  `GATEWAY_MAX_SOCKETS_PER_HOST` (default `16`),
  `GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST`, and
  `GATEWAY_MAX_FREE_SOCKETS_PER_HOST` (default `4`) bound outbound concurrency
  per gateway host, each accepting a bare integer or a per-host object. Trusted
  and untrusted gateways get separate caps so a CDN-fronted upstream is not
  overwhelmed. New `gateway_socket_acquisition_seconds` and
  `gateway_socket_connect_seconds` metrics (the latter now includes the TLS
  handshake for https gateways) plus a slow-acquisition warning
  (`GATEWAY_SLOW_SOCKET_ACQUISITION_LOG_THRESHOLD_MS`) surface pool waits
  before a request reaches the wire.
- **Connection pooling for non-data outbound clients** — root TX discovery
  sources and the GraphQL fan-out now share keep-alive agents
  (`OUTBOUND_MAX_SOCKETS_PER_HOST`, `OUTBOUND_MAX_FREE_SOCKETS_PER_HOST`). The
  goal is not throughput but avoiding a per-request `dns.lookup()`, which
  queues on the libuv threadpool behind filesystem I/O and can time out before
  a socket opens. Watch `outbound_socket_acquisition_seconds{reused="false"}`
  (#840, PE-9136).
- **Envoy circuit breakers for the core cluster** — `max_connections`,
  `max_pending_requests`, and `max_requests` for `ario_gateways` now default to
  `16384` instead of Envoy's `1024`, with a retry budget
  (`ENVOY_ARIO_GATEWAY_RETRY_BUDGET_PERCENT`, default `20`) replacing the
  default `max_retries` of 3. At Envoy's defaults a slow core pinned the pool,
  further requests failed instantly with `reset reason: overflow` (503), and
  client retries kept it pinned so the cluster never drained (#841).
- **Manifest resolution metrics** — the two resolution sites in the data
  handler are now instrumented. `manifest_resolutions_total{source,
  resolution_type}` splits resolutions by source (`index` = served from the
  resolution index without parsing the body, `data` = on-demand body parse) and
  by outcome (path / index / fallback / unresolved); the index-vs-data ratio is
  the effectiveness signal for the index and its cache.
  `manifest_unresolved_root_total` counts root/index requests that resolve to
  nothing — a malformed-manifest signal that previously surfaced only as a user
  404. `manifest_resolution_duration_seconds{source}` records latency by source
  (#835).
- **ClickHouse `TOO_MANY_ROWS` observability** — Code 158 responses are now
  logged and counted at the origin, so row-cap failures are attributable to a
  query shape instead of surfacing only as GraphQL errors (#816).
- **`TRUSTED_GATEWAYS_SEND_UNTRUSTED_PARAMS` kill-switch** — set `true` to
  restore the legacy behavior of sending `ar-io-*` provenance query params to
  every gateway (see Changed) (#798).
- **Dedicated Turbo AWS client** — the Turbo S3 data source can use its own
  credentials and endpoint via `TURBO_AWS_REGION`, `TURBO_AWS_ENDPOINT`, and
  the optional `TURBO_AWS_*` credential vars. The client is created only when
  both region and endpoint are set; otherwise the default AWS client is reused
  (#794, PE-9125).
- **Opt-in chunk over-propagation** — `CHUNK_POST_CONTINUE_PAST_THRESHOLD`
  (default `false`) keeps broadcasting a chunk to every selected peer after the
  success threshold is met, maximizing redundancy instead of stopping early. The
  dead-peer tail stays bounded by `CHUNK_POST_MAX_CONSECUTIVE_FAILURES`; pair it
  with a higher `CHUNK_POST_PEER_CONCURRENCY` so broadcasts complete promptly. A
  new `arweave_chunk_post_temporary_total` metric exposes the 200-vs-303
  acceptance split (#819).

### Changed

- **Untrusted gateways no longer receive `ar-io-*` provenance query params** —
  gateways configured with `"trusted": false` in `TRUSTED_GATEWAYS_URLS` now get
  provenance via `X-AR-IO-*` headers only. Required for CDN-fronted gateways
  such as `arweave.net`, whose CDN returns 502 on those query params. Revert
  with `TRUSTED_GATEWAYS_SEND_UNTRUSTED_PARAMS=true` (#798).
- **Local-first root TX offset resolution** — `RootParentDataSource` now
  resolves offsets from the local index before consulting remote sources, and
  the composite lookup short-circuits as soon as a source returns an actionable
  result instead of polling the rest of `ROOT_TX_LOOKUP_ORDER`. Callers can pass
  an `accept` predicate to define what counts as actionable (#827, #828, #829,
  PE-9134).
- **Offset-to-block resolution runs locally** — locating the block that
  contains an absolute offset used a chain binary search of roughly
  `log2(height)` sequential `GET /block/height/{h}` calls to the trusted node
  (~1.5s each), often exceeding `CHUNK_SERVE_DEADLINE_MS` and returning 504.
  A local index over `stable_blocks.weave_size` is consulted first and trusted
  only under a tight bracket, with re-verification and a fallback to the chain
  search on any gap, stale index, or unstable-tip offset. The block returned is
  identical. New `block_offset_resolution_total` metric (#801, #807).
- **Idle outbound gateway sockets are retired before the peer closes them** —
  `GATEWAY_AGENT_IDLE_SOCKET_TIMEOUT_MS` (default `50000`) must stay below the
  peer's server keep-alive timeout (`HTTP_KEEP_ALIVE_TIMEOUT_MS`, default
  `60000`). Reusing a socket the server is simultaneously closing caused ~8-10s
  peer-fetch stalls.
- **Chunk-post fan-out narrows to the configured threshold** — now that HTTP 303
  counts as a successful (temporary) acceptance (see Fixed), broadcasts stop at
  `CHUNK_POST_MIN_SUCCESS_COUNT` as intended instead of grinding the full peer
  list. Operators who relied on the prior 303-handling bug's accidental wide
  spread can restore it with `CHUNK_POST_CONTINUE_PAST_THRESHOLD=true` (#819).
- **Default observer image bumped to `15e285b0`** — `OBSERVER_IMAGE_TAG` moves
  from `308b6777` (2026-06-20) to the current `ar-io-observer` release build,
  picking up the full ArNS lease lifecycle in the epoch cranker, adaptive
  cranker poll/cleanup intervals derived from epoch duration, the
  `LeaveWindowNotExpired` (6079) not-ready classification, failed-gateway
  summary attribution fixes, and `@ar.io/sdk` 4.1.0 (stable, mainnet). It also
  raises ArNS prune throughput: `prune_name_to_returned` is the only
  deadline-bound step in the lease lifecycle, and draining one name per scan
  capped conversion at roughly the rate leases expire, so any backlog was
  permanent and each name aging out of its return-auction window lost that
  auction for good. `CRANK_POLL_INTERVAL_MS` and `CLEANUP_MIN_INTERVAL_MS` are
  now optional — unset, the cranker derives them from the epoch duration; an
  explicit value still wins. Those two plus the new
  `CLEANUP_TO_RETURNED_TXS_PER_CYCLE` (default `10`) are now forwarded to the
  observer container so they can be set from `.env`; leaving them empty keeps
  the derived default. Operators who pin `OBSERVER_IMAGE_TAG` in `.env` must
  update it there too, since that shadows the compose default (#842, #845).
- Test files are now typechecked in CI (#809).

### Fixed

- **Background verification no longer withholds unmined data when optimistic
  indexing is off** — the serving guard withheld verification whenever the root
  transaction had a `NULL` height, reading that as "optimistically indexed, not
  yet mined". That meaning only holds when optimistic L1 transaction indexing is
  what creates `NULL`-height rows. With `OPTIMISTIC_TX_INDEXING_ENABLED` false
  (the default), a `NULL` height means only that this node has not imported the
  transaction's block — routine for transactions indexed via
  `admin/queue-tx`, backfills, or any selectively synced deployment. The result
  was permanent, silent starvation: withholding burns no retry, so the item
  never aged out, and the log is debug-level, so nothing surfaced. The guard is
  now gated on the feature that justifies it (#855).
- **Observer cranker settings reachable from `.env`** — `CRANK_BATCH_SIZE`,
  `CRANK_CLOSE_EPOCHS`, `CRANK_EPOCH_RETENTION`, `CRANK_WARN_BALANCE_SOL`,
  `CRANK_CRITICAL_BALANCE_SOL`, `CLEANUP_BATCH_SIZE`,
  `CLEANUP_FAILURE_THRESHOLD`, `MAX_CLEANUP_TXS_PER_CYCLE`,
  `ALT_RECLAIM_SCAN_LIMIT` and `OBSERVED_GATEWAY_HOSTS` are read by the observer
  but were never passed into its container, so setting them in `.env` had no
  effect. Forwarded with the established empty-default pattern and documented,
  completing what #842 (poll/cleanup intervals) and #846 (prune budget) started.
  An empty value resolves to each setting's existing default, so nothing changes
  unless an operator sets one (#857).
- **Chunk POST treats HTTP 303 ("temporary") as success** — Arweave tip/ingress
  nodes return 303 when they validate and persist a chunk into their disk pool
  without being its long-term home (`ar_disk_pool:add_chunk/6 -> temporary`).
  Previously counted as a failure, which returned spurious errors to uploaders,
  undercounted broadcast success, and (since 303 is not a 4xx) defeated the
  consecutive-failure early exit — grinding the full non-preferred peer list
  (p95 ~33s). 303 now counts toward the overall and preferred success
  thresholds (#819).
- **Chunk metadata cache keyed by the full data root** — metadata was bucketed
  by only the first four base64url characters of the data root plus the relative
  offset, so transactions whose data roots share a 4-character prefix shared
  slots. Relative offset 0 collides constantly, and the read path served one
  transaction's `data_path` for another, failing merkle validation with
  "Failed to parse data_path: invalid proof" until the entry was removed by
  hand. The full data root is now in the path, with a read-time `data_root`
  check that deletes mismatches and treats them as misses (#820, PE-9129).
- **Zero-length chunk cache poison is rejected and self-healed** — nothing
  validated chunk length, so a source that once returned an empty chunk was
  persisted and re-served forever: `has()` reported a hit on the 0-byte file,
  `get()` served an empty chunk, and the streaming loop never advanced,
  re-requesting the same offset indefinitely. Every layer now refuses to persist
  and refuses to serve zero-length chunks, deleting the poisoned file on read.
- **Chunk streaming guarded against non-terminating loops** — a stream that
  stops making forward progress now fails instead of spinning (#799, PE-9127).
- **Optimistic chunk confirmation is sticky per `data_root`** — confirmation
  was a one-shot `UPDATE` fired by `TX_INDEXED` that only touched placements
  present at that instant. A multi-GB bundle streams its chunks in over a far
  longer window, so most arrived afterwards, were never confirmed, and hit the
  ingest TTL — leaving a gappy, unservable set that fell through to peers which
  did not yet have the bundle. Chunks ingested after the confirm event now
  self-confirm, with markers pruned by a GC sweep
  (`CHUNK_INGEST_CONFIRMED_ROOT_RETENTION_SECONDS`, default `3600`) (#815).
- **`getTxByOffset` misses no longer scan the offset index to the end** — the
  span predicate sat inside the index scan, so a miss (an offset in a coverage
  gap) walked `stable_transactions_offset_idx` from the offset to the end of the
  table with a row fetch per entry — tens of seconds to minutes on a
  production-sized DB (up to 110s observed), each one pinning a read worker and
  holding the requesting peer's socket. A miss now costs the same single index
  probe as a hit (#818).
- **`MAX(stable_blocks.block_timestamp)` no longer full-scans** — the query ran
  over ~2M rows (~900 MB) with no index. Called infrequently, the pages fell out
  of the page cache between calls and the scan became disk-bound, holding the
  core read worker — and, inline in `saveBlockAndTxs`, the write worker — for
  8-13s on busy hosts. Now a covering reverse index seek (#822, PE-9130).
- **Apex `/raw` no longer times out mid-stream** — the `root_service` catch-all
  inherited Envoy's default 15s route timeout, an absolute cap on total request
  duration regardless of whether the body was still streaming, so large objects
  streamed to normal-speed clients were reset (HTTP/2 `RST_STREAM CANCEL`, curl
  92). Clients finishing under 15s were unaffected, which made it look
  intermittent. `/raw/` now gets a dedicated `timeout: 0s` route mirroring the
  sandbox data route; the catch-all keeps its finite default for small endpoints
  (#826, PE-9132).
- **Truthful `hasNextPage` when id-dedup collapses a full ClickHouse page** —
  windowed `transactions` queries could return a partial page with
  `hasNextPage: false` and no error, silently stranding every subsequent page.
  The ClickHouse legs fetch `pageSize + 1` rows with a full-key `LIMIT 1 BY`,
  but the composite deduped by `id` alone; stale rows sharing an `id` and height
  while differing on `block_transaction_index` collapsed only in the id-dedup,
  making a full leg look short (#792, PE-9124).
- **`bundledIn` queries are never routed to the L1-only path** — routing them
  would drop every data-item result the query exists to return (#810).
- **Mis-rooted data items are detected and repaired on retrieval** — a stored
  root transaction ID is not always an L1 transaction. When the parent chain
  was incomplete as offsets were computed, the traversal stopped at the first
  ancestor it held no attributes for — often an intermediate bundle — and
  persisted that as the root; the pre-computed short-circuit then returned it
  forever without validating it. Chunk retrieval requires an L1 transaction, so
  a bundled root sent `TxChunksDataSource` after chunks that cannot exist,
  polling up to `ARWEAVE_PEER_CHUNK_GET_MAX_PEER_ATTEMPT_COUNT` peers at ~90s
  per request before the tier gave up and the request 404'd. The stored root is
  now validated locally and, when itself bundled, walked to the real root with
  offsets rebased by each parent's payload offset. The correction is persisted
  only when the chain resolves fully to an L1 transaction, and the traversal
  fallback no longer persists a root inferred from an unindexed ancestor — that
  guess is what mis-rooted items in the first place. A chain that cycles
  discards the partial rebase and returns the stored pair unchanged, since a
  half-rebased root is worse than an uncorrected one. New
  `root_tx_stored_root_rebased_total{outcome}` metric with `resolved`,
  `incomplete`, and `lookup_failed` outcomes (#843).
- **Attribute-write dedupe keyed on the root coordinates** —
  `saveDataContentAttributes` deduped on the data item ID alone with a 7-minute
  TTL, so whichever writer arrived first won and every later write for that ID
  was discarded regardless of what it carried. A retrieval that beat
  `RootParentDataSource` re-wrote the existing root attributes unchanged and
  claimed the slot, so a corrected root arriving inside the same window was
  thrown away and the row stayed mis-rooted until the entry expired. The key
  now includes the three root fields, so a write that changes nothing is still
  suppressed while a genuine correction always reaches the queue. Observed on a
  canary gateway: 346 suppressed duplicate calls in ~40 minutes, and a
  confirmed rebase onto the L1 root dropped because an unchanged write had
  landed 65 seconds earlier. `clearDataHash` now invalidates every dedupe key
  belonging to an item, so the re-save after a verification mismatch is no
  longer suppressed and the row is not left with a null hash (#843).
- **Git worktrees excluded from the Docker build context** — `./tools/wt add`
  creates worktrees under `wt/`, each with its own `node_modules` and `data/`,
  and nothing excluded them. Measured on one gateway, `wt/` alone was 31 GB
  across 9 worktrees, taking the build context from ~2.5 GB to over 30 GB;
  combined with accumulated build cache that exhausted the daemon's disk
  part-way through `COPY`, which crashed dockerd and SIGKILLed every running
  container. `COPY . .` drops to ~0.4s on that host after the change. `logs/`
  is deliberately *not* excluded: `logs/.gitkeep` is tracked and the winston
  file transport does not create the directory, so excluding it breaks running
  the test suite inside a container (#844, #846).
- Outbound keep-alive agents created by a client are now destroyed on cleanup,
  and free sockets default to the active cap so pooled connections are actually
  reused (#840, PE-9136).

### Removed

- Dead `ARWEAVE_PEER_CHUNK_POST_*` env vars, which had no remaining effect
  (#795).

## [Release 81] - 2026-06-20

This is a **recommended release** focused on **optimistic ingest and
data-serving reliability**. Key highlights include an optimistic chunk ingest
cache that verifies and serves freshly-posted chunks from local storage before
they are fetched upstream, and optimistic L1 transaction indexing that makes
signed transactions queryable before they mine (both off by default); admission
control — depth-based backpressure and per-request batch caps — on the admin
ingest endpoints so a high-volume bundler gets retryable responses instead of
overrunning the indexer; a wall-clock deadline on chunk serves so slow
retrievals fail fast as 404s rather than hanging; and a GraphQL owner-key cache
with in-flight request coalescing plus batched root-tx lookups to cut upstream
rate-limiting.

### Added

- **Optimistic chunk ingest cache** — verify and optimistically cache chunks on
  `POST /chunk`, then serve and unbundle them from local cache before fetching
  upstream. Unconfirmed chunks are reclaimed by a GC sweep and bounded by a
  synchronous pending-bytes disk cap. Off by default
  (`CHUNK_INGEST_CACHE_ENABLED`); see the new `CHUNK_INGEST_*` env vars.
- **Optimistic L1 transaction indexing** — new admin endpoint
  `POST /ar-io/admin/queue-optimistic-tx` indexes signed L1 transaction headers
  before they mine, making them queryable immediately; a serving guard withholds
  `verified` until the transaction is mined. Off by default
  (`OPTIMISTIC_TX_INDEXING_ENABLED`); see the new `OPTIMISTIC_TX_*` env vars and
  MADR 004.
- **Admission control on `POST /ar-io/admin/queue-data-item`** — a per-request
  batch-size cap (`400`) and indexer-depth backpressure (`503` + `Retry-After`),
  with the body limit raised to 10 MB, so a bundler burst gets a retryable
  response instead of unbounded queue growth. New `QUEUE_DATA_ITEM_MAX_BATCH_SIZE`
  / `QUEUE_DATA_ITEM_BACKPRESSURE_DEPTH`; `data_item_queue_rejected_total{reason}`
  metric.
- **Wall-clock deadline for chunk serves** — chunk retrieval is bounded by a
  wall-clock deadline; timeouts return `404` instead of hanging (PE-9121).
- **GraphQL owner-key cache** — address-keyed owner-key cache with an in-flight
  request coalescer, plus observability metrics (PE-9120).
- **Batched GraphQL root-tx lookups** — discovery batches root-tx lookups to
  avoid upstream rate-limit 404s (PE-9108).
- **Parquet export resilience** — DuckDB sorts spill to disk to survive dense
  partitions, with configurable DuckDB memory/thread limits (wired into compose)
  and real export errors surfaced instead of swallowed.

### Changed

- Aligned keepalive timeouts across envoy and core to prevent mid-request
  connection resets (PE-9122).
- Envoy chunk-retry behavior — conservative retry on the chunk route; stop
  retrying chunk GET/HEAD serves against a single-endpoint core (PE-9121).
- Decoupled contiguous-cache cleanup initial delay from the cleanup threshold
  (PE-9106).
- Bounded unbundling retries and cooldowns to prevent runaway loops.
- Observer now receives assessment-concurrency and log-report-sink config.
- Updated the bundled observer image to a stable `@ar.io/sdk` (`4.0.3`) build,
  including the Solana epoch-cranker `finalize_gone` window-eligibility fix and
  the report-sink failure-threshold adjustment.

### Fixed

- Chunk-retrieval 5xx classification and terminal error handling so serve
  failures surface cleanly (PE-9121).
- GraphQL data-item `owner.key` misses now coerce to a NOT_FOUND sentinel
  instead of erroring.
- Validate numeric env vars (optimistic-tx, retry caps, root-tx batch) at parse
  time to guard against NaN misconfiguration.

## [Release 80] - 2026-06-06

This is a **recommended release** focused on the **AO → Solana
migration** of AR.IO protocol state, **Solana-native observation and
HTTPSIG signing**, and **data-integrity hardening**. Key highlights
include reading the Gateway Address Registry, ArNS/ANT records,
epochs, and prescribed observers from Solana on-chain programs via
`@ar.io/sdk` 4.0.0 (#709, #765); a Solana observer that signs and
submits `save_observations`, uploads its report bundle, and can run
the permissionless epoch cranker (#709); response trust headers
signed directly with the observer's Solana key (#758); mainnet
program IDs defaulted so a fresh mainnet deploy starts the observer
out of the box (#766); the Solana 4.0.0 observer container wired in
as the default image (#767); and a fix preventing L1 transactions
with an empty/NULL `data_root` from serving unrelated content (#755).

### Added

- **Solana protocol backend — AO → Solana migration (#709)**: The
  gateway now reads all AR.IO protocol state — the Gateway Address
  Registry, ArNS names and ANT records, epochs, prescribed
  observers, and observation status — from Solana on-chain programs
  via `@ar.io/sdk`'s Solana backend, replacing the AO compute-unit
  reads. Configured by `SOLANA_RPC_URL` (defaults to mainnet-beta;
  use a dedicated provider in production) and the `ARIO_CORE_/GAR_/
  ARNS_/ANT_PROGRAM_ID` env vars (default to the mainnet program
  IDs). The `OnDemandArNSResolver` routes ANT lookups through
  `SolanaANTReadable`.
- **Solana observer, cranker, and report submission (#709)**: The
  observer signs and submits `save_observations` from a Solana
  keypair (`SOLANA_KEYPAIR_PATH` / `OBSERVER_KEYPAIR_PATH`), uploads
  its report bundle to the permaweb under a separate upload identity
  — Arweave, Ethereum, or Solana, via `ARWEAVE_UPLOAD_KEY_FILE`,
  `ETHEREUM_UPLOAD_PRIVATE_KEY*`, or `SOLANA_UPLOAD_KEYPAIR_PATH` —
  and optionally runs the permissionless epoch cranker
  (`ENABLE_EPOCH_CRANKING`, default `false`).
- **HTTPSIG signing with the Solana observer key (#758)**: Response
  trust headers (RFC 9421) are signed directly with the observer's
  Solana keypair; verifiers derive the Solana address from the
  `keyId` and look it up in the on-chain GAR, so no separate
  attestation document is needed. The key can be supplied as a file
  (`OBSERVER_KEYPAIR_PATH`) or an inline base58 secret
  (`OBSERVER_PRIVATE_KEY`).

### Changed

- **`@ar.io/sdk` → stable `4.0.0` (#765)**: Moves off the
  `4.0.0-solana.*` prereleases (#761, #763) to the published stable
  release, which includes the compound crank-step
  transaction-size fix (`ar-io/ar-io-sdk#670`) that unwedges epoch
  progression on populated networks.
- **Mainnet program IDs by default (#766)**: `docker-compose.yaml`
  now defaults the four `ARIO_*_PROGRAM_ID` vars to the mainnet
  program IDs so a fresh mainnet deploy starts the observer without
  extra configuration — the observer requires them set and will not
  start otherwise. Operators on devnet/staging override via `.env`.
- **Default observer image → Solana 4.0.0 container (#767)**:
  `OBSERVER_IMAGE_TAG` now defaults to the Solana-track observer
  build.
- The gateway is now **Solana-only** for registry/ArNS/epoch
  reads; the AO compute-unit dependency for protocol state has been
  removed.

### Fixed

- **`attachStallTimeout` timer leak**: Unref the stall and
  wall-clock timers so a settled stream no longer keeps the event
  loop alive.
- **Over-eager source-cascade abort (PE-9108)**: Abort the data
  source cascade only on a genuine client disconnect, not on
  internal stream transitions, so legitimate retrievals are no
  longer cut short.
- **Empty/NULL `data_root` could serve unrelated content (#755)**:
  An L1 transaction with an empty or NULL `data_root` matched any
  `data_roots` row keyed on an empty value, returning an unrelated
  file's hash; the unguarded insert also planted such poison rows.
  Both queries are now guarded, the insert is a no-op for empty
  input, and a forward-only migration removes any previously
  planted rows.

## [Release 79] - 2026-05-27

This is a **recommended release** focused on **ANS-104 unbundling
hang prevention**, the **ClickHouse streaming pipeline for the
unstable head**, and **byte-range / partial-content hardening**.
Key highlights include four new wall-clock-cap and timeout layers
across the unbundling pipeline (#744, #746, #748, #754) that close
every observed AbortSignal-immune hang path — workers can no longer
wedge permanently inside `DataImporter.download`,
`Ans104Parser.parseBundle`'s `getData`, the stream-to-disk pipeline,
or the worker thread itself; a new opt-in **ClickHouse streaming
pipeline** (`CLICKHOUSE_STREAMING_ENABLED`, #699) backed by
`new_blocks` / `new_transactions` ClickHouse tables so GraphQL
queries against the unstable head no longer wait for the hourly
parquet round-trip; and a **Range / 200-acceptance hardening pass**
(PE-9098) that gives the gateway a useful fallback when upstreams
strip Range headers, bounds overstreaming via
`GATEWAYS_RANGE_ACCEPT_200_MAX_OFFSET`, and corrects byte-count
recording on the sliced path. Other notable additions: configurable
`DATA_ITEM_INDEXER_WORKER_COUNT` (#747) for indexer-bound workloads,
a **chain-anchored chunk metadata fast path with outbound hint
propagation** (#705), and a substantial new observability surface
for the data-importer phase counters, bundle download
timing/size, and bundle-repair-worker. Operationally significant
fixes: a `ReadThroughDataCache` source-stream tee that eliminates a
backpressure race wedging the ar-io-network backfill (#737), an
`ar-io-data-source` limiter-slot leak on stream 'end' (#735), an
HTTPS-SNI bug in `DnsResolver` breaking the chunks-offset-aware
path (#729), a content-type predicate that crashed on stored
`null` content-types (PE-9099), and GraphQL federation
sort-order / null-height regressions (PE-9092).

### Added

- **ClickHouse Streaming Pipeline for the Unstable Head (#699,
  #696)**: Opt-in pipeline that mirrors the SQLite `new_*` tables
  into ClickHouse so GraphQL queries against the unstable head can
  read from ClickHouse directly instead of waiting for the hourly
  parquet round-trip. Adds `new_blocks` and `new_transactions`
  tables (mirroring `transactions` shape, with inline `signature` /
  `owner` and a uniform `inserted_at`-anchored TTL replacing the
  stable table's offset/size/bloom/partition machinery). Reorgs
  trigger bounded `ALTER TABLE ... DELETE WHERE` on the `new_*`
  tables. Activated by `CLICKHOUSE_STREAMING_ENABLED` (default
  `false`); tunable via `CLICKHOUSE_STREAMER_BATCH_SIZE` (default
  500), `CLICKHOUSE_STREAMER_FLUSH_INTERVAL_MS` (default 1000),
  `CLICKHOUSE_STREAMER_QUEUE_MAX_SIZE`, and
  `CLICKHOUSE_NEW_TX_TTL_MINUTES` (default 240). GraphQL gains a
  third merge leg over `new_transactions`; the SQLite leg can be
  reduced to a tight-timeout fallback via
  `CLICKHOUSE_GQL_SKIP_SQLITE_READS` with the timeout governed by
  `CLICKHOUSE_SQLITE_FALLBACK_CIRCUIT_BREAKER_TIMEOUT_MS`. Also
  adds `HTTPSIG_BODY_DIGEST_BUFFER_MAX_BYTES` (default 2 MiB) — the
  upper bound for buffering small uncached bodies to emit a
  `Content-Digest` header; larger bodies stream without one. When
  `CLICKHOUSE_STREAMING_ENABLED=false` (the default), behavior is
  identical to the pre-streaming two-leg path.

- **ANS-104 Unbundling Hang Prevention (#744, #746, #748, #754)**:
  Four cooperating layers that close every observed AbortSignal-immune
  hang path. (1) `DATA_IMPORTER_DOWNLOAD_TIMEOUT_MS` (default 20 min,
  #744) caps `DataImporter.download` via `Promise.race` independent
  of AbortSignal — fixes wedges where 32 of 32 download workers
  pinned indefinitely on backpressured streams. (2)
  `ANS104_UNBUNDLE_GET_DATA_TIMEOUT_MS` (default 30 s) +
  `ANS104_UNBUNDLE_STREAM_TOTAL_TIMEOUT_MS` (default 2 min) (#746)
  bound the parser's data-fetch and stream-to-disk phases with
  AbortSignal-based timeouts; the offset source's reader is
  rewritten to consume via `getReader()` so stream errors and
  aborts reject the promise instead of hanging. (3)
  `ANS104_PARSE_JOB_TIMEOUT_MS` (default 10 min, #748) covers the
  remaining case where a worker thread stays alive but never posts
  a terminal message — fires `worker.terminate()` and the existing
  `'exit'` handler reaps and respawns. (4)
  `ANS104_UNBUNDLE_GET_DATA_WALL_CLOCK_TIMEOUT_MS` (default 5 min,
  #754) is the parseBundle-level mirror of #744's
  `Promise.race` cap, closing the ~0.4 % of cases where the
  cascade ignores the AbortSignal. New metrics:
  `ans104_parser_get_data_wall_clock_fires_total`,
  `ans104_parser_job_timeouts_total`,
  `data_importer_worker_phase_total{phase="timer_*"}`,
  `bundles_unbundle_started_total`, `bundles_unbundle_in_flight`,
  `ans104_parser_jobs_started_total`,
  `ans104_parser_worker_pool_size`,
  `ans104_parser_worker_exits_total`. Also adds `STREAM_REQUEST_TIMEOUT_MS`
  (default 15 min) — the underlying wall-clock cap on
  `attachStallTimeout` used by `ArIODataSource` and
  `GatewaysDataSource` to bound paused-stream wedges.

- **Chain-Anchored Chunk Metadata Fast Path (#705)**: Decodes the
  `X-Arweave-Chunk-*` headers that peer gateways emit on
  `/chunk/{offset}/data` into structured chunk metadata, then
  cross-checks every field against the chain — a header that
  disagrees throws `ChainAnchorMismatchError` and the caller falls
  back to the canonical chain lookup, so peer headers are *hints*,
  never silently trusted. When the hint passes anchoring, the
  gateway skips a chain round-trip per chunk and re-emits the same
  headers on its outbound response so downstream consumers can
  anchor in turn. Enabled by default
  (`CHUNK_METADATA_ANCHOR_ENABLED=true`), with
  `CHUNK_METADATA_ANCHOR_REQUEST_TIMEOUT_MS` (default 5000),
  `CHUNK_METADATA_ANCHOR_TX_CACHE_SIZE` (default 1024), and
  `CHUNK_METADATA_ANCHOR_TX_CACHE_TTL_SECONDS` (default 300) as
  tunables.

- **Accept 200 for Range Requests and Slice Locally (PE-9098)**:
  Some upstreams (most often nginx with `proxy_cache` but no
  `slice` module) silently strip the client's `Range` header and
  return a full 200 body. `GatewaysDataSource` previously rejected
  these with `Expected 206`, falling through to a worse source.
  The gateway now accepts the 200, slices locally, and bounds the
  wire-cost via `GATEWAYS_RANGE_ACCEPT_200_MAX_OFFSET` (default
  10 MiB) — when `region.offset` exceeds the cap, the 200 is
  rejected and the next source tier is tried. Guarded against
  `NaN` / negative env values.

- **Configurable `DATA_ITEM_INDEXER_WORKER_COUNT` (#747)**:
  `DataItemIndexer`'s fastq concurrency was hardcoded to 1.
  Operators draining a large failed-bundle backlog can now raise it
  (default 1, backward compatible) to pipeline main-thread JS work
  while the prior `saveDataItem` is in flight to the SQLite worker.
  Upper bound on speedup is bounded by SQLite single-writer
  semantics.

- **Caller-Supplied Content-Type Predicate with Lazy Poisoned-Cache
  Eviction (PE-9099)**: `ContiguousDataSource.getData()` accepts an
  optional `acceptContentType` predicate. When supplied,
  `GatewaysDataSource` rejects upstream responses whose
  `Content-Type` fails the predicate before any bytes are returned
  (the cascade falls through to the next priority tier), and
  `ReadThroughDataCache` lazily evicts cache entries whose stored
  content-type fails: the on-disk blob is deleted and the request
  treated as a cache miss, so the next fall-through fetch heals the
  entry. Closes the long-standing 1134-byte `text/html`
  bundlr-network parking-page poisoning from the Sept-2024 outage
  that the indexer's ANS-104 parser couldn't unbundle.

- **Data-Importer + Bundle-Repair Observability (#728, #736,
  2c51ee6f, 58090309)**:
  `bundle_download_duration_seconds{outcome}` and
  `bundle_download_size_bytes{outcome}` (#728) correlate slow
  downloads with payload size; `data_importer_queue_full_skips_total`
  (#728) makes previously-silent queue-full drops scrapeable.
  `data_importer_worker_phase_total{phase=started|got_data|stream_ended|...}`
  (#736) pinpoints where a wedged worker is stuck (gaps between
  phases are exactly worker-count when the pipeline locks up).
  Bundle-repair gains `bundles_unbundling_backlog` (true backlog
  including bundles awaiting their first unbundle, not just retries)
  alongside the existing `bundle_repair_pending_bundles`.
  `bundles_unbundle_skipped_total{reason="no_workers"|"high_queue_depth"|"queue_full"}`
  (58090309) accounts for each pre-pipeline skip path at
  `Ans104Unbundler.queueItem`.

- **ClickHouse Pipeline Observability (8f9b1151)**: Two new gauges
  make the parquet → ClickHouse staleness gap legible from Grafana.
  `min_stable_data_item_height` exposes `MIN(height)` over
  `stable_data_items` — flat across multiple auto-import cycles
  means the prune is no-op-ing (typically because a backfill keeps
  inserting rows at low heights with an `indexed_at` newer than the
  prune threshold). `clickhouse_max_imported_height` tracks how far
  the auto-import process has advanced — the gap against the SQLite
  stable height is the lag operators actually care about.

### Changed

- **`isAcceptableBundleContentType` Widened (PE-9099)**: The
  bundle content-type predicate now accepts `binary/octet-stream`
  (a legacy MIME synonym of `application/octet-stream` present on
  ~350 rows in production gateway caches) in addition to
  `application/octet-stream`, `application/x-arweave-data`, and
  absent / `null` content-types. Input is normalized with
  `trim()` + `toLowerCase()` so cosmetic upstream variants
  (`Application/Octet-Stream`, leading/trailing whitespace) no
  longer cause spurious cache fall-through. The rejected
  content-type metric label is also stripped of parameters
  (`; charset=…`) to bound Prometheus cardinality.

- **Bundle-Repair Routes Retries Directly to the Unbundler
  (PE-9098)**: `BundleRepairWorker.retryBundles()` previously
  routed every failed bundle through `TransactionFetcher.queueTxId`,
  which is structurally wrong for BDIs (chain nodes don't index
  BDIs) and event-driven for L1s (TX_INDEXED → unbundler
  subscription drops retries silently when the unbundler queue is
  full). Retries now queue directly to the unbundler, with bundles
  that match neither path retained for the next BRW cycle.

- **`selectFailedBundleIds` Skips Non-Bundle Transactions
  (PE-9101)**: The live and backfill queue paths gate on the
  `Bundle-Format` tag, but the admin `/ar-io/admin/queue-bundle`
  endpoint does not (`bypassFilter` defaults to `true`). Non-bundle
  transactions queued through admin landed in `bundles`, never set
  `matched_data_item_count`, and were retried by `BundleRepairWorker`
  forever — each retry failing in the ANS-104 parser with
  `Invalid buffer`. The retry query now excludes rows whose root
  transaction is provably non-bundle (indexed locally and lacking
  `Bundle-Format=binary`); unknown roots stay eligible.

### Fixed

- **`ReadThroughDataCache` Source-Stream Tee (#737)**: On a cache
  miss, `ReadThroughDataCache.getData()` returned the same inner
  source stream to two consumers — the disk-cache `pipeline()` and
  the caller (`DataImporter.download` or the HTTP handler). Pipeline
  managed pause/resume internally; the caller called `.resume()` once
  at startup. When the disk cache paused for a slow write, the source
  went to recv-window-zero on its TCP socket, the peer stopped
  sending, and the worker waited indefinitely for `'end'` /
  `'error'` events that never came. Manifested as bundle-backfill
  wedges 15–30 minutes into every run with
  `BACKGROUND_RETRIEVAL_ORDER=ar-io-network,…`. Fix introduces a
  `PassThrough` that the pipeline tees into so the source has a
  single consumer governed entirely by the disk-pipeline's outcome,
  not by pause/resume races on the underlying `IncomingMessage`.
  Also fixes a `cacheStream` leak when `dataStore.finalize()` throws.

- **`ar-io-data-source` Peer-Limiter Slot Release on Stream 'end'
  (#735)**: The `streamPeerCounts` / `peerRequestLimiter` release
  path listened only on `stream.once('close', …)`. For HTTP
  `IncomingMessage` streams consumed via `pipeline()` under a
  keepAlive `http.Agent`, `'close'` can fire late or not at all —
  the socket is returned to the agent pool without the response
  object being destroyed. Every "successful" download leaked one
  limiter slot. After 15–30 minutes every peer hit `maxConcurrent`,
  `executeHedgedRequest` could no longer dispatch, and 24 download
  workers blocked silently on sources that never returned data. Now
  listens on `'end'`, `'error'`, and `'close'`, whichever fires first.

- **`DnsResolver` SNI Preservation for HTTPS URLs (#729)**: When
  `PREFERRED_CHUNK_GET_NODE_URLS` included HTTPS endpoints (e.g.
  `https://arweave.net`), the DNS resolver overwrote the URL
  hostname with the resolved IP. `fetch()` then sent TLS SNI = IP
  and `Host: IP`, mismatching the server certificate and triggering
  `ERR_TLS_CERT_ALTNAME_INVALID`. Failures bubbled up through
  `ArIOChunkSource` as silent zero-success rates from otherwise
  healthy upstreams. HTTPS URLs now return unchanged from the
  resolver; only HTTP URLs go through DNS substitution.

- **`isAcceptableBundleContentType` Null Safety (PE-9099)**: Stored
  attributes from SQLite surface `NULL` as JS `null`, not
  `undefined`. The predicate's `undefined`-only guard missed it,
  and `.trim()` on `null` threw `TypeError: Cannot read
  properties of null (reading 'trim')` — every cache lookup where
  the stored content-type was `NULL` failed, the unbundle bounced
  back to the repair pool, and post-deploy throughput collapsed.
  Signature widened to `string | null | undefined` with an explicit
  null check.

- **GraphQL Federation Sort-Order + Null-Height Preference
  (PE-9092)**: Two interacting defects in the federation merge.
  (1) `mergeEdges` could emit edges out of sort order when a richer
  duplicate replaced an earlier emission — under `HEIGHT_DESC` the
  merger picked null-height edges first and a later resolved
  duplicate overwrote slot 0, yielding e.g. `[x(100), y(200)]`
  instead of `[y(200), x(100)]`. (2) Dedup in
  `getGqlTransaction` / `getGqlTransactions` could return a
  null-height (optimistic) record over a fully-resolved record for
  the same id when both were present in the peer set. Fix
  collects emissions into a `Map` keyed by `node.id`, prefers
  height-resolved over null-height, and always forwards the full
  block sub-selection upstream so partial selections produce
  consistent block field hydration.

- **Range Path Consumer-Byte Recording on Sliced 200 (PE-9098)**:
  When the 200-with-Range fallback kicks in, the stream is sliced
  to `region.size`, but the stream-bytes total and size histogram
  were recording the upstream `content-length`. Overcounted by up
  to the full body minus `region.size` (e.g. 140 MB instead of
  512 bytes for a signature fetch). Recorded counts are now the
  consumer-visible sliced size.

- **`ReadThroughDataCache` Caller Region Size Through BDI Parent
  Resolution (PE-9098)**: When a requested item resolved via its
  parent's cached blob, the recursive `getCacheData` call replaced
  the caller's `region.size` with the child's full `data_size`
  before handing the region to `FsDataStore`. For BDI-nested items
  that meant opening an `fs.createReadStream` window spanning
  hundreds of MB to multiple GB instead of the few hundred bytes
  the caller wanted. Caller's region is now preserved end-to-end.

- **Webhook Emitter Log Bloat (#727)**: The previous catch block
  passed the entire `AxiosError` object to winston, which
  serialized the underlying keep-alive agent's Timer linked list
  until the circular guard kicked in — producing 2–4 MB log lines
  per failed delivery. At ~30 failures/hour from a single 429
  webhook target, the 100 MB × 5 docker log rotation budget was
  consumed in ~10 minutes and other log lines were evicted.
  Extracts useful fields (status, code, message, truncated body,
  target URL) and logs a structured object; response bodies clamp
  to 500 chars.

- **`min_stable_data_item_height` Gauge Moved to Main Thread
  (0fa21dee)**: `computeDebugInfo()` runs in a SQLite worker
  thread, which has its own `prom-client` registry. The scrape
  endpoint reads the main-thread registry, so a gauge set inside
  the worker never reached it. Now set in
  `StandaloneSqliteDatabase.getDebugInfo()` after `queueRead`
  returns.

- **Streaming Backpressure in `ReadThroughDataCache` Cache-Miss
  Path (4c9d1d13)**: The cache-miss path piped into the disk-cache
  write stream via `pipeline()` and also attached a `.on('data')`
  hashing/byte-counting listener. Two consumers on the same
  readable forced flowing mode and short-circuited pipeline
  backpressure: `cacheStream`'s internal buffer grew beyond
  `highWaterMark` while waiting on disk writes, holding
  multi-MB per concurrent download. At 24+ download workers this
  produced external-memory pressure. Hashing moved into a
  `Transform` so backpressure is preserved end-to-end.

- **`selectFailedBundleIds` Index (855ba8c3)**: The retry-loop
  `SELECT` is now ~200× faster after correcting the index column.
  The pre-existing `import_attempt_last_retried_idx` had been on
  `import_attempt_count` since the Jan 2025 retry-stats refactor,
  but the query orders by `retry_attempt_count` — a different
  column. Replaced with `bundles_active_retry_priority_idx` on
  `(last_fully_indexed_at, retry_attempt_count, last_retried_at)`.
  Live measurement: 4.32 s → 20.5 ms per call.

- **Shared Keep-Alive HTTP Agents in `GatewaysDataSource`
  (f487dcaa)**: `axios.create()` was called per request without
  agent configuration, so every request opened a fresh TCP+TLS
  connection that closed after the response — sending sockets
  through ~60 s TIME_WAIT. Under high `ANS104_DOWNLOAD_WORKERS`,
  500+ closed sockets accumulated in `ss -s`. Per-gateway-URL
  agent cache with `keepAlive: true` eliminates the churn.

## [Release 78] - 2026-05-08

This is a **recommended release** focused on **request-cancellation
plumbing**, **memory-safety hardening across data fetch paths**, and
**indexer backpressure**. Key highlights include **end-to-end
`AbortSignal` threading through GraphQL resolvers, attribute fetchers,
chunk fetches, and the trusted-node request queue** so client
disconnects no longer leave zombie work pinned on shared queues; a
**byte-range / Range-request hardening pass (PE-9081)** that closes
multiple paths through which oversized or truncated upstream responses
silently pinned large `Buffer`s in the external memory pool; and
**indexer backpressure** via batched matched-item draining
(`BUNDLE_DATA_ITEM_DRAIN_BATCH`) and hard caps on the
`DataItemIndexer` / `Ans104DataIndexer` queues with drop-on-full
recovery via `bundle-repair-worker`. Other notable fixes: stale ArNS
resolution-failure caching that poisoned upstream nginx caches with
the "unregistered" placeholder (PE-9072), a `CompositeArNSResolver`
fast-fail fallback gap during AO/CU flaps (PE-9075), a bundles
root-atom consistency bug that crashed the SQLite worker on optimistic
indexing races (PE-9073), an envoy `/tx` prefix routing fix for txids
starting with `tx` (PE-9079), an export-parquet metric-cardinality
bloat fix (PE-9078), an `Ans104OffsetSource` stream-lifecycle leak
(PE-9077), and a runtime dependency declaration for
`@ardrive/turbo-sdk` (PE-9074). Adds a substantial set of new
observability metrics for GraphQL request volume + cancellations,
attribute-fetch source attribution, and Node.js process / event-loop
saturation.

### Added

- **`CACHE_APEX_MAX_AGE` Configuration**: New environment variable that
  bounds the `Cache-Control` `max-age` returned for `APEX_TX_ID` responses
  (default 3600s, 1 hour) and adds the `must-revalidate` directive.
  Operators can now rotate `APEX_TX_ID` without leaving upstream proxies
  serving the previous content for the data-layer cache lifetime
  (potentially up to `CACHE_STABLE_MAX_AGE` with `immutable`). See
  PE-9072.

- **GraphQL Request Cancellation + Source-Attribution Metrics
  (PE-9087)**: A single `AbortSignal` composed from the express request
  close event and a configurable server-side deadline is now threaded
  through every GraphQL resolver and downstream attribute fetcher.
  When a client disconnects (or the deadline elapses), in-flight
  attribute fetches and `arweaveClient` requests cancel immediately
  instead of running to completion against arweave.net while their
  results are discarded. New env var `GRAPHQL_RESOLVER_DEADLINE_MS`
  (default 12000ms; set to `0` to disable) caps server-side resolver
  runtime. New metrics: `graphql_requests_total` (denominator for
  cancellation-rate alerts), `graphql_resolver_cancellations_total`
  with `{reason="client_disconnect"|"deadline_exceeded"}`,
  `attribute_fetch_total` and `attribute_fetch_duration_seconds`
  with `{kind, subject, source, outcome}` labels for end-to-end
  source attribution of L1 attribute fetches. Also fixes a bug where
  `getTransactionAttributes` always returned `owner: null` for L1
  transactions even when the wallet was already cached, forcing every
  owner query to round-trip to arweave.net.

- **Indexer Backpressure: Queue Caps + Batched Matched-Item Drain
  (PE-9089 + PE-9086)**: The unbundler's matched-item firehose now
  buffers in-process and drains in `setImmediate` batches sized by
  `BUNDLE_DATA_ITEM_DRAIN_BATCH` (default 100), guaranteeing
  event-loop turns for SQLite worker replies and other I/O when
  large bundles produce thousands of cross-thread messages per
  second. Buffer depth is exposed as
  `queue_length{queue_name="matchedItemBuffer"}`. The
  `DataItemIndexer` and `Ans104DataIndexer` queues now enforce hard
  caps (`DATA_ITEM_INDEXER_QUEUE_SIZE` and
  `ANS104_DATA_INDEXER_QUEUE_SIZE`, both default 500000; set `0` to
  disable). Non-prioritized items pushed at the cap are dropped and
  counted in `data_items_dropped_total{queue_name}`; the
  `bundle-repair-worker` recovers the dropped items on its next
  cycle, so dropping is a backpressure release valve, not data loss.
  Backpressure and depth checks are now O(1) tracked counters
  instead of linked-list walks.

- **Node.js Process and Event-Loop Observability Metrics**: The
  default `prom-client` collectors are now enabled, exposing
  `process_resident_memory_bytes`, `nodejs_heap_size_*`,
  `nodejs_eventloop_lag_seconds`, `nodejs_gc_duration_seconds`, and
  `nodejs_active_handles_total`. Adds a new
  `nodejs_event_loop_utilization` gauge (0..1, sampled at scrape
  time) — the most reliable signal for detecting main-thread
  saturation, since lag percentiles can read near-zero while the
  loop is fully pegged. Adds a `bundle_data_item_count` histogram
  (buckets up to 5M items) so heap and queue spikes can be
  correlated with the bundle size that triggered them rather than
  with aggregate ingest throughput.

### Changed

- **Manifest Resolution Type Surfaced**: `ManifestResolution` now carries
  an optional `resolutionType` field (`'path' | 'index' | 'fallback'`)
  populated by `StreamingManifestPathResolver`. Used by the data handler
  to apply different `Cache-Control` policies per resolution type — see
  Fixed below. The field is optional so external implementations of
  `ManifestPathResolver` remain compatible.

### Fixed

- **Stale ArNS Resolution-Failure Caching (affects all gateways by
  default)**: `ARNS_NOT_FOUND_ARNS_NAME` defaults to `'unregistered_arns'`,
  so on every failed ArNS resolution the middleware sets `req.dataId` to
  the resolved placeholder and calls `dataHandler` without setting any
  `Cache-Control`. `setDataHeaders` then applied the data-layer ladder —
  most commonly `CACHE_UNSTABLE_TRUSTED_MAX_AGE` (default 12h, but some
  operators run 90d). Result: the "Make this domain space yours"
  placeholder cached upstream (nginx honors upstream `Cache-Control`)
  and downstream long after a name actually registered. Same bug class
  on the `ARNS_NOT_FOUND_TX_ID` and `APEX_TX_ID` branches, and on
  manifest fallback responses where the URL → data-id binding is
  mutable across manifest revisions.

  Fixes:
  - `ARNS_NOT_FOUND_TX_ID` and `ARNS_NOT_FOUND_ARNS_NAME` resolved-404
    responses now emit
    `public, max-age=${CACHE_NOT_FOUND_MAX_AGE}, must-revalidate`
    (default 60s).
  - Manifest **fallback** responses emit the same short `Cache-Control`,
    overriding any longer ANT TTL set by the ArNS middleware. Path- and
    index-resolved manifest responses still inherit the ANT TTL.
  - `APEX_TX_ID` responses are bounded by `CACHE_APEX_MAX_AGE` with
    `must-revalidate` (see Added).
  - `sendNotFound` 404 responses now emit `must-revalidate` instead of
    `immutable`. (PE-9072)

  **Operator one-time sweep:** entries already poisoned in nginx caches
  must be evicted manually — grep cache files for the placeholder's
  `X-AR-IO-Data-Id` (or for the resolved id of `unregistered_arns` on
  default-config gateways) and remove matches.

- **ArNS cached fallback on fast-fail (PE-9075)**:
  `CompositeArNSResolver` previously fell back to a cached resolution
  only when fresh resolution exceeded
  `ARNS_CACHED_RESOLUTION_FALLBACK_TIMEOUT_MS`. When fresh resolution
  returned `undefined` faster than the timeout (names cache miss,
  AO/CU dry-run error swallowed to undefined), the fallback didn't
  fire and the gateway dropped through to `ARNS_NOT_FOUND_ARNS_NAME`
  — serving the "unregistered" placeholder for names with valid
  cached resolutions during AO/CU flaps. Now falls back to the
  cached resolution whenever fresh has no resolved id, matching the
  comment-documented intent. New metric
  `arns_cached_resolution_fallback_on_empty_total` counts how often
  this fires.

- **Byte-Range and Range-Request Hardening (PE-9081)**: Closes a set
  of related defects across the byte-range fetch paths that allowed
  oversized or truncated upstream responses to silently pass through
  to consumers and pin large `Buffer`s in the external memory pool.
  All byte-range sources now enforce a symmetric contract: the
  upstream `content-length` must equal the requested region size, and
  responses that exceed the requested range are truncated by a
  bounded transform that destroys the underlying socket on close.
  `attribute-fetchers` rejects signature/owner attributes whose size
  is `0` or undefined, and `fetchDataFromParent` pre-allocates its
  result buffer and aborts on the first oversize chunk (closing an
  unbounded accumulator path). `contiguous-data-byte-range-source`
  and `http-byte-range-source` now pass `maxContentLength` to axios
  to bound client-side buffering, and `ar-io-data-source` /
  `gateways-data-source` apply matching `206`-when-`Range`,
  `content-length` guards, and per-region byte caps.

- **AbortSignal Threading Through Chunk Fetches and Trusted-Node
  Path (PE-9076)**: When a client request aborted, the cancellation
  signal had no path into `trustedNodeRequestQueue` or the
  bucket-wait loops, so abandoned client requests still issued HTTP
  calls to arweave.net whose responses had nowhere to go.
  `trustedNodeRequest` now checks `signal.throwIfAborted()` at
  entry, after the bucket wait, and between tokens, releasing queue
  capacity immediately on disconnect. A new `abortablePromiseRace`
  helper isolates caller signals from the shared
  `chunkPromiseCache`, so a cancelled caller bails out without
  cancelling the underlying fetch for other waiters.

- **ANS-104 Offsets Stream Lifecycle Repair (PE-9077)**: Parsing
  paths in `Ans104OffsetSource` consume only a bounded prefix of the
  stream returned by `getData` and previously dropped the reference
  without destroying the stream. Under axios `responseType: 'stream'`
  the unread tail stayed pinned in the `IncomingMessage` external
  buffer pool — invisible to V8 — until the underlying socket was
  destroyed by eventual GC or unrelated cleanup. The parsing paths
  in `parseDataItemHeader`, `extractDataItemMeta`, and
  `getDataItemOffset` now explicitly destroy the stream on both
  success and error.

- **Envoy `/tx` Path Routing Fix (PE-9079)**: A redundant `/tx`
  prefix route in `envoy.template.yaml` shadowed the more specific
  `/tx/` rule for transactions whose ids start with the literal
  string "tx". Those requests were being routed to the
  `trusted_arweave_nodes` cluster intended for header-prefixed
  paths instead of the gateway data path, producing incorrect
  cache headers and skipping peer diversity. The redundant route
  is removed.

- **Export-Parquet Job-Status Metric Cardinality
  Normalization (PE-9078)**: The
  `/ar-io/admin/export-parquet/status/:jobId` route was falling
  through to the admin catch-all in the `normalizePath` helper, so
  every `randomUUID`-generated `jobId` became a permanent label
  value on `http_request_duration_seconds`. Combined with
  clickhouse-auto-import's polling cadence, metric cardinality grew
  unbounded over time and inflated scrape latency on indexer
  deployments. The path is now explicitly normalized alongside the
  singleton bundle-status endpoint.

- **Bundles Root-Atom Consistency + Optimistic-Indexing Fix
  (PE-9073)**: Fixes two interacting defects that caused the
  indexer's SQLite worker to crash when the optimistic data-item
  admin endpoint (`/ar-io/admin/queue-data-item`) raced with
  ANS-104 unbundling. The `new_data_items` table enforces a "root
  atom" invariant on eleven bundling-metadata fields (`parent_id`,
  `root_transaction_id`, `root_parent_offset`, `data_offset`,
  `offset`, `size`, `signature_offset`, `signature_size`,
  `owner_offset`, `owner_size`, `signature_type`) — they must move
  together or remain entirely `NULL` together. The admin endpoint
  unconditionally nulled three of them on every re-POST, so
  repeated admin POSTs after an unbundle regressed back-filled
  values to `NULL` and the next flush failed `NOT NULL` checks.
  The admin path now uses a dedicated `insertOptimisticDataItem`
  (`INSERT` with the root atom hardcoded `NULL`) and the unbundler
  uses `upsertNewDataItem` (atomic root-atom `UPDATE` with a
  `COALESCE`-protected safety net). GraphQL signature/owner
  resolvers guard against incomplete root-atom rows and return
  `undefined` with a warning rather than throwing. The
  `/ar-io/admin/debug` SQLite snapshot is now cached for
  `GET_DEBUG_INFO_CACHE_TTL_MS` (default 5 min, set `0` to
  disable), since each call runs unfiltered `COUNT(*)` scans on
  the SQLite worker and frequent polling was monopolizing the
  debug worker.

- **`@ardrive/turbo-sdk` Moved to Runtime Dependencies (PE-9074)**:
  `@ardrive/turbo-sdk` is required at runtime by the HTTPSIG
  attestation upload path (`src/lib/httpsig-upload.ts`) but was
  declared in `devDependencies`. The production Dockerfile's
  `yarn install --production` pruned it, producing a silent
  `MODULE_NOT_FOUND` at upload time and a fallback to the L1
  upload path — which fails when the gateway wallet has no AR
  balance. Now declared as a runtime dependency.

## [Release 77] - 2026-04-24

This is a **recommended release** focused on **cross-gateway GraphQL
fan-out**, **ClickHouse query-path hardening**, and **composite query
resilience**. Key highlights include **`GatewaysGqlQueryable`**, a new
adapter that fans GraphQL queries out to configured upstream
ar-io-node gateways and merges the results — letting a node compose
its local index with broader upstream coverage — and a **parallelized
composite ClickHouse/SQLite GraphQL path** protected by a SQLite
circuit breaker that surfaces `PARTIAL_RESULT` warnings via
`extensions.warnings` instead of silent partials. ClickHouse gets
several query-path improvements: **dropping `FINAL` in favor of
`LIMIT 1 BY` dedupe** to re-enable projection planning, a new
**`owner_address` bloom with projection skipping on tag filters**, a
**`tag_names` / `tag_values` fix for `owner_projection`**, a
**configurable query timeout** (default 3s), and a
**`max_rows_to_read` guardrail** that fails noisy full-scans fast. It
also adds **per-job status tracking** to the Parquet export admin API
and bundles an **Observer update to `ddd3a9c`** with reference-gateway
chunk-header offset validation and continuous-observer reliability
hardening, alongside a set of **ClickHouse auto-import reliability
fixes**.

### Added

- **Fan-Out GraphQL Over Upstream Gateways (`GatewaysGqlQueryable`)**: A
  new `GqlQueryable` adapter fans GraphQL queries out to configured
  upstream ar-io-node gateways and merges the results, letting a node
  act as a thin fan-out proxy or compose its local index with upstream
  sources for broader coverage. Single-record queries use
  first-non-null resolution; connection queries k-way merge by the
  ar-io-node cursor tuple and dedupe by id. Per-endpoint circuit
  breakers isolate slow or failing upstreams. Configured via
  `GATEWAYS_GQL_URLS`; disabled by default.

- **Configurable ClickHouse GraphQL Query Timeout**: The ClickHouse GQL
  backend now applies a configurable timeout both server-side (as
  `max_execution_time`, so ClickHouse aborts runaway queries and frees
  resources) and client-side (as the HTTP `request_timeout`, with a 2s
  grace window so the server-side timeout error surfaces before the
  client aborts). Default 3s.

- **`max_rows_to_read` Guardrail on ClickHouse GraphQL Queries**: Every
  GraphQL query against the ClickHouse `transactions` table now appends
  `SETTINGS max_rows_to_read = N`. Queries that would scan more than the
  configured threshold throw `Code: 158: Limit for rows ... exceeded`
  instead of silently scanning the whole table — catches
  projection-shadowing bugs and planner regressions where a skip index
  is bypassed. Default 10M rows (~20% of current table size); tunable
  via `CLICKHOUSE_GQL_MAX_ROWS_TO_READ`.

- **Per-Job Status Tracking for Parquet Export API**:
  `POST /ar-io/admin/export-parquet` now returns a `jobId`, and the
  exporter keeps a bounded per-job history (32 entries) so concurrent
  callers can each poll their own record at
  `GET /ar-io/admin/export-parquet/status/:jobId`. The legacy singleton
  status endpoint is retained for back-compat and still reflects the
  most-recent update. `scripts/parquet-export` prefers the per-job
  endpoint when a `jobId` is returned and falls back to the
  singleton-with-drift-detection path for older gateways.

### Changed

- **Observer Update to `ddd3a9c`**: Bundles two upstream PRs on top of
  the previous `21098d2` pin.
  - **Reference-gateway chunk-header offset validation**: The observer
    now HEADs the reference gateway's `/chunk/{offset}/data` and anchors
    the advertised `x-arweave-chunk-*` headers (tx id, boundaries, data
    root) to the chain via `/tx/{id}/offset` and `/tx/{id}`, replacing
    the block-and-tx binary search as the default offset-validation
    path. Typical cost drops from ~20–30 node lookups per offset to one
    HEAD plus two O(1) lookups per unique tx, with a per-tx LRU cache
    for repeated offsets. Any header/chain mismatch or missing header
    falls back to the legacy chain search, so older gateways keep
    working. New metric `observer_chunk_metadata_anchor_total{result}`
    (hit / cache_hit / metadata_missing / mismatch / error / fallback)
    tracks the rollout. Gateways that return an HTTP error on the new
    probe are no longer blacklisted from the shared pool — only
    transport failures do.
  - **Continuous observer reliability hardening**: The per-gateway
    schedule map is replaced with a flat list of `ScheduledObservation`
    events so duplicates, restart catch-up, and overdue retries are
    deterministic (legacy state auto-migrates on load). An explicit
    submission deadline (`windowEnd + submissionBufferMs`) now bounds
    the epoch — once exceeded, the scheduler clears pending work,
    marks the epoch `expired`, and stops issuing observations instead
    of spinning on stale state. Finalization is gated on both the
    window being complete and the pending queue being empty, and only
    flips `reportSubmitted` on a successful submit so transient
    submit failures retry. Unsubmitted prior epochs are discarded on
    epoch transition rather than force-finalized into the wrong epoch.
  - **Report telemetry**: Reports now record each gateway's `release`
    field from `/ar-io/info`, a `yarn summarize` script prints
    pass/fail counts grouped by release, and offset rendering now
    shows `<failures>/<observed> (<pct>)` so the denominator reflects
    the sampled subset.

- **ClickHouse GraphQL query no longer uses `FINAL`**: The composite
  ClickHouse backend previously issued `FROM transactions AS t FINAL` to
  deduplicate unmerged `ReplacingMergeTree` versions at read time. `FINAL`
  prevented `owner_projection` from being selected and forced a
  `PrimaryKeyExpand` that widened the skip-index-pruned granule set by
  ~4×. It is replaced with a `LIMIT 1 BY height, block_transaction_index,
  is_data_item, id` clause that dedupes in-engine as a post-sort filter
  without disabling projection planning or PREWHERE push-down. Safe
  because Arweave transaction data is immutable: all versions of a given
  primary key are byte-identical by construction.

- **Composite ClickHouse GraphQL Parallelized With SQLite Circuit
  Breaker**: The `CompositeClickHouseDatabase` now runs its ClickHouse
  and SQLite legs concurrently instead of serially, and wraps the
  SQLite leg in an opossum circuit breaker. ClickHouse errors
  (timeout, `max_rows_to_read`) still propagate to the caller, while
  SQLite failures degrade the response to ClickHouse-only results with
  a `PARTIAL_RESULT` warning attached via GraphQL `extensions.warnings`
  — ending silent partials for tip-of-chain rows and for the
  single-record `transaction(id)` lookup, which previously returned a
  bare `null` when SQLite was unavailable. The ClickHouse max-height
  boundary-optimization cache is now read non-blocking from the request
  path, with a background refresh keeping it warm. Fan-out preserves
  warnings end-to-end: `RemoteGqlQueryable` pulls upstream
  `extensions.warnings` off each response, `GatewaysGqlQueryable`
  merges them across sources, and synthesizes `UPSTREAM_UNAVAILABLE` /
  `UPSTREAM_CIRCUIT_OPEN` warnings for partially-failed aggregates that
  were previously logged-and-dropped. New env vars under
  `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_*` (defaults: timeout 5000ms,
  error threshold 80%, reset timeout 60000ms, rolling window 30000ms).

- **ClickHouse `owner_address` Bloom + Skip Projection on Tag Filters**:
  ClickHouse projections cannot carry inline skip indexes, so
  owner+tag GraphQL queries that routed through `owner_projection`
  scanned every granule within the owner range. An `owner_address`
  bloom filter is now defined on the main `transactions` table, and
  the per-query `optimize_use_projections = 0` guard is extended to
  tag filters. Owner-only queries still benefit from
  `owner_projection`'s sort order; owner+tag queries now fall back to
  the main table where `id_bloom` / `tag_names_bloom` /
  `tag_values_bloom` / `owner_address_bloom` can prune granules across
  all three dimensions. Existing deployments get the index registered
  via an idempotent `ALTER TABLE ... ADD INDEX IF NOT EXISTS` on the
  next `clickhouse-import` cycle; a manual
  `MATERIALIZE INDEX owner_address_bloom` is required to populate the
  index on existing parts.

- **Parquet Export Defaults to Include L1 Transactions and Tags**:
  `ParquetExporter.export()` defaults now align with the
  `scripts/parquet-export` CLI wrapper and the auto-verify harness,
  both of which already included L1 by default. Callers that want
  L2-only output must now pass `skipL1Transactions` /
  `skipL1Tags` explicitly.

### Fixed

- **ClickHouse `owner_projection` now usable for tag-filtered owner queries**:
  The projection was previously defined with `SELECT *`, which in ClickHouse
  excludes `MATERIALIZED` columns — so `tag_names` and `tag_values` were
  absent from the projection and the optimizer rejected it for any query
  with predicates on those columns (which includes all tag-filtered GraphQL
  queries). The projection body is now `SELECT *, tag_names, tag_values`, so
  the optimizer picks `owner_projection` for owner-scoped queries and reads
  orders of magnitude fewer granules. Existing deployments need a one-time
  manual migration (`DROP PROJECTION` / `ADD PROJECTION` / `MATERIALIZE
  PROJECTION`) — see the inline comment in
  `src/database/clickhouse/schema.sql`. Fresh deployments get the corrected
  projection from the `CREATE TABLE` body with no operator action required.

- **GraphQL `Block.timestamp` Non-Nullable Field Error**: Addresses a
  "Cannot return null for non-nullable field Block.timestamp" error
  that could surface when resolving blocks with incomplete data.

- **GraphQL Data Item Signature Fetch Falls Back to `NOT_FOUND`**: The
  data-item path in `resolveTxSignature` returned the fetcher result
  directly, so an `undefined` from
  `SignatureFetcher.getDataItemSignature` (e.g., missing attributes or
  a stream failure reading from the parent bundle) would trigger a
  "Cannot return null for non-nullable field" error on the `String!`
  signature field. The data-item path now mirrors the transaction
  path and falls back to `NOT_FOUND`.

- **`clickhouse-auto-import` Honors `SQLITE_DATA_PATH`**: The
  `clickhouse-auto-import` container had its SQLite bind mount
  hardcoded to `./data/sqlite`, while `core` used
  `${SQLITE_DATA_PATH:-./data/sqlite}`. When `SQLITE_DATA_PATH` was
  set, the two containers diverged: the daemon's `batch_has_data`
  pre-check resolved to a missing path and silently failed open, so
  empty height ranges were still sent through the full export/import
  pipeline. The mount is now consistent with core.

- **Fail-Fast on ClickHouse GraphQL Rejection**: Awaiting
  `Promise.allSettled` gated ClickHouse errors on the SQLite leg's
  breaker timeout. The composite flow now awaits ClickHouse first and
  rethrows immediately, absorbing SQLite rejections eagerly so bailing
  out early does not emit an unhandled rejection.

- **Reject Concurrent Parquet Exports + Skip Empty ETL Ranges**: The
  auto-import loop previously wasted cycles (and logged spurious
  "Input directory does not exist" / "Parquet file too short" errors)
  on batches that either collided with a still-running export or
  spanned empty height ranges. The admin endpoint now returns `409`
  instead of swallowing the rejection, the exporter script surfaces a
  clear error when the singleton status is stale, and batches with no
  source rows short-circuit via a `sqlite3` pre-check.

- **Hive-Layout ClickHouse Importer Requires `blocks` and
  `transactions` Files**: The Hive-layout importer iterated a
  per-table glob; when no files matched, bash left the literal pattern
  string in the loop variable and the `-f` check silently
  short-circuited, so the partition reported success even though zero
  required files were imported — combined with export races that
  produced empty staging dirs, this was silently dropping data. The
  `matched_count` validation from the flat-dir path is now ported so
  `blocks` and `transactions` each must contribute at least one file;
  `tags` may still be empty.

- **GraphQL Boundary Skips `minHeight` on SQLite "New" Tables**: The
  ClickHouse/SQLite GraphQL boundary raises `minHeight` to route
  historical queries away from SQLite. Applied to `new_transactions` /
  `new_data_items`, the resulting `height >= :minHeight` silently
  dropped pending rows whose height is `NULL`. Because the "new"
  tables only hold unstable/recent data that ClickHouse never covers,
  the predicate is now skipped entirely for those sources.

## [Release 76] - 2026-04-17

This is a **recommended release** focused on **response signing**,
**ClickHouse data lifecycle management**, and **query-path efficiency**. Key
highlights include **RFC 9421 HTTP Message Signatures** for cryptographically
verifiable gateway responses, **tag-based TTL rules for ClickHouse-exported
data** so operators can expire indexed rows by tag or uploader, and a major
**ClickHouse schema consolidation** into a single partitioned `transactions`
table with bloom filter skip indexes and native projections. It also adds
**per-host `APEX_ARNS_NAME` mapping**, **Parquet export partition progress**
in the status API, and a **`clickhouse-import --flat-dir` mode**. GraphQL
gets two performance improvements — **skipping SQLite for heights already
covered by ClickHouse** and **skipping the `owner.key` fetch when only
`owner.address` is selected** — plus a fix for duplicate transaction results
from ClickHouse and correctly-populated `indexedAt` / `blockPreviousBlock`
fields.

### Added

- **RFC 9421 HTTP Message Signatures for Gateway Responses**: The gateway
  can now sign responses using RFC 9421 HTTP Message Signatures, allowing
  clients to cryptographically verify response integrity and origin.
  Controlled by `HTTPSIG_ENABLED` (default: false) with an Ed25519 key
  auto-generated at `HTTPSIG_KEY_FILE` (default:
  `data/keys/httpsig.pem`). `HTTPSIG_BIND_REQUEST` (default: true) binds
  each response to the triggering request via `@method;req` and
  `@path;req`. An attestation linking the key to the operator wallet is
  uploaded to Arweave on startup when `HTTPSIG_UPLOAD_ATTESTATION=true`
  and `OBSERVER_WALLET` is set. HTTPSIG response metadata is documented
  in OpenAPI for `/ar-io/info` and data endpoints.

- **Tag-Based TTL Rules for ClickHouse-Exported Data**: Operators can now
  expire rows in the ClickHouse `transactions` table by tag content or
  uploader owner address. Rules are declared in
  `config/clickhouse-ttl-rules.yaml` (copy from the committed
  `.example.yaml` template) and loaded at the top of every
  `clickhouse-auto-import` cycle into four source tables, with exact-match
  lookups going through refreshing `COMPLEX_KEY_HASHED` dictionaries and
  prefix matches falling back to scanned tables. Native TTL enforcement
  deletes rows when `expires_at` elapses. Supports a top-level
  `default_ttl_seconds` fallback and per-rule `never_expire: true`
  exemptions (precedence: exempt > shortest TTL match > default > NULL).
  v1 applies only to rows imported after rules are loaded; no backfill.
  The loader fails open on missing/malformed rules to avoid blocking
  imports.

- **Per-Host `APEX_ARNS_NAME` Mapping**: `APEX_ARNS_NAME` now accepts a
  comma-separated list of values positionally mapped to `ARNS_ROOT_HOST`
  entries (e.g., `APEX_ARNS_NAME=turbo,ar-io` with
  `ARNS_ROOT_HOST=arweave.dev,g8way.io`). A single value still applies to
  all hosts.

- **Parquet Export Partition Progress in Status API**: The admin status
  endpoint and the `parquet-export` CLI poll loop now surface the
  current partition range and completed/total partition counts while a
  Parquet export is in progress.

- **`clickhouse-import --flat-dir` mode**: `scripts/clickhouse-import`
  accepts a flat directory of Parquet files named
  `<table>-minHeight:<min>-maxHeight:<max>-rowCount:<n>.parquet`
  (blocks / transactions / tags all in the same directory), as an
  alternative to the default `<table>/data/height=<min>-<max>/*.parquet`
  Hive layout.

### Changed

- **ClickHouse schema consolidation**: The ClickHouse GQL backend now uses
  a single `transactions` table with partitioning by height, bloom filter
  skip indexes on `id` and `tags`, and native projections for owner and
  recipient queries, replacing the previous four-table design
  (`transactions`, `id_transactions`, `owner_transactions`,
  `target_transactions`). Column codecs (Delta + ZSTD) and
  `LowCardinality` on `content_type` / `signature_type` reduce storage.
  The GQL query layer uses `hasAny` for multi-value tag filters and
  tuple-comparison cursor pagination. Requires ClickHouse 24.8 or later
  and a one-time full re-import from Parquet — see
  `docs/parquet-and-clickhouse-usage.md`.

- **Skip SQLite for Heights Covered by ClickHouse in GraphQL**: Opt-in
  optimization in `CompositeClickHouseDatabase` raises the SQLite
  fallback's `minHeight` to `(clickhouseMax - buffer + 1)` and skips the
  SQLite call entirely when the adjusted range is empty. Controlled by
  `CLICKHOUSE_SQLITE_MIN_HEIGHT_ENABLED` (default: false), with a
  configurable safety buffer (default: 10 heights) and a cached
  ClickHouse max-height lookup (default TTL: 60s). Degrades to prior
  behavior on lookup failure.

- **GraphQL `owner.key` Fetch Skipped When Only `owner.address` Requested**:
  Splitting the `Transaction.owner` resolver into field-level
  `Owner.address` and `Owner.key` resolvers lets GraphQL skip the per-row
  owner key fetch unless `key` is explicitly selected. Memoization on the
  `Owner` parent still fetches the key only once when multiple aliased
  selections or overlapping fragments request it in the same query.

- **Default ClickHouse Image Bumped to 26.3**: The default ClickHouse
  container image used by `clickhouse-auto-import` is now 26.3.

- **Observer Image Updated**: `OBSERVER_IMAGE_TAG` bumped to include
  epoch source fixes.

### Fixed

- **ClickHouse GQL `indexedAt` and `blockPreviousBlock` fields**: These
  fields were previously always returned as `undefined` because the base
  SELECT omitted the corresponding columns. They are now populated.

- **Duplicate GraphQL Transaction Results from ClickHouse**: The
  `transactions` table uses `ReplacingMergeTree(inserted_at)`, which only
  deduplicates during background merges. Queries now use `FINAL` so
  GraphQL returns a single edge per id instead of every un-merged
  version.

- **Tag Headers on Manifest-Resolved Responses**: When a manifest path
  resolves to an inner data item, `X-Arweave-Tag-*` headers are now
  populated from the resolved inner item rather than the manifest
  transaction.

- **Turbo Fallback Narrowed to Module-Not-Found**: The optional Turbo
  upload path now falls back only on module-not-found (instead of
  swallowing unrelated errors) and requires an explicit trigger header
  before signing.

## [Release 75] - 2026-04-08

This is a **recommended release** focused on **on-demand data item resolution**
and **response header enrichment**. Key highlights include **tag and verification
response headers** that expose transaction tags and cryptographic metadata
directly in HTTP responses, **on-demand data item metadata resolution** that
resolves unindexed data items by parsing ANS-104 bundle binaries on the fly,
**HyperBEAM as a root TX offset source** for efficient bundle navigation without
full downloads, and **GraphQL on-demand transaction resolution** for querying
unindexed data items. It also adds **configurable chunk GET retry behavior** to
reduce worst-case retrieval times and **Prometheus metrics for root TX semaphore
observability**.

### Added

- **Tag and Verification Response Headers**: Data responses on `/raw/:id` and
  `/:id` now include `X-Arweave-Tag-*` headers with transaction/data item tags,
  plus verification headers (`X-Arweave-Signature`, `X-Arweave-Owner`,
  `X-Arweave-Owner-Address`, `X-Arweave-Target`, `X-Arweave-Anchor`,
  `X-Arweave-Signature-Type`). Enabled by default
  (`ARWEAVE_TAG_RESPONSE_HEADERS_ENABLED=true`). Uses a fast local-only
  resolution path (LMDB txStore -> LRU cache -> GQL DB) with background indexing
  for uncached items. Includes a configurable byte budget
  (`ARWEAVE_TAG_RESPONSE_HEADERS_MAX_BYTES`, default 8KB) and tag count cap
  (`ARWEAVE_TAG_RESPONSE_HEADERS_MAX`, default 100). For L2 data item signatures
  and owner keys, `WRITE_ANS104_DATA_ITEM_DB_SIGNATURES=true` is also required.

- **On-Demand Data Item Metadata Resolution**: Data items not yet indexed locally
  are resolved on-demand by discovering the root bundle, parsing the binary
  header, and extracting signature/owner/tags. Results are cached in an LRU cache
  and persisted to the database for future requests. Background resolution is
  capped at 1 concurrent operation (configurable via
  `TX_METADATA_RESOLVE_CONCURRENCY`) with fail-fast semantics.

- **HyperBEAM Root TX Offset Source**: HyperBEAM can now be used as a
  root transaction offset source for on-demand data item resolution. Uses
  offset-guided recursive bundle index navigation to extract complete data item
  metadata without downloading full bundles. Controlled by
  `HYPERBEAM_ROOT_TX_ENABLED` and `HYPERBEAM_ENDPOINT` (default:
  `arweave.net`).

- **Configurable Chunk GET Retry Behavior**: Arweave chunk retrieval
  retry count and geometry timeout are now configurable, reducing worst-case
  chunk retrieval time from ~115s to ~15s. New env vars:
  `ARWEAVE_CHUNK_RETRY_COUNT` (default: 5), `ARWEAVE_TX_GEOMETRY_TIMEOUT_MS`
  (default: 5000), `ARWEAVE_TX_GEOMETRY_TIMEOUT_RETRIES` (default: 2).

- **GraphQL On-Demand Transaction Resolution**: The `transaction(id)` GraphQL
  query can now resolve unindexed data items on-demand by extracting metadata
  from ANS-104 bundle binaries. Enabled by default
  (`GRAPHQL_ON_DEMAND_RESOLUTION_ENABLED=true`). Includes a
  configurable timeout (`GRAPHQL_ON_DEMAND_RESOLUTION_TIMEOUT_MS`, default 5s)
  and concurrency limit (`GRAPHQL_ON_DEMAND_RESOLUTION_MAX_CONCURRENT`, default
  1). Only applies to single-ID lookups; the plural `transactions(...)` query is
  unaffected.

- **SignatureType in GraphQL**: The `signatureType` field is now surfaced in
  GraphQL transaction responses for data items.

- **Root TX Semaphore Prometheus Metrics**: New Prometheus metrics for root TX
  resolution semaphore observability, including acquire/release/timeout counters
  and queue depth gauge.

### Changed

- **Root TX Lookup Order**: `ROOT_TX_LOOKUP_ORDER` reordered to prefer GraphQL
  over HyperBEAM and CDB for faster local resolution.

- **HyperBEAM Request Timeout**: Default HyperBEAM request timeout lowered to
  500ms.

## [Release 74] - 2026-04-01

This is a **recommended release** focused on **cache performance**,
**multi-domain ArNS support**, and **content moderation correctness**. Key
highlights include **background caching for range request cache misses** to
improve video/media streaming performance, **multiple ArNS root hosts** for
serving ArNS names across multiple domains from a single gateway, **contiguous
data cache hit/miss Prometheus metrics** for improved observability, and
**configurable cache control for blocked responses**. It also corrects HTTP 451
handling for blocked content, simplifies the parquet export pipeline, and adds
ClickHouse and block verification to auto-verify.

### Added

- **Background Caching for Range Request Cache Misses**: When a range request
  (e.g., byte-range for video seeking) misses the local cache, the gateway now
  optionally fetches and caches the full item in the background so subsequent
  requests (range or full) are served locally. Controlled by
  `BACKGROUND_CACHE_RANGE_MAX_SIZE` (default: 0 / disabled) and
  `BACKGROUND_CACHE_RANGE_CONCURRENCY`. Includes deduplication, capacity-based
  drop semantics, and Prometheus metrics for monitoring cache activity

- **Multiple ArNS Root Hosts**: Operators can now serve ArNS names across
  multiple domains from a single gateway instance by providing a comma-separated
  list in `ARNS_ROOT_HOST` (e.g., `ARNS_ROOT_HOST=arweave.dev,g8way.io`). The
  first host is used as the "primary" for gateway identity headers. ArNS
  resolution, apex content, and sandbox redirects work per-matched host with
  longest-suffix matching (#621)

- **ClickHouse Verification in Auto-Verify**: Auto-verify is a data validation
  tool that checks consistency of indexed blockchain data (blocks, transactions,
  data items) across multiple backends (SQLite, Parquet, ClickHouse). This adds
  an optional ClickHouse source for verification in addition to the existing
  SQLite and Parquet sources.

- **Block Verification in Auto-Verify**: Verify block data alongside
  transactions and data items

- **Bundle Data Prefetch in Auto-Verify**: During auto-verify runs, raw bundle
  bytes are now fetched from the local gateway while it is still running, then
  parsed after shutdown. Previously the bundle-parser source had to fetch from
  arweave.net after the gateway was stopped, which was significantly slower.

- **Contiguous Data Cache Hit/Miss Metrics**: New
  `contiguous_data_cache_hits_total` and `contiguous_data_cache_misses_total`
  Prometheus counters in `ReadThroughDataCache`, labeled by `request_type`
  (`range` vs `full`). Enables operators to monitor cache performance per
  request type.

- **Accurate Cache Miss vs Not-Found Metrics**: Cache miss counter now fires
  only after a successful upstream fetch (data exists but wasn't cached locally).
  A new `contiguous_data_not_found_total` counter tracks requests where data is
  unavailable in any source, preventing not-found requests from inflating the
  miss count and skewing the cache hit rate.

- **Configurable Cache Control for Blocked (451) Responses**: New
  `CACHE_BLOCKED_MAX_AGE` env var (default: 30 days, matching stable data TTL)
  controls the `Cache-Control` max-age sent with 451 responses. Previously,
  blocked responses used the short not-found TTL, causing CDNs and proxies to
  re-request blocked content too frequently.

- **Parent Bundle ID in Missing Data Item Errors**: Auto-verify `compareItems`
  now includes the parent bundle ID in `missing_in_source` discrepancy messages
  for data items, making it easier to identify which bundle a missing item
  belongs to when debugging bundle-parser failures.

### Changed

- **Parquet Export Pipeline Simplification**: Eliminated DuckDB intermediate
  tables from the export pipeline. All core export logic moved from the bash
  script into `src/workers/parquet-exporter.ts`, with the CLI script becoming a
  thin wrapper around the admin API. The CLI now uses `--api-host`/`--api-port`
  instead of `--core-db`/`--bundles-db`.

- **Removed Legacy Auto-Verify CLI Options**: Cleaned up deprecated verification
  flags

### Fixed

- **ClickHouse ETL Height Range**: Fixed off-by-one errors in height range
  calculations in `clickhouse-auto-import`

- **ClickHouse ETL Exit Code Capture**: Fixed `$?` capturing the exit code of a
  variable assignment instead of the `curl` command

- **HTTP 451 for Blocked Content**: Corrects r73's blocked-content status code
  from 452 (non-standard) to 451 ("Unavailable For Legal Reasons"), the
  IANA-registered standard for content blocked due to legal or policy reasons.

- **Trusted Gateway ArNS 451 Handling**: The `TrustedGatewayArNSResolver` now
  accepts HTTP 451 responses from trusted gateways instead of treating them as
  errors. When a trusted gateway indicates a name is blocked, the local gateway
  respects that moderation signal and returns 451 to the client rather than
  falling through to the on-demand resolver.

- **Serving Cached Data with Undefined or Zero Content-Length**: The read path
  of `ReadThroughDataCache` now skips cache entries with a missing or zero
  `dataSize`, preventing responses without a `Content-Length` header. The write
  path already rejected zero-size entries; this closes the corresponding
  read-side gap.

- **Parquet Export and ClickHouse Import Robustness**: Parquet-export script now
  uses `curl -o` with temp files instead of `head`/`tail` parsing to handle
  multiline JSON API responses correctly. Auto-verify's `importToClickHouse`
  switches to `execFileSync` with an args array, preventing shell injection in
  ClickHouse import invocations.

- **Parquet Export Verify-Count Non-Zero Exit**: `parquet-export --verify-count`
  now exits non-zero when row counts don't match, making it useful in CI and
  automation pipelines. Also validates `curl` availability at startup alongside
  `python3`.

- **High-Severity Dependency Vulnerabilities**: Resolved known vulnerabilities in
  transitive production dependencies via yarn resolutions: `path-to-regexp`
  (ReDoS, via express/express-openapi-validator), `h3` (request smuggling),
  `picomatch` (glob injection), `preact` (VNode injection), `socket.io-parser`
  (unbounded binary attachments), `undici` (multiple HTTP smuggling/memory
  issues), and bumped `fast-xml-parser` from 5.3.6 to 5.5.9.

- **JSON Data Files Missing from Production Build**: `offset-block-mapping.json`
  was excluded from `dist/` because the build copy step only matched `.graphql`,
  `.sql`, and `.lua` files. This caused a startup warning and fallback to slower
  full-range block searches in containers. `.json` files are now included in the
  copy step.

- **Auto-Verify Prefetch Timing and Empty ClickHouse URL**: Removed the
  `last_fully_indexed_at` filter from the bundle prefetch query — this flag is
  set asynchronously by `BundleRepairWorker`, causing prefetch to find 0 bundles
  even when indexing was complete. Also handles `AUTO_VERIFY_CLICKHOUSE_URL=""`
  (empty string) explicitly to prevent a crash when the variable is set but
  empty in `.env`.

## [Release 73] - 2026-03-18

This is a **recommended release** focused on **operator configurability** and **P2P retrieval performance**. Key highlights include **unified Cache-Control headers** with operator-configurable durations, **hedged peer requests with consistent hash routing** to reduce tail latency and improve cache efficiency, **request trace IDs** for end-to-end correlation, and a **cache-only mode** for protecting upstream bandwidth from high-volume clients. It also fixes HTTP 452 responses for blocked content, corrects client disconnect detection, and addresses security audit vulnerabilities.

### Added

- **Unified Cache-Control Headers**: Move default Cache-Control from Envoy's
  catch-all route into Express middleware, eliminating duplicate headers and
  making data handler cache durations operator-configurable via
  `DEFAULT_CACHE_CONTROL_MAX_AGE_SECONDS`, `STABLE_CACHE_CONTROL_MAX_AGE_DAYS`,
  and related env vars

- **Cache-Only Client IPs/CIDRs**: New `CACHE_ONLY_CLIENT_IPS_AND_CIDRS` env
  var to short-circuit data retrieval requests with a 404 if the data is not
  already cached locally, useful for protecting upstream bandwidth from specific
  high-volume clients

- **Client Disconnect Prometheus Metric**: New `client_disconnect_total` counter
  metric tracks when clients abort requests before the response completes

- **P2P Contiguous Data Retrieval Improvements**: Major overhaul of peer data
  retrieval to reduce tail latency and improve cache efficiency
  - **Hedged requests**: Fires a second request to the next candidate peer after
    a configurable delay (`PEER_HEDGE_DELAY_MS`) if no response yet; first
    success cancels all others, capped at `PEER_MAX_HEDGED_REQUESTS`
  - **Per-peer concurrency limiter**: Fail-fast counter that skips saturated
    peers instead of queuing, configurable via `PEER_MAX_CONCURRENT_OUTBOUND`
  - **Consistent hash ring**: Routes each data ID to the same small set of
    "home" peers for cache locality, with weighted fallback for remaining slots;
    configured via `PEER_HASH_RING_VIRTUAL_NODES` and
    `PEER_HASH_RING_HOME_SET_SIZE`
  - **Decoupled candidate pool**: `PEER_CANDIDATE_COUNT` replaces the old
    `min(peerCount, 3)` logic, giving hedging a deeper bench to draw from
  - **Chunk peer selection via hash ring**: Chunk requests are also routed
    through the hash ring by absolute offset for improved peer cache utilization

- **Request Trace IDs**: Every HTTP request gets a unique `requestId` in Winston
  log entries via AsyncLocalStorage, independent of OTEL. Reads or generates
  `X-Request-Id` headers and echoes them in responses for end-to-end request
  correlation

### Changed

- **HTTP 452 for Blocked Content**: Blocked content now returns HTTP 452 with a
  descriptive message identifying the blocked ID and the node's content policy,
  instead of a generic 404 Not Found

### Fixed

- **HTTP 499 Only for Actual Client Disconnects**: Internal data retrieval
  timeouts (e.g., upstream gateway timeouts) were being misidentified as client
  disconnects. Now checks `req.signal.aborted` to confirm the client actually
  disconnected before returning 499

- **`/tx_anchor` Route Shadowing**: Move `/tx_anchor` route before `/tx` in
  Envoy config to prevent prefix-match shadowing that caused `/tx_anchor`
  requests to be handled by the `/tx` route

- **Security Audit Vulnerabilities**: Bump `simple-git` to fix critical RCE via
  `blockUnsafeOperationsPlugin` bypass; add `multer` resolution to fix high
  severity DoS vulnerabilities in `express-openapi-validator`

## [Release 72] - 2026-03-11

This is a **recommended release** focused on **data retrieval reliability** and **caching intelligence**. Key highlights include a **negative data cache** that reduces upstream load for consistently missing data, **direct byte offset hints** to help gateways locate data when internal lookup mechanisms fall short, **untrusted data caching with stochastic re-verification**, and significant **stream reliability improvements** that eliminate false timeouts on large transfers. It also adds **gateway loop prevention** via per-gateway via-chain detection.

### Added

- **Negative Data Cache**: Two-phase cache that tracks data IDs consistently
  missing across configurable thresholds and short-circuits future requests with
  404 responses, reducing upstream load during outages and for permanently
  unavailable data
  - Includes exponential backoff with fast re-promotion, health gating to
    prevent false positives during upstream outages, and TTL-based miss tracker
    eviction
  - Controlled via `NEGATIVE_CACHE_ENABLED` (default: true),
    `NEGATIVE_CACHE_MAX_SIZE`, `NEGATIVE_CACHE_TTL_MS`,
    `NEGATIVE_CACHE_MISS_THRESHOLD_MS`, and
    `NEGATIVE_CACHE_MISS_COUNT_THRESHOLD`

- **Direct Byte Offset Hints for Data Item Retrieval**: Clients can supply
  `X-AR-IO-Root-Transaction-Id`, `X-AR-IO-Root-Path`,
  `X-AR-IO-Root-Data-Offset`, and `X-AR-IO-Root-Data-Size` headers to bypass
  server-side bundle lookups and resolve data items via direct byte offsets
  - Includes `fetch-with-hint` CLI tool for resolving hints via GraphQL

- **DATA_CACHED Webhook Event**: Emits a webhook when data is cached for the
  first time, enabling external content moderation sidecars (e.g., phishing
  scanners)
  - Opt-in via `WEBHOOK_EMIT_DATA_CACHED_EVENTS=true` (default: false)

- **Untrusted Data Caching with Stochastic Re-verification**: Caches all
  upstream data optimistically instead of only when a hash exists locally, with
  configurable background re-verification rates to ensure integrity
  - Controlled via `UNTRUSTED_CACHE_RETRY_RATE` (default: 0.1) and
    `TRUSTED_CACHE_RETRY_RATE` (default: 0.0)
  - Evicts data on hash mismatch to maintain integrity

- **12-Hour Cache-Control Tier**: New middle tier for data that is unstable but
  from a trusted source (e.g., trusted bundlers), providing a three-tier system:
  stable (30d, immutable) > unstable trusted (12h) > unstable (2h)

- **Chunk Broadcast Improvements**: All 5 tip nodes (tip-1 through tip-5) are
  now included in default preferred chunk POST nodes, with shuffled ordering and
  a minimum success requirement
  - Controlled via `CHUNK_POST_MIN_PREFERRED_SUCCESS_COUNT` (default: 2)

- **OTEL Resource Attributes Passthrough**: Operators can set custom
  OpenTelemetry resource attributes via the standard
  `OTEL_RESOURCE_ATTRIBUTES` environment variable, with env var values
  overriding auto-detected attributes

- **Gateway Loop Prevention**: Per-gateway via-chain detection skips individual
  gateways already visited in the request path, with hop count validation
  against `MAX_DATA_HOPS` (3) as defense-in-depth. Client IP, forwarded IPs,
  and via header are now included as OTEL span attributes for observability.

### Changed

- Default `CDB64_REMOTE_RETRIEVAL_ORDER` changed to `'chunks'` only, removing
  gateways from the default order since range requests aren't effectively cached
  on gateways

### Fixed

- **Stream Reliability Improvements**: Replaced wall-clock stream timeouts with
  backpressure-aware stall-based timeouts (30s no-data threshold), preventing
  false kills and truncated responses on large or slow transfers
  - Extracted `pipeStreamToResponse` helper for consistent stream pipe and error
    handling across routes

- Fixed Axios `CanceledError` not being normalized to `AbortError`, causing
  incorrect upstream disconnection handling

- Fixed streams not being destroyed on unexpected HTTP status codes from peers,
  preventing socket leaks

- Added 206 Partial Content acceptance for ranged peer requests

- Fixed upstream stream not being destroyed on premature client disconnect

- Fixed `detectLoopInViaChain` to lowercase via entries for proper
  case-insensitive matching

## [Release 71] - 2026-02-26

This is a **recommended release** that adds **per-gateway trust configuration**
for `TRUSTED_GATEWAYS_URLS`, enabling operators to mark individual gateways as
trusted or untrusted for finer-grained data verification control. It also
includes **peer URL tracking in chunk broadcast responses** for improved
debuggability, and fixes for **upstream gateway content-length validation** to
prevent serving bogus responses from gateways that return 200 instead of 404.

### Added

- **Per-Gateway Trust Flag for `TRUSTED_GATEWAYS_URLS`**: Extended the
  `TRUSTED_GATEWAYS_URLS` configuration format to support per-gateway trust
  levels
  - Untrusted gateways only cache data when the hash matches a known value,
    providing defense-in-depth against serving incorrect data
  - Default configuration now uses `turbo-gateway.com` (trusted) with
    `arweave.net` as an untrusted fallback

- **Peer URL in Chunk Broadcast Responses**: Chunk broadcast responses now
  include the peer URL for better debuggability when troubleshooting chunk
  propagation issues

### Changed

- Default `TRUSTED_GATEWAYS_URLS` now uses `turbo-gateway.com` as the primary
  trusted gateway with `arweave.net` as an untrusted fallback

### Fixed

- **Upstream Gateway Content-Length Validation**: Added validation of
  content-length in `GatewaysDataSource` to reject responses with missing or
  zero content-length, preventing upstream gateways from serving bogus HTML
  landing pages when they return 200 instead of 404.

## [Release 70] - 2026-02-24

This is a **recommended release** that introduces **CDB64 download tooling** for
fetching remote partitioned indexes with resume support, **chunk request
concurrency limiting** and first-data timeouts for improved data retrieval
reliability, and **expanded CDB64 root TX index coverage** with new AO and
without-content-type index sources. It also includes critical fixes for event
listener leaks, stream data loss, and request cancellation under high
concurrency, along with **defense-in-depth loop protection** enhancements and a
new **auto-verify indexing tool** for cross-source bundle data validation.

### Added

- **CDB64 Download Tool**: New CLI tool (`tools/download-cdb64`) for fetching
  remote partitioned CDB64 indexes with production-grade reliability
  - Downloads partition files from manifest sources (HTTP URLs, Arweave TX IDs,
    byte-range specifications, local files)
  - HTTP Range request resume for interrupted downloads — partial `.tmp` files
    are preserved and downloads resume from where they left off
  - Per-partition retry support with configurable retry count (`--retries/-r`,
    default: 5)
  - Concurrent downloads with configurable parallelism (`--concurrency/-c`,
    default: 3)
  - SHA-256 verification of downloaded partitions against manifest checksums
  - Generates updated manifest with local file locations on completion

- **Streaming Partitioned CDB64 Writer**: New low-memory CDB64 generation mode
  for large indexes
  - Reduces peak memory from O(total_records) to O(largest_partition) via
    two-phase scatter/build approach
  - Phase 1 writes records to per-partition temp files; Phase 2 builds CDB files
    sequentially
  - New `--low-memory` flag added to all CDB64 CLI tools (requires
    `--partitioned`)
  - Progress callbacks show partition-level build status

- **AO and Without-Content-Type CDB64 Index Sources**: Expanded default CDB64
  root TX index coverage
  - New AO data items index (~1.6B records) covering AO-tagged data items up to
    block height 1,820,000
  - New without-content-type index (~1.2B records) covering data items lacking a
    Content-Type tag up to block height 1,820,000
  - Default `CDB64_ROOT_TX_INDEX_SOURCES` now includes all three indexes

- **Chunk Request Concurrency Limiting and First-Data Timeout**: New controls for
  chunk-based data retrieval under load
  - `CHUNK_REQUEST_CONCURRENCY` (default: 50): Limits concurrent chunk fetch
    requests to prevent overwhelming backends
  - `CHUNK_FIRST_DATA_TIMEOUT_MS` (default: 10000): Timeout for receiving the
    first chunk of data; if exceeded, request falls through to alternative data
    sources. Set to 0 to disable.

- **Root Bundle Gateway Fallback Path**: Added `/<id>` fallback path for root
  bundle gateway requests when `/raw/<id>` fails
  - Enables bundle retrieval from HyperBEAM endpoints that may not support the
    `/raw/<id>` path
  - Separate `rootBundleGatewaysDataSource` instance configured with fallback
    enabled

- **ArNS Resolver Host Override**: New `TRUSTED_ARNS_RESOLVER_HOST_HEADER`
  environment variable decouples connection target from Host header in ArNS
  resolver, with `__NAME__` placeholder substitution for dynamic values

- **Defense-in-Depth Loop Protection Improvements**: Additional
  safeguards against request forwarding loops between gateways
  - Hop count validation added to `GatewaysDataSource` (MAX_DATA_HOPS = 3)
  - Origin/IP blocking applied to peer forwarding path via
    `FilteredContiguousDataSource`
  - Startup warning when `TRUSTED_GATEWAYS_URLS` contains this gateway's own
    `ARNS_ROOT_HOST`, indicating a self-forwarding loop

- **Auto-Verify Indexing Tool**: New tool (`tools/auto-verify`) for cross-source
  bundle data comparison and indexing consistency validation
  - Indexes configurable block ranges, then compares data items across SQLite,
    Parquet, GraphQL, and independently-parsed raw bundles
  - Verifies field consistency: offset, size, ownerOffset, ownerSize,
    signatureOffset, signatureSize, rootParentOffset
  - Bundle indexing timeout (5 minutes) for graceful handling of slow indexing
  - Cache preserved by default for faster iteration across runs
  - Detailed discrepancy output shows data item ID, field name, and per-source
    values

- **Git Worktree Helper**: New development tool (`tools/wt`) for parallel
  development using git worktrees
  - Creates worktrees under `wt/<branch>` with symlinked `.env` and
    `CLAUDE.local.md`
  - Commands: `add`, `rm`, `ls` with `--existing` flag for checking out existing
    branches
  - Automatically runs `yarn install` with a clean `data/` directory per
    worktree

### Changed

- Default ArNS gateway changed from `ar-io.net` to `turbo-gateway.com`
- Default Arweave gateway in test suites changed from `arweave.net` to
  `turbo-gateway.com`
- Renamed AO CDB directory from `cdb64-root-tx-index-ao` to
  `cdb64-root-tx-index-ao-to-height-1820000` for naming consistency
- Removed unused Cucumber dependency

### Fixed

- **Stream Data Loss Prevention**: Fixed range stream being put into flowing mode
  prematurely, causing data to be emitted and lost before the consumer could
  attach
- **HTTP Request Cancellation**: Threaded AbortController through chunk timeout
  to properly cancel in-flight HTTP requests instead of only rejecting the
  promise
- **Timer Leaks**: Fixed timeout timer leaks in chunk request implementation by
  hoisting timeout variable for proper cleanup in finally blocks
- **Event Listener Leaks**: Replaced `AbortSignal.any()` with `anySignal()` from
  the `any-signal` package and added proper `ClearableSignal.clear()` calls in
  finally blocks across composite ArNS resolver and chunk request paths to
  prevent listener accumulation under high concurrency
- **CDB64 Config Order**: Fixed CDB64 root TX index source search order to
  preserve configuration order instead of sorting alphabetically
- **Data Item Tag Handling**: Content-Type and Content-Encoding tag processing
  now uses first match instead of last match, aligning with legacy gateway
  behavior
- **Docker Build**: Multi-stage Dockerfile now copies `resources/` directory into
  runtime stage so the default CDB64 manifest is accessible inside containers
- **Cache Directory Cleanup**: `.gitkeep` files properly recreated after cleaning
  cache directories
- **CDB64 Test Reliability**: Conditional test skipping when native CDB64 module
  is unavailable; direct module probing prevents async rejection leaks in CI
- **Dependency Security**: Updated axios (DoS via `__proto__`), tar (symlink/
  overwrite CVEs), qs (arrayLimit bypass DoS), fast-xml-parser (RangeError DoS),
  and other transitive dependencies with known vulnerabilities

## [Release 69] - 2026-02-11

This is a **recommended release** that introduces **DNS-based multi-peer
discovery** via Envoy's Endpoint Discovery Service, enabling automatic Arweave
peer detection with health-checked routing and consensus-based failover. It also
adds **multi-layered HyperBEAM request loop prevention** using header, via-chain,
and User-Agent detection to block infinite forwarding loops between gateways.
Additionally, this release includes **comprehensive CDB64 documentation** covering
operator guides, format specifications, and tooling reference.

### Added

- **DNS-Based Multi-Peer Discovery with Envoy EDS**: Automatic Arweave peer
  discovery and health-checked routing via Envoy's Endpoint Discovery Service
  - Resolves DNS records (e.g., `peers.arweave.xyz`) to discover Arweave peers
    automatically
  - Health-checks peers and classifies them as "full" (complete blockchain data)
    or "partial" (incomplete) based on sync status
  - Routes requests to fully-synced peers first with automatic failover to
    partial peers
  - Consensus-based reference height calculation prevents routing to stale or
    outlier peers
  - New Prometheus metrics for peer discovery, classification, and health check
    monitoring
  - Configurable via `ARWEAVE_PEER_DNS_RECORDS`, `ARWEAVE_PEER_DNS_PORT`,
    `ARWEAVE_PEER_HEALTH_CHECK_INTERVAL_MS`, and related environment variables
  - Enabled by default with `ENABLE_ARWEAVE_PEER_EDS=true`; falls back to
    static `TRUSTED_NODE_HOST` when disabled
  - EDS files validated on startup with corrupt files automatically removed and
    re-seeded

- **HyperBEAM Request Loop Prevention**: Multi-layered detection of
  compute-origin requests to prevent infinite forwarding loops between gateways.
  Local data sources (cache, S3, database) always continue to serve data
  normally.
  - **Header-based detection**: Requests with configured headers (default:
    `ao-peer-port`) are identified as compute-origin and blocked from remote
    forwarding. Configurable via `SKIP_FORWARDING_HEADERS`.
  - **Via-chain loop detection**: New `X-AR-IO-Via` header tracks the chain of
    gateway identities across hops. When a gateway detects its own identity in
    the via chain, it stops forwarding to prevent loops. Gracefully degrades when
    `ARNS_ROOT_HOST` is not configured.
  - **User-Agent detection**: Requests with missing or empty User-Agent headers
    skip remote forwarding by default, catching HTTP clients like Erlang's `gun`
    (used by HyperBEAM) that don't send a User-Agent. Configurable via
    `SKIP_FORWARDING_EMPTY_USER_AGENT` (default: `true`). Additionally,
    `SKIP_FORWARDING_USER_AGENTS` allows specifying User-Agent substrings for
    case-insensitive matching (e.g., `HyperBEAM`).

- **CDB64 Documentation**: Comprehensive documentation for the CDB64 root
  transaction index feature
  - Operator guide (`docs/cdb64-guide.md`) covering configuration, monitoring,
    custom index sources, and troubleshooting
  - Format specification (`docs/cdb64-format.md`) detailing the CDB64 binary
    format, key encoding, and location types
  - Tools reference (`docs/cdb64-tools.md`) documenting all 6 CDB64 CLI tools
    with usage examples
  - Overview page (`docs/cdb64.md`) linking all CDB64 documentation
  - Documentation index (`docs/INDEX.md`) providing a navigable overview of all
    gateway documentation
  - CDB64 section added to README for quick orientation
  - Glossary entries for CDB64-related terms
  - Fixed default values for `CDB64_CACHE_SIZE` and `CDB64_ROOT_TX_INDEX_SOURCES`
    in `docs/envs.md`
  - Build script (`tools/build-cdb64-napi`) for compiling the native CDB64
    N-API module from source

### Changed

- Updated observer to increase default chunk observation sample rate to 20%

## [Release 68] - 2026-02-04

This is an **optional release** that introduces **A/B testing infrastructure for data sources** via the
new `SamplingContiguousDataSource`, enabling operators to safely evaluate
alternative retrieval strategies with controlled traffic exposure and built-in
metrics. It also includes **CDB64 location type renames** for improved clarity in
manifest schemas, and **data retrieval tooling enhancements** with content
validation options for easier gateway comparison testing.

### Added

- **SamplingContiguousDataSource for A/B Testing**: New data source wrapper that
  probabilistically routes requests through an experimental source
  - Enables safe A/B testing of new retrieval strategies with controlled traffic
    exposure
  - Two sampling strategies: `random` (per-request) or `deterministic`
    (consistent per ID using SHA-256 hash)
  - Configurable sampling rate (0-1) with validation to reject invalid values
  - New Prometheus metrics: `sampling_decision_total`, `sampling_request_total`,
    `sampling_latency_ms`
  - OpenTelemetry span instrumentation for sampled requests

- **Data Retrieval Tool Enhancements**: New options in `tools/test-data-retrieval`
  for content validation
  - `--bytes <n>`: Fetch first N bytes using HTTP Range headers for content
    comparison
  - `--max-size <bytes>`: Two-phase retrieval (HEAD then GET) for content under
    size threshold
  - RPS (requests per second) statistics in console and JSON output
  - SHA-256 hash computation for content comparison between gateways

### Changed

- **CDB64 Arweave Location Type Renames**: Renamed location types in CDB64
  manifests for clarity
  - `arweave-tx` → `arweave-id` (field: `txId` → `id`)
  - `arweave-bundle-item` → `arweave-byte-range` (field: `txId` → `rootTxId`,
    `offset` → `dataOffsetInRootTx`, added optional `dataItemId`, removed `size`
    from location)
  - Includes script to add `dataItemId` fields by reading ANS-104 headers
  - Updated bundled manifest with new format

## [Release 67] - 2026-01-29

This is a **recommended release** focusing on **CDB64 index accessibility** and
**operational observability**. CDB64 lookups are now enabled by default with a
shipped manifest covering data items up to block height 1,820,000, eliminating
the need for operators to generate or configure their own indexes. New
Prometheus metrics provide detailed visibility into root TX index and ANS-104
offset lookup performance. The release also includes tooling for uploading
partitioned CDB64 indexes to Arweave and an observer update with network gateway
fallback for more resilient reference resolution.

### Added

- **CDB64 Upload Tool**: New CLI tool (`tools/upload-cdb64-to-arweave`) to upload
  partitioned CDB64 indexes to Arweave via Turbo SDK
  - Three-phase workflow: upload partitions, poll for bundle offsets, upload
    manifest
  - Supports concurrent uploads with configurable parallelism (`--concurrency`)
  - Resumable: saves progress to output manifest, restart with `--resume`
  - Dry-run mode for cost estimation before uploading
  - Optional L1 manifest upload for permanent on-chain storage

- **Default Remote CDB64 Index**: Ships with a pre-built CDB64 manifest for
  remote index lookups on Arweave
  - Covers non-AO, non-Redstone data items with content types up to block height
    1,820,000
  - CDB lookups now enabled by default in `ROOT_TX_LOOKUP_ORDER`
    (`db,gateways,cdb,graphql`)
  - Default `CDB64_ROOT_TX_INDEX_SOURCES` points to shipped manifest

- **Prefix-Partitioned CDB64 Indexes**: CDB64 root TX indexes can now be split
  across 256 partition files based on key prefix
  - Each partition file (00.cdb - ff.cdb) contains records for keys starting
    with that byte
  - Enables parallel processing and reduces memory requirements for large
    indexes
  - Manifest file tracks partition metadata and record counts
  - Tools updated: `generate-cdb64-root-tx-index` and
    `generate-cdb64-root-tx-index-rs` support `--partitioned` flag

- **HTTP Request Concurrency Limit for Remote CDB64 Sources**: New environment
  variables limit concurrent HTTP requests across all remote CDB64 sources
  - `CDB64_REMOTE_MAX_CONCURRENT_REQUESTS`: Global limit on concurrent HTTP
    requests (default: 4)
  - `CDB64_REMOTE_SEMAPHORE_TIMEOUT_MS`: Maximum time to wait for a request slot
    before failing (default: 5000ms)
  - Prevents request pile-up when reading CDB files from HTTP/S3 endpoints
  - Requests exceeding the timeout fail fast rather than waiting indefinitely

- **Prometheus Metrics for Root TX Index and ANS-104 Offset Lookups**: New
  instrumentation to track lookup performance, cache effectiveness, and data
  source usage
  - `root_tx_lookup_total`: Per-source lookups by status and offset availability
  - `root_tx_lookup_duration_ms`: Per-source lookup latency
  - `root_tx_cache_hit_total` / `root_tx_cache_miss_total`: LRU cache
    effectiveness tracking
  - `composite_root_tx_lookup_total`: Final lookup outcome with winning source
  - `composite_root_tx_lookup_duration_ms`: Total composite lookup duration
  - `ans104_offset_lookup_total`: Offset lookups by method (`path_guided` vs
    `linear_search`)
  - `ans104_offset_lookup_duration_ms`: Offset lookup latency by method
  - `ans104_offset_path_depth`: Histogram of bundle nesting depth (buckets: 2,
    3, 4)

### Changed

- **Observer Update to `8fb7b2f`**: Updated observer with network gateway
  fallback for reference resolution
  - When explicit reference gateways fail or disagree, the observer now falls
    back to querying healthy gateways from the AR.IO network for consensus
  - Three operating modes: explicit only, explicit + network fallback (default),
    or network only for fully decentralized observation
  - New `REFERENCE_GATEWAY_HOSTS` supports multiple comma-separated fallback
    gateways with sequential failover
  - Network consensus uses configurable gateway selection criteria (pass rate,
    consecutive passes, epoch count) with stale-while-error caching
  - Treats 404/410 responses as authoritative "chunk not found" to avoid
    unnecessary network fallback queries
  - New metrics: `observer_network_fallback_total`,
    `observer_network_consensus_agreement`, `observer_network_eligible_gateways`
  - See `docs/envs.md` for full configuration options

### Fixed

- Updated `TRUSTED_ARNS_GATEWAY_URL` default value in documentation from
  `arweave.dev` to `ar-io.net` to match the actual code default

## [Release 66] - 2026-01-21

This is a **recommended release** focusing on **nested bundle performance** and
**resource stability**. Key improvements include path-based CDB64 indexes that
enable O(n) navigation through deeply nested bundles (vs O(n*m) previously), new
CDB64 verification tools for validating index completeness, and several fixes
for stream leaks and file descriptor exhaustion. Abort signal handling has also
been extended to contiguous data requests, preventing wasted work when clients
disconnect.

### Added

- **Nested Bundle Path Support**: CDB64 root TX indexes now support path-based
  formats for efficient retrieval of deeply nested data items
  - New value formats: path-only `{p}` and path-complete `{p, i, d}` store the
    bundle hierarchy from root to parent
  - Enables O(n) navigation through nested bundles instead of O(n*m) linear
    search at each level
  - GraphQL parent chain traversal now collects and returns path information
  - `Ans104OffsetSource` uses path-guided navigation with fallback to legacy
    behavior
  - CDB64 tools updated to generate and export path-based indexes

- **Reference Gateway Comparison in Data Retrieval Tool**: New `--reference`
  option in `tools/test-data-retrieval` compares responses against a reference
  gateway
  - Compares status codes, Content-Length, and Content-Type headers
  - Shows detailed mismatch info in verbose mode
  - Includes comparison stats in summary and JSON output

- **CDB64 Verification Tool**: New CLI tool for validating CDB64 root TX index
  completeness (`tools/verify-cdb64`)
  - Random and sequential sampling modes for efficient verification of large
    indexes
  - Value comparison mode to validate stored values match actual gateway
    responses
  - Detailed statistics on index coverage and discrepancies

- **CDB64 Verification in Data Retrieval Tool**: The data retrieval test tool
  now verifies CDB64 index entries for every tested ID, reporting match rates
  and discrepancies

### Changed

- **Abort Signal Propagation to Data Requests**: Extended abort signal handling
  to contiguous data sources
  - Client disconnections now abort all in-flight data requests, not just chunk
    requests
  - Prevents wasted bandwidth and resources on abandoned data transfers

### Fixed

- **Stream Leak in ANS-104 Worker**: Fixed stream not being properly destroyed
  in `hashDataItemData` function, which could cause file descriptor exhaustion
  during bundle processing

- **Stream Leak in Data-Root Worker**: Fixed read stream not being destroyed
  after use, preventing file descriptor leaks during data root verification

- **fs-cleanup-worker Resource Exhaustion**: Improved cleanup worker to reduce
  file descriptor pressure
  - Throttled concurrent file deletes to prevent overwhelming the filesystem
  - Reduced parallel file operations during cleanup cycles

- **Abort Signal Race Condition in Cached Chunk Requests**: Fixed a race
  condition in `ArIOChunkSource` where subsequent callers could not abort their
  requests when sharing a cached promise
  - Added `withAbortSignal()` helper that races a promise against the caller's
    abort signal
  - Each caller now respects their own abort signal independently when using
    cached chunk promises

## [Release 65] - 2026-01-14

This is a **recommended release** focusing on **resource stability** and **remote
index distribution**. Key improvements include AbortSignal propagation through
the chunk retrieval pipeline to prevent wasted work when clients disconnect,
reduced parallel peer requests and concurrency limits to lower resource pressure,
and the ability to load CDB64 root TX indexes from remote sources (Arweave
transactions, bundle data items, or HTTP URLs). Several high-severity dependency
vulnerabilities have also been addressed.

### Added

- **Remote CDB64 Index Sources**: CDB64 root TX indexes can now be loaded from
  remote sources in addition to local files (GitHub #569)
  - Arweave transactions: specify a 43-character TX ID
  - Bundle data items: use `txId:offset:size` format for indexes stored within bundles
  - HTTP URLs: load indexes from S3, CDNs, or dedicated index servers
  - New environment variables:
    - `CDB64_ROOT_TX_INDEX_SOURCES`: comma-separated list of sources (local paths,
      TX IDs, bundle items, URLs)
    - `CDB64_REMOTE_RETRIEVAL_ORDER`: data sources for fetching remote indexes
      (gateways, chunks, tx-data)
    - `CDB64_REMOTE_CACHE_MAX_REGIONS`: max cached byte-range regions per source
    - `CDB64_REMOTE_CACHE_TTL_MS`: TTL for cached regions
    - `CDB64_REMOTE_REQUEST_TIMEOUT_MS`: request timeout for remote sources
  - Intelligent caching: CDB64 headers (4KB) cached permanently, other regions
    use LRU cache with configurable size and TTL

- **Chunk Retrieval Load Testing Tool**: New CLI tool for load testing chunk
  retrieval endpoints (`tools/test-chunk-retrieval`)
  - Configurable concurrency with `--concurrency` flag (default: 10)
  - Duration mode (`--duration`) and count mode (`--count`) for flexible testing
  - File descriptor tracking (`--track-fds <pid>`) for resource monitoring
  - Error categorization with resource exhaustion detection (EMFILE, etc.)
  - Response time percentiles and requests/second metrics

### Changed

- **AbortSignal Propagation in Chunk Retrieval**: Client disconnections now
  abort all downstream operations promptly
  - Adds middleware that attaches AbortSignal to all requests
  - Propagates signal through retrieval service, composite sources, and caches
  - Returns HTTP 499 status for client-aborted requests
  - Prevents wasted work when clients disconnect during chunk requests

- **Reduced Parallel Peer Requests**: Lowered parallel peer request count from
  3 to 2 to reduce resource pressure during chunk retrieval

- **Chain Fallback Concurrency Limit**: Added concurrency limit to
  CompositeTxBoundarySource for chain fallback operations to prevent resource
  exhaustion from expensive binary search operations

- **Node.js Update**: Bumped Node.js from 20.11.1 to 20.19.6 (latest available
  Docker image)

- **Dependency Security Updates**: Addressed high-severity vulnerabilities
  - Added resolution for `qs@6.14.1` (fixes DoS via memory exhaustion in Express)
  - Added resolution for `cookie@0.7.0` (fixes out-of-bounds character handling)
  - Updated `@aws-sdk/client-dynamodb` and `@aws-sdk/credential-providers` to
    3.968.0

### Fixed

- **Abort Losing Parallel Peer Requests**: Parallel peer chunk requests now
  properly abort when one peer succeeds, freeing resources immediately instead
  of letting losing requests complete wastefully

- **Dead TxOffsetSource Code Removed**: Removed obsolete TxOffsetSource
  implementation files that were superseded by TxBoundarySource refactor

## [Release 64] - 2026-01-07

This is an **optional release** focusing on observer improvements and container
reliability. Key changes include the updated observer with continuous observation
mode support and explicit file descriptor limits for core and envoy services that
may improve reliability for some operators.

### Changed

- **Observer Update**: Updated observer to version `e34a7f0` with continuous
  observation mode support
  - New `OBSERVER_STATE_PATH` environment variable for configuring observer
    state storage location (default: `./data/observer`)
  - Added volume mount for observer state persistence across container restarts
  - Increased default chunk observation sample rate to 10%

- **File Descriptor Limits**: Added explicit `ulimits` configuration for core
  and envoy services in docker-compose.yaml
  - Sets `nofile` soft/hard limits to 65536 for both services
  - Ensures consistent behavior across different host configurations
  - May help resolve connection issues some operators have been experiencing

## [Release 63] - 2025-12-22

This is an **optional release** focusing on operator tooling and observability
improvements. Key additions include a data retrieval testing tool for gateway
validation, separate credentials support for legacy S3 chunk sources, and
OTEL-Winston integration for distributed trace correlation in logs.

### Added

- **CDB64 Extension Support**: Accept `.cdb64` file extension in addition to
  `.cdb` for CDB64 root TX index files

- **Data Retrieval Testing Tool**: New CLI tool for testing data item retrieval
  from a gateway using TX/data item IDs from a CSV file
  (`tools/test-data-retrieval`)
  - Sequential mode: streams through file line by line
  - Random mode: O(1) random byte seeking, no file scan required
  - Continuous mode: runs indefinitely until Ctrl+C, writes JSON results to file
  - Configurable concurrency for parallel requests
  - Comprehensive statistics: success/failure rates, response time percentiles
    (p50/p95/p99), cache hit rates, status codes, bytes transferred
  - Console table and JSON output formats

- **Separate Credentials for Legacy S3 Chunk Source**: Add ability to configure
  separate AWS credentials for the legacy S3 chunk data source, enabling access
  to S3 buckets in different AWS accounts
  - `LEGACY_AWS_S3_ACCESS_KEY_ID`: AWS access key for legacy S3 bucket
  - `LEGACY_AWS_S3_SECRET_ACCESS_KEY`: AWS secret key for legacy S3 bucket
  - `LEGACY_AWS_S3_REGION`: AWS region (required when using separate credentials)
  - `LEGACY_AWS_S3_ENDPOINT`: Custom endpoint (optional)
  - Falls back to main AWS client when legacy credentials are not set

- **Docker Compose Environment Variables**: Expose additional environment
  variables in docker-compose.yaml for legacy chunk sources and chunk
  rebroadcasting
  - Legacy S3/PostgreSQL chunk source configuration
  - Chunk rebroadcast rate limiting and deduplication settings

- **OTEL Winston Integration for Trace ID Correlation**: Automatic injection of
  OpenTelemetry trace context (`trace_id`, `span_id`, `trace_flags`) into all
  Winston log entries, enabling correlation of logs with distributed traces
  - All logs within a request share the same `trace_id` for easy filtering
  - Spans properly nested via `span_id` for hierarchical trace analysis
  - Request handlers (data, chunk, ArNS) wrapped with active OTEL context
  - ArNS middleware includes span attributes for resolution timing and results
  - `startChildSpan()` helper auto-detects parent from active context

### Fixed

- Fix missing `parentSpan` parameter in `handleRangeRequest` calls for proper
  OTEL trace hierarchy in range requests

## [Release 62] - 2025-12-14

This is an **optional release** that introduces the CDB64-based historical root
TX index, a new lookup source for resolving data items to their root
transactions. The CDB64 format provides fast O(1) lookups from pre-generated
index files, enabling efficient root TX resolution for historical data without
database queries. Pre-generated CDB64 index files will be made available in the
coming weeks.

### Added

- **CDB64-Based Historical Root TX Index**: New lookup source for root
  transaction IDs using the compact CDB64 file format
  - Enable by adding 'cdb' to `ROOT_TX_LOOKUP_ORDER` (e.g.,
    `db,cdb,gateways,graphql`)
  - Reads from `data/cdb64-root-tx-index` directory (configurable via
    `CDB64_ROOT_TX_INDEX_DATA_PATH` in Docker)
  - Multi-file directory support with runtime file watching (new `.cdb` files
    automatically loaded without restart)
  - File watching can be disabled via `CDB64_ROOT_TX_INDEX_WATCH=false`
  - CLI tools for index generation and data extraction:
    - SQLite to CDB64 export (`tools/export-sqlite-to-cdb64`)
    - CSV to CDB64 generation with RFC 4180 support
      (`tools/generate-cdb64-root-tx-index`)
    - Rust-backed CDB64 generation for high-throughput
      (`tools/generate-cdb64-root-tx-index-rs`)
    - CDB64 to CSV export (`tools/export-cdb64-root-tx-index`)
  - Binary compatibility with Rust cdb64-rs for cross-language interoperability
  - CDB64 file format specification documentation (`docs/cdb64-format.md`)

### Fixed

- Expose `ROOT_TX_LOOKUP_ORDER` environment variable in docker-compose.yaml
  (was missing from previous releases)

## [Release 61] - 2025-12-10

This release addresses potential memory growth issues observed on some r60 nodes
by converting unbounded caches to bounded LRU caches and adding cleanup for
stale peer chunk queues. Users currently on r60 experiencing memory issues
should upgrade. Users on earlier releases may want to wait until these
improvements have been confirmed in production.

### Changed

- **Observer Performance Improvements**: Updated bundled observer with chunk
  verification optimizations ported from ar-io-node
  - TX path Merkle proof parsing eliminates 7-10 API calls per chunk by
    extracting transaction boundaries directly from tx_path
  - Pre-computed offset-to-block mapping narrows binary search range by 97-99%
    (from ~1.8M to ~26K blocks)

### Fixed

- **Memory Leak Prevention**: Address potential memory growth vectors identified
  during OOM investigation on low-memory nodes
  - Add hourly cleanup of stale `peerChunkQueues` entries for peers no longer in
    the active peer list (only removes idle queues)
  - Convert `blockByHeightPromiseCache` and `txPromiseCache` from unbounded
    NodeCache to LRUCache with size limits (1000 blocks, 10000 transactions)
  - Convert SQLite dedupe caches (`insertDataHashCache`,
    `saveDataContentAttributesCache`) from unbounded NodeCache to LRUCache
    (10000 entries each)
  - Add `arweave_peer_chunk_queues_size` Prometheus gauge for monitoring

## [Release 60] - 2025-12-03

This is a **recommended release** due to significant chunk retrieval performance
improvements. AR.IO peer chunk fetching now uses parallel requests, reducing
worst-case latency from ~150 seconds to ~4 seconds. Additional optimizations
include tx_path Merkle proof validation to avoid expensive chain binary
searches, offset-based chunk cache lookups via symlinks, and a static
offset-to-block mapping that reduces block search iterations by ~29%.

### Added

- **OTEL Nested Bundle Sampling Policies**: Add targeted tail-sampling policies
  to detect scenarios where nested bundle offset issues could occur
  - `nested-bundle-policy`: Samples traces with `turbo.offsets_has_parent=true`
    (default: 5%, configurable via `OTEL_TAIL_SAMPLING_NESTED_BUNDLE_RATE`)
  - `offset-overwrite-risk-policy`: Samples traces where both offsets AND raw
    data paths executed (default: 10%, configurable via
    `OTEL_TAIL_SAMPLING_OFFSET_OVERWRITE_RATE`)
  - Adds `turbo.cache_path` diagnostic span attribute to identify which caching
    code path was used (rootParentInfo, parentInfo, or rawData)

- **Chunk Rebroadcasting**: Optional wrapper that asynchronously rebroadcasts
  chunks from configured sources (e.g., legacy S3) to the Arweave network
  - Fire-and-forget pattern that never blocks chunk fetches
  - Configurable via `CHUNK_REBROADCAST_SOURCES` environment variable
  - Includes rate limiting, deduplication cache, and concurrency controls
  - Disabled by default (empty `CHUNK_REBROADCAST_SOURCES`)

- **Block Search Optimization**: Ship static offset-to-block mapping to optimize
  binary search when looking up transactions by offset
  - Reduces block search iterations from ~21 to ~15 (~29% reduction)
  - Most significant impact during cold starts when block caches are empty
  - Each saved iteration means one fewer network call to fetch a block
  - Includes generation tool (`tools/generate-offset-mapping`) for updating
    mapping before releases

- **Chunk POST Early Termination and Smart Status Codes**: Reduce wasted
  resources on invalid chunk uploads and improve client error feedback
  - Early termination: Stop broadcasting after N consecutive 4xx failures when
    no peers have accepted the chunk (~96% reduction in wasted requests)
  - Smart status codes: Return most common peer error code instead of hardcoded
    500 (e.g., 400 for invalid chunks, 504 for timeouts)
  - New `CHUNK_POST_MAX_CONSECUTIVE_FAILURES` config variable (default: 5)
  - Only 4xx responses count toward consecutive failures (timeouts/5xx reset counter)

- **tx_path Chunk Validation**: Optimize chunk retrieval with a DB-first lookup
  strategy that falls back to tx_path Merkle proof validation for unindexed data
  - Lookup order: Database (fastest) → tx_path validation → Chain binary search (slowest)
  - tx_path proofs are cryptographically validated against the block's tx_root
  - Eliminates expensive chain binary search when tx_path is available from peers

- **Chunk Cache by Absolute Offset**: Enable chunk cache lookups by absolute
  weave offset for faster retrieval when chunk data is already cached
  - Creates symlinks indexed by absolute offset for O(1) cache lookups
  - Background worker periodically cleans up dead symlinks when cached data expires
  - Configurable via `ENABLE_CHUNK_SYMLINK_CLEANUP` (default: true) and
    `CHUNK_SYMLINK_CLEANUP_INTERVAL` (default: 24 hours)

### Changed

- **AR.IO Peer Chunk Retrieval Optimization**: Improved chunk retrieval
  performance from AR.IO network peers
  - Reduced per-peer request timeout from 10 seconds to 1 second
  - Changed from sequential to parallel peer requests (3 peers raced simultaneously)
  - Reduced retry strategy from 5 attempts to 2 attempts with different peers
  - Selects all peers upfront to ensure different peers on each retry attempt
  - Worst-case latency reduced from ~150 seconds to ~4 seconds
  - Maximum peer requests reduced from 15 to 6

## [Release 59] - 2025-11-24

This is a **recommended release** due to important fixes for nested bundle data
item offset handling that could cause incorrect data retrieval. The release
fixes offset calculations in both the TurboDynamoDB data source and database
root TX lookups, ensuring correct data is served for deeply nested bundle
items. It also includes fixes for ArNS manifest path encoding and Observer
wallet failure reporting for shared FQDN gateways. New features include a
dry-run mode for testing transaction uploads without posting to the network,
and a monitoring tool for historical DHA chunk nodes.

### Added

- **Historical DHA Chunk Nodes Monitoring Tool**: New operator utility for
  monitoring response times and availability of Arweave data endpoints
  (`tools/monitor-historical-dha-chunk-nodes`)
  - Monitors data-N (1-17) and tip-N (1-5) endpoints with configurable ranges
  - Continuous monitoring mode with real-time table output and statistics
  - JSON export with detailed results and metadata
  - Note: This is a special-purpose tool included for reference and potential
    usefulness to operators debugging data retrieval issues

- **Dry-Run Mode for Upload Testing**: New `ARWEAVE_POST_DRY_RUN` environment
  variable enables testing transaction and chunk uploads without posting to the
  Arweave network
  - When enabled, both `POST /tx` and `POST /chunk` requests are simulated with
    200 OK responses
  - Works on both port 3000 (Envoy) and port 4000 (direct to Node.js app)
  - Envoy routing is conditional: routes to core for dry-run, to Arweave nodes
    when disabled
  - Perfect for testing apps like ArDrive and large uploads without burning AR
    tokens
  - By default, transactions are validated (signature verification) and chunks
    are validated (merkle proof verification) before returning success
  - Set `ARWEAVE_POST_DRY_RUN_SKIP_VALIDATION=true` to skip validation for faster
    testing
  - Only the final network broadcast is skipped

### Changed

- When CDP API keys are provided (`CDP_API_KEY_ID` and `CDP_API_KEY_SECRET`),
  the gateway now automatically uses the Coinbase facilitator with enhanced
  Onramp integration, overriding the `X_402_USDC_FACILITATOR_URL` setting

### Fixed

- **PostgreSQL SSL Configuration**: Fixed inverted SSL flag logic where
  `LEGACY_PSQL_SSL_REJECT_UNAUTHORIZED=true` (default) was incorrectly disabling
  certificate validation instead of enabling it
  - Now correctly applies strict SSL validation by default
  - Set to `false` to disable certificate validation for cloud providers with
    self-signed certificates

- **PostgreSQL Connection Timeouts**: Added timeout configuration for the legacy
  PostgreSQL chunk metadata source to prevent system hangs
  - Server-side `statement_timeout` (default: 5s) prevents queries from running
    forever
  - `idle_in_transaction_session_timeout` (default: 10s) cleans up stuck
    transactions
  - Connection pool settings: `max`, `idle_timeout`, `connect_timeout`,
    `max_lifetime`
  - All settings configurable via environment variables
    (`LEGACY_PSQL_STATEMENT_TIMEOUT_MS`, etc.)
  - Graceful Postgres connection cleanup on shutdown
  - Prevents the chunk serving system from becoming completely unresponsive when
    Postgres is slow or unreachable

- **Security Dependency Updates**: Fixed 6 security vulnerabilities identified by
  `yarn audit`
  - Added `tar@7.5.2` resolution to fix moderate severity race condition in
    duckdb-async dependency chain
  - Upgraded `@cucumber/cucumber`, `@testcontainers/localstack`, `testcontainers`,
    and `rimraf` to fix high severity glob CLI command injection vulnerabilities
  - Upgraded `viem` to ^2.39.3
  - All existing resolutions (secp256k1, elliptic, ws, semver) remain required for
    vulnerabilities in `@dha-team/arbundles` transitive dependencies

- **ArNS Manifest Path Encoding**: Fixed manifest paths with URL-encoded
  characters (e.g., spaces as `%20`) failing when accessed via ArNS subdomain
  - Direct TX ID access worked because Express auto-decodes `req.params`
  - ArNS subdomain access failed because `req.path` is not auto-decoded
  - Now decodes manifest paths in the ArNS middleware for consistent behavior
- **TurboDynamoDB Data Source**: Fixed nested bundle data items having incorrect
  `rootDataItemOffset` values when retrieved from Turbo's DynamoDB
  - The raw data path was overwriting correct absolute offsets cached from the
    `rootParentInfo` path with incorrect values (offset: 0, dataOffset: payloadDataStart)
  - Now preserves the correct offsets by only caching size and contentType from raw data
- **Database Root TX Offset Lookup**: Fixed `getRootTxFromData` returning incorrect
  offset for nested bundle data items
  - Was returning `root_parent_offset` (parent bundle offset) instead of
    `root_data_item_offset` (absolute data item offset)
  - Added fallback calculations for `rootDataItemOffset` and `rootDataOffset` when
    pre-computed values are NULL
- **Observer**: Updated to 2515e6a - Fixed incorrect wallet failure reporting
  for shared FQDN gateways
  - When multiple wallets share the same FQDN, now correctly identifies which
    specific wallets failed ownership verification
  - Reports non-matching wallets as failed even when gateway passes overall
  - Ensures save-observations contract interactions accurately reflect actual
    ownership failures
- BUNDLER_URLS environment variable was missing from docker-compose.yaml
- x402 payment processor now correctly uses Coinbase CDP facilitator when CDP
  credentials are configured

## [Release 58] - 2025-11-10

This is a **recommended release** due to significant improvements in data
retrieval efficiency and payment system reliability. This release introduces a
new raw binary chunk data endpoint providing ~40% bandwidth savings,
comprehensive rate limit balance management APIs, and intelligent OpenTelemetry
tail-based sampling for cost-effective observability. The release also includes
critical payment validation fixes and enhanced bundler service discovery for
improved client integration.

> **⚠️ EXPERIMENTAL FEATURES**: The rate limiter and x402 payment protocol are
> experimental features subject to change. API endpoints, parameters, behavior,
> and configuration options (environment variables) may evolve in future
> releases as these systems continue to be developed. See
> [docs/x402-and-rate-limiting.md](docs/x402-and-rate-limiting.md) for
> comprehensive documentation.

### Added

- **Raw Binary Chunk Data Endpoint**: New `/chunk/<offset>/data` endpoint
  returns raw binary chunk data (`application/octet-stream`) with metadata in
  response headers instead of base64url-encoded JSON
  - Provides ~40% bandwidth savings compared to the base64url-encoded
    `/chunk/<offset>` endpoint
  - Supports both GET and HEAD requests
  - Returns comprehensive metadata in custom headers:
    - `X-Arweave-Chunk-Data-Path` - Base64url-encoded merkle proof path for
      chunk verification
    - `X-Arweave-Chunk-Data-Root` - Merkle tree root hash
    - `X-Arweave-Chunk-Start-Offset` - Absolute start offset of chunk in the
      weave
    - `X-Arweave-Chunk-Relative-Start-Offset` - Chunk offset relative to its
      transaction's data
    - `X-Arweave-Chunk-Read-Offset` - Number of bytes to skip from the start
      of the returned chunk to reach the requested offset
    - `X-Arweave-Chunk-Tx-Data-Size` - Total data size of the transaction
      containing this chunk
    - `X-Arweave-Chunk-Tx-Id` - Transaction ID containing this chunk
    - `X-Arweave-Chunk-Tx-Start-Offset` - Absolute start offset of the
      transaction in the weave
    - `X-Arweave-Chunk-Tx-Path` - Transaction-level merkle path (when available)
    - `X-Arweave-Chunk-Source` - Data source identifier (e.g., `arweave`,
      `trusted-gateway`)
    - `X-Arweave-Chunk-Source-Version` - Version identifier of the data source
  - Supports ETag-based conditional requests (304 Not Modified)
  - Supports `Content-Digest` header (RFC 9530) for data integrity verification
  - Rate limited at 256 KiB (raw chunk size) vs. 360 KiB for base64url
    endpoint, resulting in lower per-chunk fees
- **Bundler Service Discovery**: The `/ar-io/info` endpoint now includes a
  `bundlers` field for client service discovery
  - Configurable via `BUNDLER_URLS` environment variable (comma-separated URLs)
  - Defaults to `https://turbo.ardrive.io/`
  - URLs are validated on startup with descriptive error messages
  - Returns array of objects with `url` property
  - Enables clients to discover available bundler services for data uploads
- **Rate Limit Balance Management API**: New REST API endpoints for querying
  and managing rate limit bucket balances
  - `GET /ar-io/rate-limit/ip/:ip` - Query IP-based rate limit bucket balance
  - `POST /ar-io/rate-limit/ip/:ip` - Top up IP-based bucket via x402 payment
    or admin API key
  - `GET /ar-io/rate-limit/resource` - Query resource-based bucket balance with
    optional query parameters (`method`, `resource`, `host`)
  - `POST /ar-io/rate-limit/resource` - Top up resource-based bucket via x402
    payment or admin API key
  - Dual authentication: x402 payment protocol (public) or admin API key
    (private/testing via `ADMIN_API_KEY` environment variable)
  - 10x capacity multiplier applied to x402 payments compared to raw admin
    top-ups
  - Smart defaults for resource endpoints (method defaults to GET, host
    defaults to current request host)
  - Enables programmatic balance queries and top-ups for testing and automated
    payment workflows
- **OpenTelemetry Collector with Tail-Based Sampling** ⚠️ **EXPERIMENTAL**: New
  OTEL Collector sidecar in docker-compose deployments implements intelligent
  tail-based sampling to reduce telemetry costs by 80-95% while maintaining
  complete visibility into errors, performance issues, and paid traffic. This
  feature is experimental and subject to change in future releases
  - Five intelligent sampling policies make decisions after traces complete:
    - 100% of traces with errors (5xx responses, exceptions)
    - 100% of slow requests exceeding configurable threshold (default: 2 seconds)
    - 100% of x402 verified payment requests for billing and compliance
    - 100% of paid rate limit token usage for revenue tracking
    - 1% (configurable) of successful, fast, unpaid requests for baseline metrics
  - Traces flow through architecture: ar-io-node → otel-collector → telemetry backend
  - Support for multiple telemetry backends via environment variables:
    - Honeycomb (`OTEL_COLLECTOR_HONEYCOMB_API_KEY`)
    - Grafana Cloud Tempo (`OTEL_COLLECTOR_GRAFANA_CLOUD_API_KEY`)
    - Datadog (`OTEL_COLLECTOR_DATADOG_API_KEY`)
    - New Relic (`OTEL_COLLECTOR_NEW_RELIC_API_KEY`)
    - Elastic APM (`OTEL_COLLECTOR_ELASTIC_API_KEY`)
  - Configurable sampling rates for each policy via environment variables:
    - `OTEL_TAIL_SAMPLING_SUCCESS_RATE` - Baseline success sampling (default: 1%)
    - `OTEL_TAIL_SAMPLING_SLOW_THRESHOLD_MS` - Slow request threshold (default: 2000ms)
    - `OTEL_TAIL_SAMPLING_ERROR_RATE` - Error sampling rate (default: 100%)
    - `OTEL_TAIL_SAMPLING_SLOW_RATE` - Slow request sampling rate (default: 100%)
    - `OTEL_TAIL_SAMPLING_PAID_TRAFFIC_RATE` - Paid traffic sampling (default: 100%)
    - `OTEL_TAIL_SAMPLING_PAID_TOKENS_RATE` - Paid token sampling (default: 100%)
  - Optional deployment via docker-compose profile (`docker compose --profile otel up`)
  - Set `OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318` when using the
    profile, or configure it to send traces directly to an external backend to
    bypass the local collector
  - Enhanced span attributes for paid traffic tracking including client IP,
    payment verification status, and token consumption

### Changed

- **Chunk Request Routing**: Removed Envoy proxy route for GET `/chunk/`
  requests - all chunk endpoints now route directly to the AR.IO gateway
  application instead of being proxied to trusted Arweave nodes. This enables
  rate limiting, x402 payment processing, and local caching for chunk requests
- **Transaction-Level Merkle Path Support**: The `/chunk/<offset>` endpoint now
  includes `tx_path` in JSON responses when available (both GET and HEAD
  requests), providing transaction-level merkle proofs
- **Observer**: Updated to fcd0f36 - Doubled offset observation sample rate to
  2% for improved network robustness
- **AR.IO Info Endpoint Structure**: The `bundlers` field in `/ar-io/info`
  endpoint response is now nested under `services.bundlers` instead of being a
  top-level field for improved API organization and future extensibility
- **Rate Limiter Bucket Keys**: Standardized bucket key format across Redis and
  Memory rate limiter implementations - extracted shared utility functions to
  `src/limiter/utils.ts` for consistent key generation
- **Browser Paywall for Chunk Requests**: Optimized payment flow for chunk
  endpoints (`/chunk/*`) in browser paywall mode to use direct URL payment
  instead of redirect endpoint
  - **Browser only**: This optimization only affects requests detected as browser
    requests (Accept includes text/html AND User-Agent includes Mozilla)
  - **API flows unchanged**: Programmatic API payments continue to work the same
    way for all endpoints
  - Reduces latency by eliminating redirect round-trip for small chunk requests
  - Paywall sends payment directly to original chunk URL with x-payment header
  - Payment verification, settlement, and token granting happen inline before
    serving content
  - Other browser endpoints (transactions, manifests) continue using redirect
    endpoint for larger content
  - Both approaches use the same payment verification and settlement process

### Documentation

- Updated OpenAPI specification with comprehensive documentation for new
  `/chunk/<offset>/data` endpoint (GET and HEAD methods)
- Improved OpenAPI specification with complete header documentation for
  transaction data and chunk endpoints, including ANS-104 bundle navigation
  headers, data verification headers, and detailed examples with offset
  calculations
- Updated rate limiting documentation to include both chunk endpoint pricing
  models
- Updated glossary to reference both chunk endpoint formats
- Updated x402 and rate limiting documentation to explain chunk-specific direct
  payment flow and general redirect flow for browser paywall requests

### Fixed

- **Payment Processor and x402 Validation**: Multiple improvements to payment
  validation and error handling
  - Prevent infinite redirect loop on payment failure - browser paywall now
    shows user-friendly error page instead of redirecting
  - Validate payment type before settlement to prevent charging users without
    granting rate limit tokens
  - Prevent silent success when payment settlement cannot grant tokens - now
    returns error instead of charging without providing service
  - Validate resource target format before settling payment to prevent invalid
    payment processing
  - Validate Host header presence and format before processing payments to
    prevent malformed payment requests
  - Use configured capacity multiplier instead of hardcoded value in payment
    processor for consistent multiplier application
  - Update paywall redirect URL to use new canonical path for consistency with
    updated route structure
- **Rate Limiter Configuration**: Fixed configuration handling for consistent behavior
  - Ensure consistent resource key normalization across Redis and Memory rate
    limiter implementations to prevent cache misses
  - Use configurable capacity multiplier in rate limit routes instead of
    hardcoded values for consistency with payment processor

## [Release 57] - 2025-11-03

This is a **recommended release**. This release focuses on improving gateway
infrastructure with enhanced CDN compatibility and a new gateway-based offset
discovery system. Key improvements include a new root transaction index using
HEAD requests to AR.IO gateways, configurable Cache-Control headers for better
CDN integration, and numerous bug fixes for proxy support. The release also
includes extensive documentation improvements.

> **⚠️ EXPERIMENTAL FEATURES**: The rate limiter and x402 payment protocol are
> experimental features subject to change. API endpoints, parameters, behavior,
> and configuration options (environment variables) may evolve in future
> releases as these systems continue to be developed. See
> [docs/x402-and-rate-limiting.md](docs/x402-and-rate-limiting.md) for
> comprehensive documentation.

### Added

- **GatewaysRootTxIndex for Offset Discovery**: New root transaction index
  using HEAD requests to AR.IO gateways for discovering data item offsets
  - Multi-gateway support with priority tiers and automatic fallback (single
    attempt per gateway to prevent thundering herd)
  - Per-gateway rate limiting with TokenBucket
  - LRU caching for offset results
  - Configuration via `GATEWAYS_ROOT_TX_URLS`,
    `GATEWAYS_ROOT_TX_REQUEST_TIMEOUT_MS`,
    `GATEWAYS_ROOT_TX_RATE_LIMIT_BURST_SIZE`,
    `GATEWAYS_ROOT_TX_RATE_LIMIT_TOKENS_PER_INTERVAL`,
    `GATEWAYS_ROOT_TX_RATE_LIMIT_INTERVAL`, `GATEWAYS_ROOT_TX_CACHE_SIZE`
- **Configurable Cache-Control Private Directive**: CDN compatibility via
  `CACHE_PRIVATE_SIZE_THRESHOLD` (default: 100 MB) and
  `CACHE_PRIVATE_CONTENT_TYPES` environment variables
  - Adds `private` directive to Cache-Control headers for content exceeding
    size threshold or matching content types
  - Ensures rate limiting and x402 payment requirements are enforced even when
    CDNs are deployed in front of ar-io-node
- **Enhanced Rate Limiting Observability**: Client IP now logged separately in
  rate limit exceeded messages for better debugging and monitoring

### Fixed

- **Proxy Support Fixes**:
  - Fixed x402 resource URLs to use `SANDBOX_PROTOCOL` when behind reverse
    proxies/CDNs
  - Fixed inconsistent IP extraction between rate limiter bucket keys and
    allowlist checks
- **Chunk Endpoint Performance**: Apply rate limits before expensive txResult
  lookup
  - Reordered operations to check rate limits first, improving performance
    under high load
- **Cache-Control Content Type Matching**: Normalize content types by stripping
  parameters (e.g., `text/html; charset=utf-8` → `text/html`)
  - Ensures proper Cache-Control header matching for configured content types

### Documentation

- Comprehensive rate limiting documentation cleanup (~200-300 lines of
  duplication removed)
- Documented all 4 rate limit metrics (request, IP, chunk, x402 token
  consumption)
- Added automated payment workflow testing examples for x402
- Removed private key export recommendations from x402 testing examples
- Clarified complete IP extraction fallback order for proxy scenarios
- Clarified Cloudflare header extraction behavior
- Removed redundant mentions of x402 requiring rate limiter

## [Release 56] - 2025-10-27

This is a recommended release due to fixes for nested bundle offset
calculations.

This release continues the x402 payment protocol expansion from Release 55,
extending payment and rate limiting support to the chunk endpoint and adding
comprehensive operator documentation. The `/ar-io/info` endpoint now exposes
rate limiter and payment configuration for programmatic gateway discovery.
This release also includes important fixes for nested bundle offset calculations
that could affect data retrieval, making it a recommended upgrade for all
operators.

> **⚠️ EXPERIMENTAL FEATURES**: The rate limiter and x402 payment protocol are
> experimental features subject to change. API endpoints, parameters, behavior,
> and configuration options (environment variables) may evolve in future
> releases as these systems continue to be developed. See
> [docs/x402-and-rate-limiting.md](docs/x402-and-rate-limiting.md) for
> comprehensive documentation.

### Added

- **Chunk Endpoint Payment and Rate Limiting**: Added x402 payment and rate
  limiting support to `GET /chunk/:offset` endpoint for gateway monetization
  and traffic control:
  - Uses fixed size assumption (~360 KiB) for predictable pricing without
    waiting for chunk retrieval
  - Configurable via `CHUNK_GET_BASE64_SIZE_BYTES` environment variable
    (default: 368,640 bytes)
  - HEAD requests consume one token (to prevent spam)
  - 304 Not Modified responses consume one token (to prevent spam)
  - Compatible with all existing x402 and rate limiter configuration
- **Configuration Validation**: Added startup validation that ensures
  `ENABLE_RATE_LIMITER=true` when `ENABLE_X_402_USDC_DATA_EGRESS=true`. The
  application will fail to start with a clear error message if x402 is enabled
  without the rate limiter, since x402 payments require rate limiting to
  function (402 responses are only sent when rate limits are exceeded)
- **Gateway Info Endpoint**: The `/ar-io/info` endpoint now exposes rate limiter
  and x402 payment configuration when these features are enabled. This allows
  clients to programmatically discover gateway capabilities, pricing, and limits.
  New optional response fields:
  - `rateLimiter` - Per-resource and per-IP bucket capacities, refill rates, and
    byte convenience fields (when `ENABLE_RATE_LIMITER=true`)
  - `x402` - Payment network, wallet address, facilitator URL, per-byte pricing
    with min/max bounds, example costs for common sizes (1KB/1MB/1GB), and
    capacity multiplier for paid tier (when `ENABLE_X_402_USDC_DATA_EGRESS=true`)
- **x402 and Rate Limiter Documentation**: Added comprehensive operator guide at
  `docs/x402-and-rate-limiting.md` covering x402 payment protocol and rate
  limiting configuration:
  - Configuration via `.env` files with detailed examples
  - Secrets management using volume mounts for Coinbase Develop Program
    credentials
  - Complete list of rate limited endpoints including `GET /chunk/:offset`
  - Token consumption patterns and pricing models for all endpoints
  - Integration with x402 facilitator services
  - Testing and troubleshooting guidance
- **Coinbase Developer Platform Environment Variables**: Added environment
  variables for Coinbase Developer Platform (CDP) integration:
  - `CDP_API_KEY_ID` - CDP API key identifier
  - `CDP_API_KEY_SECRET` - CDP API secret key
  - `CDP_API_KEY_SECRET_FILE` - Load CDP secret from file for improved security

### Changed

- **Glossary**: Added new "Rate Limiter & x402 Payment Protocol" section
  consolidating related terms:
  - Facilitator - Payment verification and settlement service
  - Rate Limiter - Traffic control system overview
  - Rate Limiter Token Types - Paid vs regular token pools
  - Token Bucket Algorithm - Rate limiting algorithm details
  - x402 Protocol - HTTP 402 payment protocol definition
- **CDP Environment Variables**: Refactored Coinbase Developer Platform API key
  configuration:
  - Removed `X_402_CDP_CLIENT_KEY_FILE` (client key is public, doesn't need
    file-based loading)
  - Split into separate `CDP_API_KEY_ID` and `CDP_API_KEY_SECRET` variables
  - Added `CDP_API_KEY_SECRET_FILE` for secure file-based loading of sensitive
    API secret

### Fixed

- **Docker Compose Configuration**: Added `ENABLE_DATA_ITEM_ROOT_TX_SEARCH` and
  `ENABLE_PASSTHROUGH_WITHOUT_OFFSETS` environment variables to
  `docker-compose.yaml`, `.env.example`, and `docs/envs.md`. These options
  control offset-aware data source behavior and were previously only defined in
  `src/config.ts`, making them unavailable for Docker Compose users to configure
  via `.env` files
- **Data Handler Rate Limiting**: Fixed rate limiting for non-indexed data by:
  - Removing `dataAttributes !== undefined` check that prevented rate limiting
    before data indexing
  - Using `data.size` (always available) as primary source for content size
    calculation with fallback to `dataAttributes?.size`
  - Aligning content size calculation with actual `Content-Length` header values
  - Ensuring consistent rate limiting across all data endpoints (raw data,
    manifest, and bundled data)
- **Nested Bundle Data Item Offset Calculation**: Fixed multiple
  offset calculation issues affecting nested bundle data item retrieval:
  - Corrected Turbo DynamoDB dataOffset to use absolute semantics (offset +
    headerSize) instead of relative semantics, ensuring consistency with bundle
    parsing and type documentation
  - Fixed rootDataItemOffset calculation to include target item's offset, not
    just parent dataOffset values
  - Fixed region boundary validation to handle offset=0 correctly (was
    incorrectly skipping validation due to truthy checks)
  - Added backward compatibility fallback for data without dataOffset attributes
- **304 Not Modified Pre-Charging**: Prevented 304 Not Modified responses from
  being pre-charged with full content size before checking If-None-Match
  headers. Now correctly predicts 304 responses and applies minimal token charge
  without denying legitimate cached requests

## [Release 55] - 2025-10-20

This is an optional release focused on x402 payment protocol improvements.

This release represents a major milestone in the gateway x402 payment protocol
implementation. The x402 capabilities have evolved from an MVP supporting only
limited data endpoints to a full, mostly feature-complete solution. The browser
paywall now uses redirect mode to properly handle content-type metadata, and
rate limiting has been extended to work correctly across all content delivery
paths including manifests, ArNS names, and range requests.

> **⚠️ EXPERIMENTAL FEATURES**: The rate limiter and x402 payment protocol are
> experimental features subject to change. API endpoints, parameters, behavior,
> and configuration options (environment variables) may evolve in future
> releases as these systems continue to be developed. See
> [docs/x402-and-rate-limiting.md](docs/x402-and-rate-limiting.md) for
> comprehensive documentation.

### Added

- **Token Consumption Metrics**: New `rate_limit_tokens_consumed_total`
  Prometheus counter for monitoring rate limiter usage with labels:
  - `bucket_type` (ip/resource) - Which bucket consumed tokens
  - `token_type` (paid/regular) - Which token pool was used
  - `domain` - Domain consuming the tokens
  - Enables monitoring and alerting on token consumption patterns
- **Environment Variables**:
  - `RATE_LIMITER_TYPE`: Configure rate limiter implementation ("memory" for
    development/testing, "redis" for production)
  - `CDP_API_KEY_SECRET_FILE`: Load CDP secret API key from file instead of
    environment variable for improved security (for Coinbase Onramp integration)
  - `RATE_LIMITER_ARNS_ALLOWLIST`: Comma-separated list of ArNS names that
    bypass rate limiting and payment verification

### Changed

- **Token Consumption Priority**: Changed token consumption order to prioritize
  regular tokens:
  - Regular tokens consumed first, then paid tokens
  - Paid tokens now act as overflow capacity instead of being consumed
    immediately
  - Paid token balance still provides bypass of per-resource rate limits
  - This change provides better value to paying users as paid tokens last
    longer
- **Rate Limiting and Payment Architecture**: Refactored internal architecture
  for improved maintainability (no operator-visible behavior changes beyond those
  listed above)

### Fixed

- **X402 Browser Paywall**: Implemented redirect mode to fix blob URL
  content-type handling issues:
  - Browser requests now receive proper redirects after payment verification
  - Resolves content-type metadata loss that occurred with blob URLs
  - Preserves original content metadata in browser delivery
- **Rate Limiting for Manifests and ArNS**: Fixed rate limits to correctly
  apply to manifest-resolved and ArNS resources:
  - Rate limits now apply after manifest resolution to actual content size
  - ArNS resources are now properly rate limited
  - Ensures consistent rate limiting across all content delivery paths
- **Range Request Token Consumption**: Fixed rate limiter to charge tokens
  based on actual bytes served in range requests instead of full content size
- **Rate Limiter Token Tracking**: Fixed internal token bucket tracking to
  properly record consumption in all edge cases
- **Token Consumption for Non-Data Responses**: Prevented token consumption for
  304 Not Modified and HEAD responses which don't transfer content data

## [Release 54] - 2025-10-13

This is a **recommended release** due to the improvements to chunk observation
and retrieval. The release enhances peer selection for chunk operations,
introduces experimental X402 payment protocol support, and enables offset
observation enforcement by default to strengthen network reliability.

> **⚠️ EXPERIMENTAL FEATURES**: The rate limiter and x402 payment protocol are
> experimental features subject to change. API endpoints, parameters, behavior,
> and configuration options (environment variables) may evolve in future
> releases as these systems continue to be developed. See
> [docs/x402-and-rate-limiting.md](docs/x402-and-rate-limiting.md) for
> comprehensive documentation.

### Added

- **X402 Payment Protocol (Experimental)**: Optional USDC-based payment
  system for accessing rate-limited content. This feature is experimental
  and will be rapidly built out in upcoming releases.
  - Dynamic content-based pricing (default: $0.0000000001/byte = $0.10/GB)
  - USDC payments via Coinbase facilitator on Base network (mainnet and
    testnet supported)
  - Rate limiter integration with 10x capacity multiplier for paid tier
  - Proportional bucket top-off capped to actual price paid
  - HTML paywall for browser clients, JSON responses for API clients
  - Settlement timeout protection (5s default)
  - Configuration via 13 new `X_402_*` environment variables (see
    `.env.example` for details)
  - **Note**: Currently only applies to `/<id>` and `/raw/<id>` endpoints
    for non-manifests
- Expanded default preferred chunk GET node pool from 12 to 22 nodes,
  adding data-13 through data-17 (5 additional data nodes) and tip-1
  through tip-5 (5 tip nodes) for improved redundancy and load
  distribution across the Arweave network.
- Added GraphQL as third fallback option in `ROOT_TX_LOOKUP_ORDER` (after
  db and turbo) to prevent lookup failures when Turbo's circuit breaker
  is open, enabling more resilient root transaction discovery.
- Added deterministic weight-based peer selection for chunk operations
  (both GET and POST). Peers are now sorted by weight in descending order
  and top N selected, ensuring preferred peers (weight 100) are always
  tried first instead of probabilistic selection.
- Added preferred peer weight preservation for chunk operations to prevent
  weight degradation for operator-configured preferred peers during
  temporary failures. Preferred peers maintain their initial weight (100)
  regardless of success/failure, honoring operator configuration while
  allowing discovered peers to adapt based on performance.

### Changed

- **Observer**: Enabled offset observation enforcement by default.
  `OFFSET_OBSERVATION_ENFORCEMENT_ENABLED` now defaults to `true` instead
  of `false`. Gateway assessments will fail if offset validation fails,
  strengthening network reliability requirements. Operators can opt-out by
  explicitly setting `OFFSET_OBSERVATION_ENFORCEMENT_ENABLED=false`.
- Reduced logging verbosity by moving DNS resolution and sync bucket
  operational logs from debug/info to silly level. DNS resolution messages
  ('Resolving hostname', 'Resolved IPv4/IPv6 addresses') and sync bucket
  updates ('Parsed ETF sync buckets', 'Updated sync buckets') now use
  silly level, while completion and peer selection messages remain at
  debug level for visibility.

### Fixed

- Fixed preferred peer weight preservation to only apply to chunk
  operations (GET/POST) instead of all operation categories. Previously,
  preferred chunk peers maintained constant weight across chain, getChunk,
  and postChunk operations. Now preferred chunk peers can undergo normal
  warming/cooling when used for chain operations, preventing indefinite
  selection of peers that perform poorly for chain operations while still
  maintaining constant weight for chunk operations.
- Fixed ANS-104 data item header parsing for Ethereum signatures (type 3)
  by using correct 65-byte uncompressed public key length instead of
  20-byte address length. This resolves "Invalid buffer" errors when
  parsing Ethereum-signed data items. Also updated
  `MAX_DATA_ITEM_HEADER_SIZE` from 6228 to 8257 bytes to account for
  MultiAptos signature type (largest supported), and replaced custom
  signature/owner length methods with `getSignatureMeta()` from arbundles
  library for consistency.
- Fixed root TX discovery to use non-blocking rate limiting instead of
  blocking when rate limits are reached. Services now use
  `tryRemoveTokens()` and skip rate-limited gateways/sources immediately
  rather than waiting indefinitely, preventing request blocking and
  improving responsiveness. Also fixed GraphQL service to return
  `dataSize` instead of incorrect `size` field.

### Known Issues

- The x402 browser paywall currently uses blob URLs for content delivery
  after successful payment. This causes issues with content-type handling
  and browser behavior as the blob URL doesn't preserve the original
  content metadata. **We plan to fix this in upcoming releases** by
  either contributing to the x402 SDK to add a page reload option, or
  implementing a custom paywall template that properly handles redirects
  after payment verification.

## [Release 53] - 2025-10-06

This is an optional release that introduces root transaction offset tracking
for nested bundles and observer performance improvements. The release enables
more efficient data retrieval through comprehensive offset tracking with Turbo
and GraphQL integration, while improving observer reliability with increased
chunk validation success rates.

### Added

- **Root Transaction and Offset Tracking**: Comprehensive offset tracking
  system for nested ANS-104 bundles:
  - Turbo `/offsets` endpoint integration for accurate root transaction
    discovery and offset calculations
  - Handles multi-level nested bundles with cumulative offset tracking
  - Cycle detection and maximum nesting depth protection (10 levels)
  - Database persistence of root transaction IDs and absolute offset values
- **GraphQL Root TX Index**: Dedicated GraphQL endpoint configuration for root
  transaction lookups:
  - `GRAPHQL_ROOT_TX_GATEWAYS_URLS`: JSON object mapping GraphQL endpoints to
    weights (default: `{"https://arweave-search.goldsky.com/graphql": 1}`)
  - Parent chain traversal with metadata extraction (content type, size)
  - Fallback mechanism when Turbo is unavailable
  - Configurable lookup order via `ROOT_TX_LOOKUP_ORDER` (default: "db,turbo")
- **Database Migration**: Added offset tracking columns to
  `contiguous_data_ids` table:
  - `root_transaction_id`: Top-level Arweave transaction containing the data
  - `root_data_item_offset`: Absolute position where data item headers begin in
    root bundle
  - `root_data_offset`: Absolute position where data payload begins in root
    bundle
- **HTTP Headers**: New headers exposing absolute root offset information:
  - `X-AR-IO-Root-Data-Item-Offset`: Enables direct byte-range requests to data
    item headers
  - `X-AR-IO-Root-Data-Offset`: Enables direct byte-range requests to data
    payloads
- **Outbound Rate Limiting for External APIs**: Token bucket rate limiting for
  outbound calls to Turbo and GraphQL services (separate from the Redis-based
  inbound rate limiter added in Release 52):
  - Turbo API: Configurable via `TURBO_ROOT_TX_RATE_LIMIT_BURST_SIZE` (default: 5),
    `TURBO_ROOT_TX_RATE_LIMIT_TOKENS_PER_INTERVAL` (default: 6),
    `TURBO_ROOT_TX_RATE_LIMIT_INTERVAL` (default: "minute")
  - GraphQL API: Configurable via `GRAPHQL_ROOT_TX_RATE_LIMIT_BURST_SIZE` (default: 5),
    `GRAPHQL_ROOT_TX_RATE_LIMIT_TOKENS_PER_INTERVAL` (default: 6),
    `GRAPHQL_ROOT_TX_RATE_LIMIT_INTERVAL` (default: "minute")
  - Prevents excessive API usage and respects external service limits (defaults
    to 6 requests per minute = 1 per 10 seconds)
- **Configuration Options**:
  - `ENABLE_DATA_ITEM_ROOT_TX_SEARCH`: Enable/disable root transaction search
    for data items in offset-aware sources (default: true)
  - `ENABLE_PASSTHROUGH_WITHOUT_OFFSETS`: Control whether offset-aware sources
    allow data retrieval without offset information (default: true)
  - Dedicated rate limiting configuration for Turbo and GraphQL root TX lookups
  - Separate GraphQL gateway configuration for root lookups vs data retrieval
- **Documentation and Testing**:
  - Comprehensive bundle offsets documentation in
    `docs/drafts/bundle-offsets.md`
  - Rate limiting behavior tests validating token accumulation and request
    delays
  - Enhanced test coverage for offset tracking and nested bundle scenarios

### Changed

- **Observer**: Increased `OFFSET_SAMPLE_COUNT` default from 3 to 4 to improve
  chunk validation success rate with early stopping
- Increased rate limiter defaults to accommodate larger response payloads:
  - `RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET`: 10,000 → 1,000,000 tokens (~10 MiB → ~976 MiB bucket capacity)
  - `RATE_LIMITER_IP_TOKENS_PER_BUCKET`: 2,000 → 100,000 tokens (~2 MiB → ~98 MiB bucket capacity)
  - Resource refill rate remains 100 tokens/second (~98 KiB/s)
  - IP refill rate remains 20 tokens/second (~20 KiB/s)
  - Note: 1 token = 1 KiB (where 1 KiB = 1,024 bytes) of response data, minimum 1 token per request
  - Rate limiter remains disabled by default (`ENABLE_RATE_LIMITER=false`)
- **Performance Optimization**: RootParentDataSource now uses pre-computed root
  offsets when available:
  - Skip bundle traversal entirely when offsets are cached in database
  - Direct offset-based data retrieval without parent chain traversal
  - Use `rootDataOffset` to skip headers when fetching data payloads
  - Significantly reduces latency for nested bundle data retrieval

### Fixed

- **Security**: Resolved transitive dependency vulnerabilities by adding yarn
  resolutions:
  - `ws@7.5.10`: Fixed DoS vulnerability when handling requests with many HTTP
    headers (CVE in ws <7.5.10)
  - `semver@7.6.3`: Fixed Regular Expression Denial of Service (ReDoS)
    vulnerability (CVE in semver <7.5.2)

## [Release 52] - 2025-09-29

This is a **recommended release** that introduces critical observer reliability
improvements and a new Redis-based rate limiting system. The release significantly
improves observer stability under load through reduced sample rates, optimized
timeouts, and better concurrency management. Additionally, it introduces a complete
rate limiting solution with token bucket algorithm and IP allowlist support for
enhanced DDoS protection.

> **⚠️ EXPERIMENTAL FEATURES**: The rate limiter and x402 payment protocol are
> experimental features subject to change. API endpoints, parameters, behavior,
> and configuration options (environment variables) may evolve in future
> releases as these systems continue to be developed. See
> [docs/x402-and-rate-limiting.md](docs/x402-and-rate-limiting.md) for
> comprehensive documentation.

### Added

- **Rate Limiter**: Complete Redis/Valkey-based rate limiting system with:
  - Token bucket algorithm with configurable limits per IP and resource
  - IP allowlist support with CIDR block matching
  - Lua scripts for atomic Redis operations
  - Support for both cluster and non-cluster Redis deployments
  - Configuration via environment variables:
    - `ENABLE_RATE_LIMITER`: Enable/disable rate limiting (default: false)
    - `RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET`: Resource token bucket size
      (default: 10000)
    - `RATE_LIMITER_RESOURCE_REFILL_PER_SEC`: Resource token refill rate
      (default: 100)
    - `RATE_LIMITER_IP_TOKENS_PER_BUCKET`: IP token bucket size (default: 2000)
    - `RATE_LIMITER_IP_REFILL_PER_SEC`: IP token refill rate (default: 20)
    - `RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST`: Comma-separated allowlist of
      IPs/CIDRs
    - `RATE_LIMITER_REDIS_ENDPOINT`: Redis endpoint (default: localhost:6379)
    - `RATE_LIMITER_REDIS_USE_TLS`: Enable TLS for Redis (default: false)
    - `RATE_LIMITER_REDIS_USE_CLUSTER`: Use Redis cluster mode (default: false)
- **Chunk Offset Concurrency Control**: Added
  `CHUNK_OFFSET_CHAIN_FALLBACK_CONCURRENCY` config (default: 5) to limit
  concurrent fallback requests to the Arweave network, preventing resource
  exhaustion under high load
- **Observer Metrics**: Added comprehensive Prometheus metrics for observer
  performance including:
  - Ownership, ArNS name, and offset assessment metrics with pass/fail tracking
  - Report generation timing and success/failure counters
  - Gateway assessment overall status tracking
  - AR.IO node release version as global label on all metrics
  - `/ar-io/observer/metrics` endpoint for Prometheus scraping

### Changed

- **Security Updates**: Updated dependencies to address security
  vulnerabilities:
  - @ar.io/sdk to 3.20.0
  - @dha-team/arbundles to 1.0.4
  - axios to 1.12.0
  - Multiple other minor/patch updates for security fixes
- **Observer Performance Improvements**:
  - Reduced default offset observation sample rate from 5% to 1% to minimize
    observation failures under load
  - Added quick chunk validation to skip expensive binary search operations
  - Reduced concurrent connections and serialized ownership checks for better
    reliability
  - Optimized timeout configurations (7 seconds) for more reliable assessments

### Fixed

- **GraphQL Pagination**: Corrected transaction ID sorting to match ClickHouse
  binary ordering, eliminating duplicate transactions across consecutive query
  pages
- **Security Vulnerabilities**:
  - Resolved critical elliptic ECDSA private key extraction vulnerability
  - Resolved secp256k1 ECDH private key extraction vulnerability
- **Metrics**: Only increment `requestChunkTotal` counter for actual chunk
  requests
- **API Response**: Replaced `syncBuckets` with `bucketCount` in `/ar-io/peers`
  response

## [Release 51] - 2025-09-22

This is a **recommended release** that introduces significant enhancements to
metrics, observer capabilities, gateway filtering, and performance. The release
includes metrics improvements with release tracking, major observer enhancements
with offset observation capabilities, enhanced trusted gateway filtering to
prevent caching issues, and optimized chunk retrieval performance.

### Added

- **Metrics Enhancement**: Added release number as default label to all
  Prometheus metrics, enabling filtering and comparison across releases
- **Enhanced Data Stream Metrics**: Added comprehensive byte tracking with
  `getDataStreamBytesTotal` counter and `getDataStreamSizeHistogram` with 4
  buckets (100KB, 1MB, 10MB, 100MB)
- **Peer Metrics**: Added metrics for preferred peers and peer types, tracking
  "preferred" vs regular "peer" sources and "bucket" vs "general" peer selection
- **Observer Offset Observation**: Added complete V1 implementation of offset
  observation with cryptographic chunk validation using arweave.js validatePath()
  for enhanced data integrity verification. This feature is currently in testing
  phase and will be gradually enabled across the network
- **Observer Gateway Sampling**: Added configurable gateway sampling for offset
  observations with `OFFSET_OBSERVATION_SAMPLE_RATE` (default 5%)
- **Observer Reference Gateway Comparison**: Added chunk availability comparison
  against reference gateway to identify gateway-specific vs network-wide issues
- **Observer Configuration Controls**: Added `OFFSET_OBSERVATION_ENFORCEMENT_ENABLED`
  to control whether offset failures cause gateway assessment failures (default: false)
- **Trusted Gateway Filtering**: Added comprehensive IP and origin filtering for
  trusted gateways with support for CIDR blocks and X-Real-IP header to prevent
  gateway loops and unexpected caching behavior
- **Chunk Offset Sampling Tool**: Added gateway health monitoring tool for chunk
  offset sampling (see `tools/sample-chunk-offsets`)
- **Storage Partition Converter**: Added Arweave storage partition to height
  range converter script (see `tools/arweave-partitions-to-heights`)

### Changed

- **Request Type Labels**: Simplified metric request_type labels to 'full'
  (complete data) and 'range' (partial data) for consistency
- **Peer Management**: Refactored peer management architecture with extracted
  ArweavePeerManager from ArweaveCompositeClient
- **Cache Management**: Improved cache handling with proper timer cleanup in
  NodeCache
- **Chunk Retrieval Optimization**: Optimized chunk retrieval to use single
  peer selection per request, reducing overhead
- **Offset-Aware Architecture**: Implemented TxOffsetSource architecture for
  more efficient chunk retrieval with sync bucket support

### Fixed

- **Root Transaction Detection**:
  - Enhanced logic to prevent incorrect root detection for self-referencing
    transactions
  - Added early exit for self-referencing root transactions
- **Data Cache**: Fixed data cache to respect SKIP_DATA_CACHE setting and skip
  writes when disabled

## [Release 50] - 2025-09-15

This is a **recommended release** due to cache safety improvements that prevent
caching incomplete data and enhance data validation reliability.

This release introduces significant robustness improvements with offset-aware
data sources, experimental datasets HTTP endpoint for analytics workloads, and
enhanced Parquet/Iceberg tooling. It also includes important fixes for data
validation and root parent traversal.

### Added

- **Offset-Aware Data Sources**: Added two new offset-aware data sources that
  leverage cached upstream offset attributes for improved performance:
  - `chunks-offset-aware` (renamed from `chunks-data-item` with backwards
    compatibility) - enables automatic data item resolution within ANS-104
    bundles using cached offsets
  - `trusted-gateways-offset-aware` - uses cached upstream offsets without
    expensive searching for faster data retrieval
- **Cache Skip Configuration**: Added `SKIP_DATA_CACHE` environment variable to
  bypass cache retrieval and always fetch from upstream sources for testing and
  debugging
- **Datasets HTTP Endpoint (Experimental)**: Added optional `/local/datasets`
  endpoint (disabled by default) for HTTP access to Parquet files and Iceberg
  metadata, enabling remote DuckDB queries. Note: This feature is experimental
  and subject to change
- **Datasets Proxy Configuration**: Added configurable datasets proxy via Envoy
  with `DATASETS_PROXY_HOST` and `DATASETS_PROXY_PORT` environment variables
- **Parquet Repartitioning Tool**: Added comprehensive `parquet-repartition`
  script supporting both tag-based and owner address-based partitioning with
  height chunking and Iceberg metadata generation
- **Minimal Iceberg Metadata Generator**: Added lightweight
  `generate-minimal-iceberg-metadata` script optimized for DuckDB compatibility
  with HTTP URL support
- **Multi-Architecture Support**: Added multi-arch support to ClickHouse
  auto-import Docker image for broader platform compatibility

### Changed

- **Default Retrieval Order**: Updated default `ON_DEMAND_RETRIEVAL_ORDER` to
  use new `chunks-offset-aware` name (backwards compatible with
  `chunks-data-item`)
- **Iceberg Metadata Implementation**: Replaced complex PyIceberg-based
  implementation with minimal fastavro-based version for better performance and
  DuckDB compatibility
- **Zero-Size Data Handling**: Skip caching and indexing for zero-size data to
  prevent unnecessary storage operations

### Fixed

- **Root Parent Traversal**: Fixed RootParentDataSource to properly handle root
  transactions without cached attributes
- **Data Size Validation**: Added validation to prevent caching incomplete data
  and prevent ID to hash mapping queue on partial stream errors
- **Parquet Export Issues**: Fixed CSV column type specification to prevent
  DuckDB type inference errors
- **ClickHouse Build Workflow**: Updated build workflow to include missing file
  paths

## [Release 49] - 2025-09-07

This is an optional release that significantly improves the ClickHouse ETL pipeline with better performance, reliability, and Apache Iceberg metadata support. While optional for most users, this release is important for anyone experimenting with Parquet exports and ClickHouse integration.

### Added

- **Apache Iceberg Metadata Generation**: Added `generate-iceberg-metadata`
  script to create Apache Iceberg table metadata for exported Parquet datasets,
  enabling compatibility with query engines like DuckDB and Spark. Controlled by
  new `ENABLE_ICEBERG_GENERATION` environment variable (default: false).
  **Note: Iceberg metadata generation is still under active development and
  currently incomplete.**

- **HyperBEAM Sidecar Support**: Added optional HyperBEAM container
  configuration with `.env.hb.example` template for running AO processes
  alongside the gateway.

- **ETL Configuration Documentation**: Documented existing ClickHouse
  auto-import environment variables in `.env.example`:
  - `CLICKHOUSE_AUTO_IMPORT_SLEEP_INTERVAL` - interval between import cycles
    (default: 3600 seconds)
  - `CLICKHOUSE_AUTO_IMPORT_HEIGHT_INTERVAL` - batch size in blocks (default:
    10000)
  - `CLICKHOUSE_AUTO_IMPORT_MAX_ROWS_PER_FILE` - Parquet file size limit
    (default: 1000000)

### Changed

- **ETL Pipeline Architecture**: Refactored the ClickHouse ETL pipeline for
  improved reliability and modularity:
  - Implemented staging-based workflow to prevent data corruption
  - Changed from API-based triggering to direct script execution
  - Made L1 transaction export the default behavior
  - Changed default export location from `data/parquet` to
    `data/datasets/default`
  - **Performance**: Greatly improved query performance through better index
    usage in the refactored pipeline
  - **Stability**: Fixed issue where the 'core' service would occasionally
    crash due to long-running SQLite queries

## [Release 48] - 2025-09-02

This is an optional release that introduces Turbo root transaction ID lookups,
DNS resolution for preferred chunk nodes, and automatic data item resolution
from chunks. These features continue the network's journey toward complete
independence from legacy infrastructure while improving retrieval performance
and reliability.

### Added

- Added Turbo root transaction ID source with configurable lookup order and
  circuit breakers (configurable via `ROOT_TX_INDEX_` environment variables),
  enabling efficient data item to root transaction resolution via Turbo's
  API with automatic failover handling.
- Added `chunks-data-item` data source option for retrieval orders that enables
  automatic data item resolution within ANS-104 bundles. When used in
  `ON_DEMAND_RETRIEVAL_ORDER` or `BACKGROUND_RETRIEVAL_ORDER`, this source
  transparently resolves data items to their root transactions and calculates
  correct byte ranges, allowing direct retrieval of data items from chunks
  without requiring separate data item indexing.
- Added DNS resolution for preferred chunk GET and POST nodes with automatic
  failover, enabling dynamic resolution of node hostnames to IP addresses with
  configurable refresh intervals.
- Added `X-AR-IO-Node-Release` header to all outbound HTTP requests for better
  network observability and version tracking.

### Changed

- Updated default `ON_DEMAND_RETRIEVAL_ORDER` from
  `trusted-gateways,chunks,tx-data,ar-io-network` to
  `trusted-gateways,ar-io-network,chunks-data-item,tx-data`, removing the
  deprecated `chunks` source and prioritizing AR.IO network retrieval.

## [Release 47] - 2025-08-25

This is an optional release that lays the groundwork for the gateway network to
become completely independent of the legacy arweave.net gateway by enabling
data item retrieval directly from chunks using root transaction ID lookups. It
also introduces comprehensive distributed tracing for improved observability.

### Added

- Added comprehensive OpenTelemetry distributed tracing with proper span linking
  across all data request operations, providing end-to-end visibility from HTTP
  requests through data source failovers, cache operations, and peer retrievals
  with hierarchical parent-child span relationships.
- Added `RootParentDataSource` and `Ans104OffsetSource` to enable retrieving
  data items by ID directly from chunks when only the root transaction ID is
  indexed, transparently resolving data items to their root transactions and
  calculating correct byte ranges within ANS-104 bundles (temporarily disabled
  pending additional offset sources).

### Changed

### Fixed

- Removed incorrect `Content-Digest` header from chunk endpoint which was
  misleadingly representing only chunk data hash instead of complete JSON
  response body hash (will be reintroduced with correct semantics).

## [Release 46] - 2025-08-18

This is a recommended release that introduces AR.IO network chunk retrieval with
cryptographic validation and enhanced observability. Gateway operators can now
retrieve chunks directly from AR.IO peers with the same security guarantees as
Arweave network chunks, significantly improving chunk caching and retrieval
performance.

### Added

- Added AR.IO network chunk source enabling chunk retrieval from AR.IO peers
  with weighted peer selection, retry logic, and cryptographic validation to
  prevent serving of corrupted or malicious data.
- Added comprehensive OpenTelemetry tracing for chunk retrieval operations
  providing visibility into performance, cache behavior, and source attribution
  across the entire pipeline.
- Added HEAD request support to `/chunk/{offset}` endpoint with ETag headers
  for efficient caching and conditional request handling with If-None-Match
  support.
- Added chunk source headers for traceability: `X-AR-IO-Chunk-Source-Type`
  indicating data source, `X-AR-IO-Chunk-Host` with peer hostname, and
  `X-Cache` for cache status.
- Added RFC 9530 `Content-Digest` header support for standard-compliant content
  integrity verification in data and chunk responses.
- Added configurable composite chunk sources with parallelism control via
  `CHUNK_DATA_RETRIEVAL_ORDER` and `CHUNK_METADATA_RETRIEVAL_ORDER` environment
  variables supporting comma-separated source ordering.
- Added OpenAPI documentation for `/ar-io/peers` endpoint.

### Changed

- Renamed `ar-io-peers` to `ar-io-network` as the preferred configuration name
  while maintaining backwards compatibility.
- Enhanced `/ar-io/peers` endpoint to include both data and chunk weights for
  AR.IO gateway peers.

### Fixed

- Fixed ArNS custom 404 pages to prevent incorrect ArNS headers from being
  propagated to other gateways.

## [Release 45] - 2025-08-11

This is an optional release that enhances chunk broadcasting with improved preferred
peer management, adds a hash-based partitioning filter for distributed data processing,
fixes ArNS basename cache refresh issues, and includes comprehensive documentation
improvements with a new glossary of AR.IO Node terminology.

### Added

- Added hash partitioning filter (`MatchHashPartition`) for distributing
  transaction and data item processing across multiple nodes with configurable
  partition ranges.
- Added comprehensive glossary documentation covering AR.IO Node terminology,
  concepts, and architectural components.

### Changed

- Improved chunk broadcasting preferred peer management with doubled default
  per-node queue depth threshold and ensured preferred peers are always
  prioritized first.
- Enhanced circuit breaker metrics with more detailed labels for better
  monitoring of data source failures.
- Improved ArNS resolution to properly propagate 404 errors from trusted
  gateway resolution (a more complete fix is coming in the next release).
- Expanded OTEL tracing to include ArNS cache operations for improved
  observability of name resolution and cache hydration.

### Fixed

- Fixed unreliable ArNS basename cache refreshes by adding retry logic for
  pagination failures and replacing p-debounce with timestamp-based debouncing
  for more predictable behavior.
- Fixed undefined headers handling in data requests.
- Fixed invalid cache hits by ensuring base64url encoded IDs are properly
  validated before use.
- Fixed routes data handling for undefined IDs in validity checks.

## [Release 44] - 2025-07-28

This is a recommended release that introduces efficient range request support
for contiguous data retrieval from chunks, adds bundle metadata columns with
offset indexing to improve offset availability throughout the network, enhances
Merkle path parsing compatibility, and includes comprehensive documentation for
offsets and Merkle paths.

### Added

- Added efficient range request support for chunk data retrieval, enabling
  optimized verifiable contiguous data fetching directly from Arweave nodes.
- Added bundle metadata columns to `data.db` to improve offset availability
  across the gateway network.
- Added OTEL (OpenTelemetry) tracing support for chunk POST operations,
  providing better observability for chunk broadcasting performance.
- Added OTEL environment variables to `docker-compose.yaml` for easier
  configuration of distributed tracing.
- Added comprehensive Arweave Merkle tree structure documentation detailing
  the data organization and validation rules.
- Added detailed documentation explaining Arweave transaction and chunk offset
  calculations.
- Added merkle-path-parser with full Arweave compatibility for improved
  Merkle proof validation.

### Changed

- Implemented promise-based chunk caching system replacing the previous WeakMap
  implementation, improving memory efficiency and cache reliability.
- Extended CompositeChunkSource to implement all chunk interfaces, providing
  a more unified chunk data access layer.

## [Release 43] - 2025-07-21

This is a recommended release that enables data verification by default for data
items linked to ArNS names, improves chunk broadcasting efficiency, and adds
automatic chunk data cache cleanup.

### Added

- Added automatic chunk data cache cleanup functionality with configurable
  retention period. Chunks are now automatically removed after 4 hours by default
  (configurable via `CHUNK_DATA_CACHE_CLEANUP_THRESHOLD`). The cleanup can be
  disabled by setting `ENABLE_CHUNK_DATA_CACHE_CLEANUP=false`. This helps manage
  disk space usage while maintaining cache performance benefits.
- Added demand-driven opt-out background verification for ArNS data. When ArNS
  names are requested, the system now proactively verifies the underlying data
  asynchronously in the background by unbundling verified chunk data retrieved
  directly from Arweave nodes. This ensures ArNS-served content is prioritized
  for verification, improving data integrity guarantees for frequently accessed
  named content.

### Changed

- Simplified chunk data storage by removing the dual-storage approach (by-hash
  and by-dataroot with symlinks). Chunks are now stored directly by data root
  only, reducing complexity and improving performance.
- Revamped chunk broadcasting architecture from 3-tier system to unified
  peer-based approach. Chunk broadcasting now uses individual fastq queues per
  peer with configurable concurrency and queue depth protection. Added support
  for preferred chunk POST peers via `PREFERRED_CHUNK_POST_URLS` environment
  variable. Configuration defaults have been optimized:
  `CHUNK_POST_PEER_CONCURRENCY` now defaults to match
  `CHUNK_POST_MIN_SUCCESS_COUNT` (3) to avoid over-broadcasting, and
  `CHUNK_POST_PER_NODE_CONCURRENCY` defaults to match
  `CHUNK_POST_QUEUE_DEPTH_THRESHOLD` (10) for consistent per-node load
  management. This change improves broadcast reliability and performance while
  simplifying the codebase by removing circuit breakers and tier-based logic.
- Modified `DataVerificationWorker` to ensure data item IDs (not just root IDs)
  have their retry count incremented, preventing IDs from being stuck without
  retry attempts. This improves the reliability of the data verification
  process.

### Fixed

- Fixed experiment bash Parquet export script generating filenames with
  `count_star()` instead of actual row counts for blocks and tags files. The
  script now correctly uses the `-noheader` flag when retrieving counts for
  filename generation.
- Fixed missing directory existence checks in FsCleanupWorker to prevent errors
  when attempting to scan non-existent directories during filesystem cleanup
  operations.

## [Release 42] - 2025-07-14

This is an optional release that improves peer request traceability, adds
HyperBEAM URL support, and includes draft AI-generated technical documentation.

### Added

- Added support for optional HyperBEAM URL configuration via
  `AO_ANT_HYPERBEAM_URL` environment variable. In the future this allows ANT
  processes to use HyperBEAM nodes for caching and serving state, reducing
  pressure on compute units for simple read requests.
- Added AI-generated technical documentation covering AR.IO gateway
  architecture, data retrieval, Arweave connectivity, ArNS name resolution
  system, centralization analysis, and database architecture. These guides in
  `docs/drafts/` are generally correct but should not be considered
  authoritative.
- Added origin and release information to query string parameters in outbound
  requests to both peer gateways and trusted gateways. Data requests now
  include `ar-io-hops`, `ar-io-origin`, `ar-io-origin-release`,
  `ar-io-arns-record`, and `ar-io-arns-basename` as query parameters,
  improving network observability and request tracing across the entire
  gateway network.

### Changed

- Implemented X-AR-IO header initialization for outbound peer requests while
  removing `x-ar-io-origin` and `x-ar-io-origin-node-release` headers from
  responses. This change maintains necessary header functionality for peer
  communication while reducing unnecessary header overhead in responses.
- Updated `@ar.io/sdk` dependency to support optional HyperBEAM URL
  functionality.

## [Release 41] - 2025-06-30

Upgrading to this release is recommended but not urgent due to improvements in
peer data fetching safety.

### Added

- Added preferred chunk GET node URLs configuration via
  `PREFERRED_CHUNK_GET_NODE_URLS` environment variable to enable chunk-specific
  peer prioritization. Preferred URLs receive a weight of 100 for
  prioritization and the system selects 10 peers per attempt by default.
- Added hash validation for peer data fetching by including
  `X-AR-IO-Expected-Digest` header in peer requests when hash is available,
  validating peer responses against expected hash, and immediately rejecting
  mismatched data.
- Added `DOCKER_NETWORK_NAME` environment variable to configure the Docker
  network name used by Docker Compose.
- Added draft guide for running a community gateway.
- Added draft data verification architecture document.

### Changed

- Removed trusted node fallback for chunk retrieval. Chunks are now retrieved
  exclusively from peers, with the retry count increased from 3 to 50 to ensure
  reliability without the trusted node fallback.

### Fixed

- Fixed inverted logic preventing symlink creation in `FsChunkDataStore`.
- Fixed `Content-Length` header for range requests and 304 responses, properly
  setting header for single and multipart range requests and removing entity
  headers from 304 Not Modified responses per RFC 7232.
- Fixed `MaxListenersExceeded` warnings by adding `setMaxListeners` to
  read-through data cache.
- Fixed potential memory leaks in read-through data cache by using `once`
  instead of `on` for `error` and `end` event listeners.

## [Release 40] - 2025-06-23

This is an optional release that primarily improves caching when data is
fetched from peers.

### Added

- Added experimental `flush-to-stable` script for manual database maintenance.
  This script allows operators to manually flush stable chain and data item
  tables, mirroring the logic of
  `StandaloneSqliteDatabase.flushStableDataItems`.
  **WARNING: This script is experimental and directly modifies database contents.
  Use with caution and ensure proper backups before running.**

### Changed

- Replaced yesql with custom SQL loader that handles comments better, improving
  SQL file parsing and maintenance.
- Switched to SPDX license headers to reduce LLM token usage, making the
  codebase more efficient for AI-assisted development.
- Improved untrusted data handling and hash validation in cache operations. The
  cache now allows caching when a hash is available for validation even for
  untrusted data sources, but only finalizes the cache when the computed hash
  matches a known trusted hash. This prevents cache poisoning while still
  allowing data caching from untrusted sources when the data can be validated.

## [Release 39] - 2025-06-17

This release enhances observability and reliability with new cache metrics,
improved data verification capabilities, and automatic failover between chain
data sources. The addition of ArNS-aware headers enables better data
prioritization across the gateway network. This is a recommended but not urgent
upgrade.

### Added

- Added filesystem cache metrics with cycle-based tracking. Two new Prometheus
  metrics track cache utilization: `cache_objects_total` (number of objects in
  cache) and `cache_size_bytes` (total cache size in bytes). Both metrics include
  `store_type` and `data_type` labels to differentiate between cache types (e.g.,
  headers, contiguous_data). Metrics are updated after each complete cache scan
  cycle, providing accurate visibility into filesystem cache usage.
- Added `X-AR-IO-Data-Id` header to all data responses. This header shows the
  actual data ID being served, whether from a direct ID request or manifest path
  resolution, providing transparency about the content being delivered.
- Added automatic data item indexing when data verification is enabled. When
  `ENABLE_BACKGROUND_DATA_VERIFICATION` is set to true, the system now
  automatically enables data item indexing (`ANS104_UNBUNDLE_FILTER`) with an
  `always: true` filter if no filter is explicitly configured. This ensures
  bundles are unbundled to verify that data items are actually contained in
  the bundle associated with the Arweave transaction's data root.
- Added ArNS headers to outbound gateway requests to enable data prioritization.
  The `generateRequestAttributes` function now includes ArNS context headers
  (`X-ArNS-Name`, `X-ArNS-Basename`, `X-ArNS-Record`) in requests to other
  gateways and Arweave nodes, allowing downstream gateways to effectively
  prioritize ArNS data requests.
- Added configurable Docker Compose host port environment variables
  (`CORE_PORT`, `ENVOY_PORT`, `CLICKHOUSE_PORT`, `CLICKHOUSE_PORT_2`,
  `CLICKHOUSE_PORT_3`, `OBSERVER_PORT`) to allow flexible port mapping while
  maintaining container-internal port compatibility and security.
- Added Envoy aggregate cluster configuration for automatic failover between
  primary and fallback chain data sources. The primary cluster (default:
  arweave.net:443) uses passive outlier detection while the fallback cluster
  (default: peers.arweave.xyz:1984) uses active health checks. This enables
  zero-downtime failover between HTTPS and HTTP endpoints with configurable
  `FALLBACK_NODE_HOST` and `FALLBACK_NODE_PORT` environment variables.

### Changed

- Streamlined background data retrieval to reduce reliance on centralized sources.
  The default `BACKGROUND_RETRIEVAL_ORDER` now only includes `chunks,s3`, removing
  `trusted-gateways` and `tx-data` from the default configuration. This prioritizes
  verifiable chunk data and S3 storage for background operations like unbundling.
- Removed ar-io.net from default trusted gateways list and removed
  TRUSTED_GATEWAY_URL default value to reduce load on ar-io.net now that P2P data
  retrieval is re-enabled. Existing deployments with TRUSTED_GATEWAY_URL
  explicitly set will continue to work for backwards compatibility.

## [Release 38] - 2025-06-09

This release focuses on data integrity and security improvements, introducing
trusted data verification and enhanced header information for data requests.
Upgrading to this release is recommended but not urgent.

### Added

- Added `X-AR-IO-Trusted` header to indicate data source trustworthiness in
  responses. This header helps clients understand whether data comes from a
  trusted source and works alongside the existing `X-AR-IO-Verified` header to
  provide data integrity information. The system now filters peer data by
  requiring peers to indicate their content is either verified or trusted,
  protecting against misconfigured peers that may inadvertently serve
  unintended content (e.g., provider default landing pages) instead of actual
  Arweave data.
- Added If-None-Match header support for HTTP conditional requests enabling
  better client-side caching efficiency. When clients send an If-None-Match
  header that matches the ETag, the gateway returns a 304 Not Modified response
  with an empty body, reducing bandwidth usage and improving performance.
- Added digest and hash headers for data HEAD requests to enable client-side
  data integrity verification.
- Added EC2 IMDS (instance-profile) credential support for S3 data access,
  improving AWS authentication in cloud environments.
- Added trusted data flag to prevent caching of data from untrusted sources,
  ensuring only verified and reliable content is stored locally while still
  allowing serving of untrusted data when necessary.

### Changed

- Re-enabled ar-io-peers as fallback data source in configuration for improved
  data availability.
- Updated trusted node configuration to use arweave.net as the default trusted
  node URL.
- Updated ETag header format to use properly quoted strings (e.g., `"hash"`
  instead of `hash`) following HTTP/1.1 specification standards for improved
  compatibility with caching proxies and clients.

## [Release 37] - 2025-06-03

This is a _recommended release_ due to the included observer robustness
improvements. It also adds an important new feature - data verification for
preferred ArNS names. When preferred ArNS names are set, the bundles containing
the data they point to will be locally unbundled (verifying data item
signatures), and the data root for the bundle will be compared to the data root
in the Arweave chain (establishing that the data is on Arweave). To enable this
feature, set your preferred ArNS names, turn on unbundling by setting
`ANS104_DOWNLOAD_WORKERS` and `ANS104_UNBUNDLE_WORKERS` both to 1, and set your
`ANS104_INDEX_FILTER` to a filter that will match the data items for your
preferred names. If you don't know the filter, use `{"always": true}`, but be
aware this will index the entire bundle for the IDs related to your preferred
names.

Note: this release contains migrations to `data.db`. If your node appears
unresponsive please check `core` service logs to determine whether migrations
are running and wait for them to finish.

### Added

- Added prioritized data verification system for preferred ArNS names,
  focusing computational resources on high-priority content while enabling
  flexible root transaction discovery through GraphQL fallback support.
- Added verification retry prioritization system with tracking of retry counts,
  priority levels, and attempt timestamps to ensure bundles do not get stuck
  retrying forever.
- Added improved observer functionality with best-of-2 observations and higher
  compression for more reliable network monitoring.
- Added `MAX_VERIFICATION_RETRIES` environment variable (default: 5) to limit
  verification retry attempts and prevent infinite loops for consistently
  failing data items.
- Added retry logic with exponential backoff for GraphQL queries to handle rate
  limiting (429) and server errors with improved resilience when querying
  trusted gateways for root bundle IDs.

### Changed

- Updated dependencies: replaced deprecated express-prometheus-middleware with
  the actively maintained express-prom-bundle library and updated prom-client
  to v15.1.3 for better compatibility and security.
- Updated Linux setup documentation to use modern package installation methods,
  replacing apt-key yarn installation with npm global install and updating
  Node.js/nvm versions.
- Improved route metrics normalization with explicit whitelist function for
  better granularity and proper handling of dynamic segments.

### Fixed

- Fixed docker-compose configuration to use correct NODE_MAX_OLD_SPACE_SIZE
  environment variable name.
- Fixed production TypeScript build configuration to exclude correct "test"
  directory path.
- Fixed Parquet exporter to properly handle data item block_transaction_index
  exports, preventing NULL value issues.
- Fixed bundles system to copy root_parent_offset when flushing data items to
  maintain data integrity.
- Fixed ClickHouse auto-import script to handle Parquet export not_started
  status properly.
- Fixed docker-compose ClickHouse configuration to not pass conflicting
  PARQUET_PATH environment variable to container scripts.
- Fixed verification process for data items that have not been unbundled by
  adding queue bundle support and removing bundle join constraint to ensure
  proper verification of data items without indexed root parents.

## [Release 36] - 2025-05-27

This is a recommended but not essential upgrade. The most important changes are
the preferred ArNS caching feature for improved performance on frequently
accessed content and the observer's 80% failure threshold to prevent invalid
reports during network issues.

### Added

- Added preferred ArNS caching functionality that allows configuring lists of
  ArNS names to be cached longer via `PREFERRED_ARNS_NAMES` and
  `PREFERRED_ARNS_BASE_NAMES` environment variables. When configured, these names
  will be cleaned from the filesystem cache after
  `PREFERRED_ARNS_CONTIGUOUS_DATA_CACHE_CLEANUP_THRESHOLD` instead of the
  standard cleanup threshold (`CONTIGUOUS_DATA_CACHE_CLEANUP_THRESHOLD`). This
  is accomplished by maintaining an MRU (Most Recently Used) list of ArNS names
  in the contiguous metadata cache. When filesystem cleanup runs, it checks
  this list to determine which cleanup threshold to apply. This feature enables
  gateway operators to ensure popular or important ArNS names remain cached
  longer, improving performance for frequently accessed content.
- Added ArNS headers to responses: `X-ArNS-Name`, `X-ArNS-Basename`, and
  `X-ArNS-Record` to help identify which ArNS names were used in the resolution.

### Changed

- Updated observer to prevent report submission when failure rate exceeds 80%.
  This threshold helps guard against both poorly operated observers and
  widespread network issues. In the case of a widespread network issue, the
  assumption is that most gateway operators are well intentioned and will work
  together to troubleshoot and restore both observations and network stability,
  rather than submitting reports that would penalize functioning gateways.
- Updated default trusted gateway in docker-compose Envoy configuration to
  ar-io.net for improved robustness and alignment with core service
  configuration.
- Improved range request performance by passing ranges directly to getData
  implementations rather than streaming all data and extracting ranges.

### Fixed

- Fixed missing cache headers (`X-Cache` and other data headers) in range
  request responses to ensure consistent cache header behavior across all request
  types.
- Fixed async streaming for multipart range requests by using async iteration
  instead of synchronous reads, preventing potential data loss.
- Fixed ArNS resolution to properly exclude www subdomain from resolution
  logic.
- Fixed test reliability issues by properly awaiting stream completion before
  making assertions.
- Fixed chunk broadcasting to not await peer broadcasts, as they are
  best-effort operations.

## [Release 35] - 2025-05-19

This is a low upgrade priority release. It contains a small caching improvement
and routing fix. Upgrading to help test it is appreciated but not essential.

### Changed

- Adjusted filesystem data expiration to be based on last request times rather
  than file access times which may be inaccurate.
- Adjusted CORS headers to include `content-*` headers.

### Fixed

- Fixed regex used to expose `/api-docs` when an apex ArNS name is set.

## [Release 34] - 2025-05-05

Given the resilience provided by adding a second trusted gateway URL, it is
recommended that everyone upgrade to this release.

### Added

- Added peer list endpoints for retrieving information about Arweave peers and
  ar.io gateway peers.
- Added ar-io.net as a secondary trusted gateway to increase data retrieval
  resilience by eliminating a single point of failure.
- Added circuit breaker for Arweave peer chunk posting.

### Changed

- Created directories for DuckDB and Parquet to help avoid permission issues
  by the directories being created by containers.

### Fixed

- Fixed GraphQL ClickHouse error when returning block ID and timestamp.
- Fixed the tx-chunks-data-source to throw a proper error (resulting in a 404)
  when the first chunk is missing rather than streaming a partial response.

## [Release 33] - 2025-05-05

### Added

- Added a [Parquet and ClickHouse usage guide]. Using ArDrive as an example, it
  provides step by step instructions about how to bulk load Parquet and
  configure continuous ingest of bundled data items into ClickHouse. This
  allows the ar-io-node to support performant GraphQL queries on larger data
  sets and fascilitates sharing indexing work across gateways via distribution
  of Parquet files.
- Added support for configurable ArNS 404 pages using either:
  - `ARNS_NOT_FOUND_TX_ID`: Transaction ID for custom 404 content
  - `ARNS_NOT_FOUND_ARNS_NAME`: ArNS name to resolve for 404 content
- Added experimental `/chunk/<offset>` GET route for serving chunk data by
  absolute offset either the local cache.
- Added support for `AWS_SESSION_TOKEN` in the S3 client configuration.
- Expanded ArNS OTEL tracing to improve resolution behavior observability.
- Added support for setting a ClickHouse username and password via the
  `CLICKHOUSE_USERNAME` and `CLICKHOUSE_PASSWORD` environment variable. When
  using ClickHouse, `CLICKHOUSE_PASSWORD` should always be set. However,
  `CLICKHOUSE_USERNAME` can be left unset. The username `default` will be used
  in that case.
- Added support for configuring the port used to connect to ClickHouse via
  the `CLICKHOUSE_PORT` environment variable.

### Changed

- Disabled ClickHouse import timing logging by default. It can be enabled via
  environment variable - `DEBUG` when running the service standalone or
  `CLICKHOUSE_DEBUG` when using Docker Compose
- Upgraded to ClickHouse 25.4.

### Fixed

- Ensure `.env` is read in `clickhouse-import` script.

## [Release 32] - 2025-04-22

### Changed

- Reenabled parallel ArNS resolution with removal of misplaced global limit.
  Refer to release 30 notes for more details on configuration and rationale.
- Added a timeout for the last ArNS resolver in `ARNS_RESOLVER_PRIORITY_ORDER`.
  It defaults to 30 seconds and is configurable using
  `ARNS_COMPOSITE_LAST_RESOLVER_TIMEOUT_MS`. This helps prevent promise build
  up if the last resolver stalls.

### Fixed

- Fixed apex ArNS name handling when a subdomain is present in
  `ARNS_ROOT_HOST`.
- Fixed a case where fork recovery could stall due to early flushing of
  unstable chain data.
- Restored observer logs by removing unintentional default log level override
  in `docker-compose.yaml`.

## [Release 31] - 2025-04-11

### Changed

- Improved peer TX header fetching by fetching from a wider range of peers and
  up/down weighting peers based on success/failure.

### Fixed

- Rolled back parallel ArNS resolution changes that were causing ArNS
  resolution to slow down over time.

## [Release 30] - 2025-04-04

### Added

- Added support for filtering Winston logs with a new `LOG_FILTER` environment
  variable.
  - Example filter: `{"attributes":{"class":"ArweaveCompositeClient"}}` to only
    show logs from that class.
  - Use `CORE_LOG_FILTER` environment variable when running with
    docker-compose.
- Added parallel ArNS resolution capability.
  - Configured via `ARNS_MAX_CONCURRENT_RESOLUTIONS` (default: 1).
  - This foundation enables future enhancements to ArNS resolution and should
    generally not be adjusted at present.

### Changed

- Improved ClickHouse auto-import script with better error handling and
  continuous operation through errors.
- Reduced maximum header request rate per second to trusted node to load on
  community gateways.
- Optimized single owner and recipient queries on ClickHouse with specialized
  sorted tables.
- Used ID sorted ClickHouse table for ID queries to improve performance.

### Fixed

- Fixed data alignment in Parquet file name height boundaries to ensure
  consistent import boundaries.
- Removed trailing slashes from AO URLs to prevent issues when passing them to
  the SDK.
- Only prune SQLite data when ClickHouse import succeeds to prevent data loss
  during exports.

## [Release 29] - 2025-03-21

### Changed

- Temporarily default to trusted gateway ArNS resolution to reduce CU load as
  much possible. On-demand CU resolution is still available as a fallback and
  the order can be modified by setting `ARNS_RESOLVER_PRIORITY_ORDER`.
- Remove duplicate network process call in on-demand resolver.
- Don't wait for network process debounces in the on-demand resolver.
- Slow network process dry runs no longer block fallback to next resolver.

### Added

- Added support for separate CUs URLs for the network and ANT processes via the
  `NETWORK_AO_CU_URL` and `ANT_AO_CU_URL` process URLs respectively. If either
  is missing the `AO_CU_URL` is used instead with a fallback to the SDK default
  URL if `AO_CU_URL` is also unspecified.
- Added CU URLs to on-demand ArNS resolver logs.
- Added circuit breakers for AR.IO network process CU dry runs. By default
  they use a 1 minute timeout and open after 30% failure over a 10 minute
  window and reset after 20 minutes.

### Fixed

- Owners in GraphQL results are now correctly retrieved from data based on
  offsets when using ClickHouse.

## [Release 28] - 2025-03-17

### Changed

- Raised name not found name list refresh interval to 2 minutes to reduce load
  on CUs. This increases the maximum amount of time a user may wait for a new
  name to be available. Future releases will introduce other changes to
  mitigate this delay.
- Adjusted composite ArNS resolver to never timeout resolutions from the last
  ArNS resolver in the resolution list.

### Added

- Added support for serving a given ID or ArNS name from the apex domain of a
  gateway. If using an ID, set the `APEX_TX_ID` environment variable. If using
  an ArNS name, set the `APEX_ARNS_NAME` environment variable.
- Added `BUNDLE_REPAIR_UPDATE_TIMESTAMPS_INTERVAL_SECONDS`,
  `BUNDLE_REPAIR_BACKFILL_INTERVAL_SECONDS`, and
  `BUNDLE_REPAIR_FILTER_REPROCESS_INTERVAL_SECONDS` environment variables to
  control the interval for retrying failed bundles, backfilling bundle records,
  and reprocessing bundles after a filter change. Note: the latter two are
  rairly used. Queuing bundles for reprocessing via the
  `/ar-io/admin/queue-bundle` endpoint is usually preferrable to automatic
  reprocessing as it is faster and offers more control over the reprocessing
  behavior.

### Fixed

- Signatures in GraphQL results are now correctly retrieved from data based on
  offsets when using ClickHouse.
- Adjusted exported Parquet file names to align with expectations of ClickHouse
  import script.
- Ensured that bundle indexing status is properly reset when bundles are
  manually queued after an unbundling filture change has been made.

## [Release 27] - 2025-02-20

### Changed

- Set process IDs for mainnet.
- Increase default AO CU WASM memory limit to 17179869184 to support mainnet
  process.

## [Release 26] - 2025-02-13

### Added

- Added a per resolver timeout in the composite ArNS resolver. When the
  composite resolver attempts resolution it is applied to each resolution
  attempt. It is configurable via the `ARNS_COMPOSITE_RESOLVER_TIMEOUT_MS` and
  defaults to 3 seconds in order to allow a fallback attempt before the default
  observer timeout of 5 seconds.
- Added a `TURBO_UPLOAD_SERVICE_URL` environment variable to support
  configuration of the bundler used by the observer (TurboSDK defaults are
  used if not set).
- Added a `REPORT_DATA_SINK` environment variable that enables switching the
  method used to post observer reports. With the default, `turbo`, it sends
  data items via a Turbo compatible bundler. Switching it to `arweave` will
  post base layer transactions directly to Arweave instead.
- Added a `/ar-io/admin/bundle-status/<id>` endpoint that returns the counters
  and timestamps from the `bundles` row in `data.db`. This can be used for
  monitoring unbundling progress and scripting (e.g., to skip requeuing already
  queued bundles).
- Added more complete [documentation](docs/filters.md) for filters.

### Changed

- Use arweave.net as the default GraphQL URL for AO CUs since most gateways
  will not have a complete local AO data item index.
- Use a default timeout of 5 seconds when refreshing Arweave peers to prevent
  stalled peer refreshes.
- Cache selected gateway peer weights for the amount of time specified by the
  `GATEWAY_PEERS_WEIGHTS_CACHE_DURATION_MS` environment variable with a default
  of 5 seconds to avoid expensive peer weight recomputation on each request.
- Chunk broadcasts to primary nodes occur in parallel with a concurrency limit
  defaulting to 2 and configurable via the `CHUNK_POST_CONCURRENCY_LIMIT`
  environment variable.
- Added circuit breakers for primary chunk node POSTs to avoid overwhelming
  chunk nodes when they are slow to respond.

### Fixed

- Properly cleanup timeout and event listener when terminating the data
  root computation worker.
- Count chunk broadcast exceptions as errors in the
  `arweave_chunk_broadcast_total` metric.

## [Release 25] - 2025-02-07

### Added

- Added support for indexing and querying ECDSA signed Arweave transactions.
- Expanded the OpenAPI specification to cover the entire gateway API and
  commonly used Arweave node routes.
- ArNS undername record count limits are now enforced. Undernames are sorted
  based on their ANT configured priority with a fallback to name comparisons
  when priorities conflict or are left unspecified. Enforcement is enabled by
  default but can be disabled by setting the
  `ARNS_RESOLVER_ENFORCE_UNDERNAME_LIMIT` to `false`.

### Changed

- Renamed the `ario-peer` data source to `ar-io-peers` for consistency and
  clarity. `ario-peer` will continue to work for backwards compatibility but is
  considered deprecated.
- Use AR.IO gateway peers from the ar.io gateway address registry (GAR) as the
  last fallback for fetching data when responding to client data requests. This
  has the benefit of making the network more resilient to trusted gateway
  disruptions, but it can also result in nodes serving data from less trusted
  sources if it is not found in the trusted gateway. This can be disabled by
  using a custom `ON_DEMAND_RETRIEVAL_ORDER` that does not include
  `ar-io-peers`.
- Arweave data chunk requests are sent to the trusted node first with a
  fallback to Arweave peers when chunks are unavailable on the trusted node.
  This provides good performance by default with a fallback in case there are
  issues retrieving chunks from the trusted node.
- Increased the observer socket timeout to 5 seconds to accommodate initial
  slow responses for uncached ArNS resolutions.
- Disabled writing base layer Arweave signatures to the SQLite DB by default to
  save disk space. When signatures are required to satisfy GraphQL requests,
  they are retrieved from headers on the trusted node.

### Fixed

- Updated dependencies to address security issues.
- Improved reliability of failed bundle indexing retries.
- Fixed failure to compute data roots for verification for base layer data
  larger than 2GiB.
- Fixed observer healthcheck by correcting node.js path in healthcheck script.

## [Release 24] - 2025-02-03

### Added

- Added a `ARNS_ANT_STATE_CACHE_HIT_REFRESH_WINDOW_SECONDS` environment
  variable that determines the number of seconds before the end of the TTL at
  which to start attempting to refresh the ANT state.
- Added a `TRUSTED_GATEWAYS_REQUEST_TIMEOUT_MS` environment that defaults to
  10,000 and sets the number of milliseconds to wait before timing out request
  to trusted gateways.
- Added `BUNDLE_REPAIR_RETRY_INTERVAL_SECONDS` and
  `BUNDLE_REPAIR_RETRY_BATCH_SIZE` environment variables to control the time
  between queuing batches of bundle retries and the number of data items
  retrieved when constructing batches of bundles to retry.
- Added support for configuring the ar.io SDK log level via the
  `AR_IO_SDK_LOG_LEVEL` environment variable.
- Added a `request_chunk_total` Prometheus counter with `status`, `source` (a
  URL) and `source_type` (`trusted` or `peer`) labels to track success/failure
  of chunk retrieval in the Arweave network per source.
- Added a `get_chunk_total` Prometheus metric to count chunk retrieval
  success/failure per chunk.
- Added `arns_cache_hit_total` and `arns_cache_miss_total` Prometheus counters
  to track ArNS cache hits and misses for individual names respectively.
- Added `arns_name_cache_hit_total` and `arns_name_cache_miss_total` Prometheus
  counters to track ArNS name list cache hits and misses
  respectively.
- Added a `arns_resolution_duration_ms` Prometheus metric that tracks summary
  statistics for the amount of time it takes to resolve ArNS names.

### Changed

- In addition to the trusted node, the Arweave network is now searched for
  chunks by default. All chunks retrieved are verified against data roots
  indexed from a trusted Arweave node to ensure their validity.
- Default to a 24 hour cache TTL for the ArNS name cache. Record TTLs still
  override this, but in cases where resolution via AO CU is slow or fails, the
  cache will be used. In the case of slow resolution, CU based resolution will
  proceed in the background and update the cache upon completion.
- Switched to the `ioredis` library for better TLS support.
- Updated minor dependency minor versions (more dependencies will be updated in
  the next release).
- Bundles imports will no longer be re-attempted for bundles that have already
  been fully unbundled using the current filters if they are matched or
  manually queued again.
- Replaced references `docker-compose` in the docs with the more modern `docker
compose`.

### Fixed

- Ensure duplicate data item IDs are ignored when comparing counts to determine
  if a bundle has been fully unbundled.
- Fixed worker threads failing to shut down properly when the main process
  stopped.
- Ensure bundle import attempt counts are incremented when bundles are skipped
  to avoid repeatedly attempting to import skipped bundles.
- Use observe that correctly ensure failing gateways are penalized in the AR.IO
  AO process.

## [Release 23] - 2025-01-13

### Added

- Added `FS_CLEANUP_WORKER_BATCH_SIZE`,
  `FS_CLEANUP_WORKER_BATCH_PAUSE_DURATION`, and
  `FS_CLEANUP_WORKER_RESTART_PAUSE_DURATION` environment variables to allow
  configuration of number of contiguous data files cleaned up per batch, the
  pause between each batch, and the pause before restarting the entire cleanup
  process again.
- Added `data_items_unbundled_total` Prometheus metric that counts the total
  number of data items unbundled, including those that did not match the
  unbundling filter.
- Added a `parent_type` label that can be one of `transaction` or `data_item`
  to data item indexing metrics.
- Added a `files_cleaned_total` total Prometheus metric to enable monitoring of
  contiguous data cleanup.
- Added support for specifying the admin API via a file specified by the
  `ADMIN_API_KEY_FILE` environment variable.
- Added experimental support for posting chunks in a non-blocking way to
  secondary nodes specified via a comma separate list in the
  `SECONDARY_CHUNK_POST_URLS` environment variable.

### Changed

- Renamed the `parent_type` lable to `contiguous_data_type` on bundle metrics
  to more accurately reflect the meaning of the label.
- Reduced the maximum time to refresh the ArNS name list to 10 seconds to
  minimize delays in ArNS availability after a new name is registered.
- Changed `/ar-io/admin/queue-bundle` to wait for `bundles` rows to be written
  to the DB before responding to ensure that errors that occur due to DB
  contention are not silently ignored.
- Data items are now flushed even when block indexing is stopped. This allows
  for indexing batches of data items using the admin API with block indexing
  disabled.
- Adjust services in `docker-compose` to use `unless-stopped` as their restart
  policy. This guards against missing restarts in the case where service
  containers exit with a success status even when they shouldn't.

### Fixed

- Added missing `created_at` field in `blocked_names` table.
- Fixed broken ArNS undername resolution.

## [Release 22] - 2024-12-18

### Added

- Added the ability to block and unblock ArNS names (e.g., to comply with
  hosting provider TOS). To block a name, POST `{ "name": "<name to block>" }`
  to `/ar-io/admin/block-name`. To unblock a name, POST
  `{ "name": "<name to unblock>" }` to `/ar-io/admin/unblock-name`.

### Changed

- Return an HTTP 429 response to POSTs to `/ar-io/admin/queue-bundle` when the
  bundle data import queue is full so that scripts queuing bundles can wait
  rather than overflowing it.

### Fixed

- Adjust ArNS length limit from <= 48 to <= 51 to match the limit enforced by
  the AO process.

## [Release 21] - 2024-12-05

### Added

- Added a ClickHouse auto-import service. When enabled, it calls the Parquet
  export API, imports the exported Parquet into ClickHouse, moves the Parquet
  files to an `imported` subdirectory, and deletes data items in SQLite up to
  where the Parquet export ended. To use it, run Docker Compose with the
  `clickhouse` profile, set the `CLICKHOUSE_URL` to `http://clickhouse:8123`,
  and ensure you have set an `ADMIN_KEY`.
  Using this configuration, the core service will also combine results from
  ClickHouse and SQLite when querying transaction data via GraphQL. Note: if
  you have a large number of data items in SQLite, the first export and
  subsequent delete may take an extended period. Also, this functionality is
  considered **experimental**. We expect there are still bugs to be found in it
  and we may make breaking changes to the ClickHouse schema in the future. If
  you choose to use it in production (not yet recommended), we suggest backing
  up copies of the Parquet files found in `data/parquet/imported` so that they
  can be reimported if anything goes wrong or future changes require it.
- Added a background data verification process that will attempt to recompute
  data roots for bundles and compare them to data roots indexed from Arweave
  nodes. When the data roots match, all descendant data items will be marked as
  verified. This enables verification of data initially retrieived from sources,
  like other gateways, that serve contiguous data instead of verifiable chunks.
  Data verification can be enabled by setting the
  `ENABLE_BACKGROUND_DATA_VERIFICATION` environment variable to true. The
  interval between attempts to verify batches of bundles is configurable using
  the `BACKGROUND_DATA_VERIFICATION_INTERVAL_SECONDS` environment variable.
- Added a `CHUNK_POST_MIN_SUCCESS_COUNT` environment variable to configure how
  many Arweave nodes must accept a chunk before a chunk broadcast is considered
  successful.
- Added `arweave_chunk_post_total` and `arweave_chunk_broadcast_total`
  Prometheus metrics to respectively track the number of successful chunk POSTs
  to Arweave nodes and the number of chunks successfully broadcast.
- When resolving ArNS names, the entire list of names is now cached instead of
  individually checking whether each name exists. This reduces the load on AO
  CUs since the entire list can be reused across multiple requests for
  different names. Note: due to the default 5 minute interval between name list
  refreshes, newly registered may now take longer to resolver after initial
  registration. We intend to make further caching refinements to address this
  in the future.
- Added support for multiple prioritized trusted gateways configurable by
  setting the `TRUSTED_GATEWAYS_URLS` environment variable to a JSON value
  containing a mapping of gateway hosts to priorities. Data requests are sent
  to other gateways in ascending priority order. If multiple gateways share the
  same priority, all the gateways with the same priority are tried in a random
  order before continuing on to the next priority.
- Added support for caching contiguous data in S3. It is enabled by default
  when the `AWS_S3_CONTIGUOUS_DATA_BUCKET` and `AWS_S3_CONTIGUOUS_DATA_PREFIX`
  environment variables are set.

### Changed

- `trusted-gateway` was changed to `trusted-gateways` in
  `ON_DEMAND_RETRIEVAL_ORDER` and `BACKGROUND_RETRIEVAL_ORDER`.
- Renamed the S3 contiguous environment variables - `AWS_S3_BUCKET` to
  `AWS_S3_CONTIGUOUS_DATA_BUCKET` and `AWS_S3_PREFIX` to
  `AWS_S3_CONTIGUOUS_DATA_PREFIX`.

## [Release 20] - 2024-11-15

### Added

- Exposed the core service chunk POST endpoint via Envoy. It accepts a Arweave
  data chunk and broadcasts it to either the comma separated list of URLs
  specified by the CHUNK_POST_URLs environment variable or, if none are
  specified, the `/chunk` path on URL specified by the TRUST_GATEWAY_URL
  environment variable.
- Added a `X-AR-IO-Root-Transaction-Id` HTTP header to data responses
  containing the root base layer transaction ID for the ID in question if it's
  been indexed.
- Added a `X-AR-IO-Data-Item-Data-Offset` HTTP header containing the offset of
  the data item relative to the root bundle base layer transaction for it. In
  conjunction with `X-AR-IO-Root-Transaction-Id`, it enables retrieving data
  for data item IDs from base layer data using first a `HEAD` request to
  retrieve the root ID and data offset followed by a range request into the
  root bundle. This greatly increases the likelihood of retriving data item
  data by ID since only an index into the base layer and Arweave chunk
  availability is needed for this access method to succeed.
- Added an experimental ClickHouse service to `docker-compose.yaml` (available
  via the `clickhouse` profile). This will be used as a supplemental GraphQL DB
  in upcoming releases.
- Added a data item indexing healthcheck that can be enabled by setting the
  `RUN_AUTOHEAL` environment variable to `true`. When enabled, it will restart
  the `core` service if no data items have been indexed since the value
  specified by the `MAX_EXPECTED_DATA_ITEM_INDEXING_INTERVAL_SECONDS`
  environment variable.

## [Release 19] - 2024-10-21

### Fixed

- Adjusted data item flushing to use the bundle DB worker instead of the core
  DB worker to prevent write contention and failed flushes under heavy
  unbundling load.

### Added

- Added `X-AR-IO-Digest`, `X-AR-IO-Stable`, `X-AR-IO-Verified`, and `ETag`
  headers. `X-AR-IO-Digest` contains a base64 URL encoded representation of the
  SHA-256 hash of the data item data. It may be empty if the gateway has not
  previously cached the data locally. `X-AR-IO-Stable` contains either `true`
  or `false` depending on whether the associated Arweave transaction is more
  than 18 blocks old or not. `X-AR-IO-Verified` contains either `true` if the
  gateway has verified the data root of the L1 transaction or the L1 root
  parent of the data item or `false` if it has not. `ETag` contains the same
  value a `X-AR-IO-Digest` and is used to improve HTTP caching efficiency.
- Added support for using a different data source for on-demand and background
  data retrieval. Background data retrieval is used when unbundling. The
  background retrieval data source order is configurable using the
  `BACKGROUND_RETRIEVAL_ORDER` environment variable and defaults to
  `chunks,s3,trusted-gateway,tx-data`. Priority is given to chunk retrieval
  since chunks are verifiable.
- Added an `/ar-io/admin/export-parquet/status` to support monitoring of
  in-progress Parquet export status.
- Added `sqlite_in_flight_ops` Prometheus metric with `worker` (`core`,
  `bundles`, `data`, or `moderation`) and `role` (`read` or `write`) labels to
  support monitoring the number of in-flight DB operations.
- Added experimental Grafana and Prometheus based observability stack. See the
  "Monitoring and Observability" section of the README for more details.

### Changed

- Bundle data is now retrieved as chunks from Arweave nodes by default so that
  data roots can be compared against the chain (see entry about background
  retrieval above).
- Changed observer configuration to use 8 instead of 5 chosen names. These are
  combined with 2 names prescribed from the contract for a total of 10 names
  observed each epoch to provide increased ArNS observation coverage.
- Verification status is set on data items when unbundling a parent that has
  already been verified.

## [Release 18] - 2024-10-01

### Fixed

- Improved performance of data attributes query that was preventing `data.db`
  WAL flushing.

### Added

- Added WAL `sqlite_wal_checkpoint_pages` Prometheus metric to help monitor WAL
  flushing.
- Added a POST `/ar-io/admin/export-parquet` endpoint that can be used to
  export the contents of the SQLite3 core and bundle DBs as Parquet. To trigger
  an export, POST JSON containing `outputDir`, `startHeight`, `endHeight`, and
  `maxFileRows` keys. The resulting Parquet files can then be queried directly
  using DuckDB or loaded into another system (e.g. ClickHouse). Scripts will be
  provided to help automate the latter in a future release.
- Added `ARNS_RESOLVER_OVERRIDE_TTL_SECONDS` that can be used to force ArNS
  names to refresh before their TTLs expire.
- Added a GET `/ar-io/resolver/:name` endpoint that returns an ArNS resolution
  for the given name.

### Changed

- Removed ArNS resolver service in favor of integrated resolver. If a
  standalone resolver is still desired, the core service can be run with the
  `START_WRITERS` environment variable set to `false`. This will disable
  indexing while preserving resolver functionality.
- Deduplicated writes to `data.db` to improve performance and reduce WAL growth
  rate.

## [Release 17] - 2024-09-09

### Notes

- This release includes a **LONG RUNNING MIGRATION**. Your node may appear
  unresponsive while it is running. It is best to wait for it to complete. If
  it fails or is interrupted, removing your SQLite DBs (in `data/sqlite` by
  default) should resolve the issue, provided you are willing to lose your
  GraphQL index and let your node rebuild it.

### Fixed

- Use the correct environment variable to populate WEBHOOK_BLOCK_FILTER in
  `docker-compose.yaml`.
- Don't cache data regions retrieved to satisfy range requests to avoid
  unnecessary storage overhead and prevent inserting invalid ID to hash
  mappings into the data DB.

### Added

- Added a new ClickHouse based DB backend. It can be used in combination with
  the SQLite DB backend to enable batch loading of historical data from
  Parquet. It also opens up the possibility of higher DB performance and
  scalability. In its current state it should be considered a technology
  preview. It won't be useful to most users until we either provide Parquet
  files to load into it or automate flushing of the SQLite DB to it (both are
  planned in future release). It is not intended to be standalone solution. It
  supports bulk loading and efficient GraphQL querying of transactions and data
  items, but it relies on SQLite (or potentially another OLTP in the future) to
  index recent data. These limitations allow greatly simplified schema and
  query construction. Querying the new ClickHouse DB for transaction and data
  items via GraphQL is enabled by setting the 'CLICKHOUSE_URL' environment
  variable.
- Added the ability to skip storing transaction signatures in the DB by setting
  WRITE_TRANSACTION_DB_SIGNATURES to false. Missing signatures are fetched from
  the trusted Arweave node when needed for GraphQL results.
- Added a Redis backed signature cache to support retrieving optimistically
  indexed data item signatures in GraphQL queries when writing data items
  signatures to the DB has been disabled.
- Added on-demand and composite ArNS resolvers. The on-demand resolver
  fetches results directly from an AO CU. The composite resolver attempts
  resolution in the order specified by the ARNS_RESOLVER_PRIORITY_ORDER
  environment variable (defaults to 'on-demand,gateway').
- Added a queue_length Prometheus metric to fasciliate monitoring queues and
  inform future optimizations
- Added SQLite WAL cleanup worker to help manage the size of the `data.db-wal`
  file. Future improvements to `data.db` usage are also planned to further
  improve WAL management.

### Changed

- Handle data requests by ID on ArNS sites. This enables ArNS sites to use
  relative links to data by ID.
- Replaced ARNS_RESOLVER_TYPE with ARNS_RESOLVER_PRIORITY_ORDER (defaults to
  'on-demand,gateway').
- Introduced unbundling back pressure. When either data item data or GraphQL
  indexing queue depths are more than the value specified by the
  MAX_DATA_ITEM_QUEUE_SIZE environment variable (defaults to 100000),
  unbundling is paused until the queues length falls bellow that threshold.
  This prevents the gateway from running out of memory when the unbundling rate
  exceeds the indexing rate while avoiding wasteful bundle reprocessing.
- Prioritized optimistic data item indexing by inserting optimistic data items
  at the front of the indexing queues.
- Prioritized nested bundle indexing by inserting nested bundles at the front
  of the unbundling queue.

## [Release 16] - 2024-08-09

### Fixed

- Fixed promise leak caused by missing await when saving data items to the DB.
- Modified ArNS middleware to not attempt resolution when receiving requests
  for a different hostname than the one specified by `ARNS_ROOT_HOST`.

### Added

- Added support for returning `Content-Encoding` HTTP headers based on user
  specified `Content-Encoding` tags.
- Added `isNestedBundle` filter enables that matches any nested bundle when
  indexing. This enables composite unbundling filters that match a set of L1
  tags and bundles nested under them.
- Added ability to skip writing ANS-104 signatures to the DB and load them
  based on offsets from the data instead. This significantly reduces the size
  of the bundles DB. It can be enabled by setting the
  `WRITE_ANS104_DATA_ITEM_DB_SIGNATURES` environment variable to `false`.
- Added `data_item_data_indexed_total` Prometheus counter to count data items
  with data attributes indexed.

### Changed

- Queue data attributes writes when serving data rather than writing them
  syncronously.
- Reduced the default data indexer count to 1 to lessen the load on the data
  DB.
- Switched a number of overly verbose info logs to debug level.
- Removed docker-compose on-failure restart limits to ensure that services
  restart no matter how many times they fail.
- Modified the `data_items_indexed_total` Prometheus counter to count data
  items indexed for GraphQL querying instead of data attributes.
- Increased aggressiveness of contiguous data cleanup. It now pauses 5 seconds
  instead of 10 seconds per batch and runs every 4 hours instead of every 24
  hours.

## [Release 15] - 2024-07-19

### Fixed

- Fixed query error that was preventing bundles from being marked as fully
  imported in the database.

### Added

- Adjusted data item indexing to record data item signature types in the DB. This
  helps distinguish between signatures using different key formats, and will
  enable querying by signature type in the future.
- Adjusted data item indexing to record offsets for data items within bundles
  and signatures and owners within data items. In the future this will allow us
  to avoid saving owners and signatures in the DB and thus considerably reduce
  the size of the bundles DB.
- Added `ARNS_CACHE_TTL_MS` environment variable to control the TTL of ARNS cache
  entries (defaults to 1 hour).
- Added support for multiple ranges in a single HTTP range request.
- Added experimental chunk POST endpoint that broadcasts chunks to the
  comma-separate list of URLS in the `CHUNK_BROADCAST_URLS` environment
  variable. It is available at `/chunk` on the internal gateway service port
  (4000 by default) but is not yet exposed through Envoy.
- Added support for running an AO CU adjacent to the gateway (see README.md for
  details).
- Added `X-ArNS-Process-Id` to ArNS resolved name headers.
- Added a set of `AO_...` environment variables for specifying which AO
  URLs should be used (see `docker-compose.yaml` for the complete list). The
  `AO_CU_URL` is of particular use since the core and resolver services only
  perform AO reads and only the CU is needed for reads.

### Changed

- Split the monolithic `docker-compose.yaml` into `docker-compose.yaml`,
  `docker-compose.bundler.yaml`, and `docker-compose.ao.yaml` (see README for
  details).
- Replaced references to 'docker-compose' with 'docker compose' in the docs
  since the former is mostly deprecated.
- Reduce max fork depth from 50 to 18 inline to reflect Arweave 2.7.2 protocol
  changes.
- Increased the aggressiveness of bundle reprocessing by reducing reprocessing
  interval from 10 minutes to 5 minutes and raising reprocessing batch size
  from 100 to 1000.
- Use a patched version of Litestream to work around insufficient S3 multipart
  upload size in the upstream version.

## [Release 14] - 2024-06-26

### Fixed

- Correctly handle manifest `index` after `paths`.

## [Release 13] - 2024-06-24

### Added

- Added support for optimistically reading data items uploaded using the
  integrated Turbo bundler via the LocalStack S3 interface.
- Added `X-AR-IO-Origin-Node-Release` header to outbound data requests.
- Added `hops`, `origin`, and `originNodeRelease` query params to
  outbound data requests.
- Added support for `fallback` in v0.2 manifests that is used if no path in
  the the manifest is matched.

### Changed

- Updated Observer to read prescribed names from and write observations to the
  ar.io AO network process.
- Updated Resolver to read from the ar.io AO network process.

### Fixed

- Modified optimistic indexing of data items to use a null `parent_id` when
  inserting into the DB instead of a placeholder value. This prevents
  unexpected non-null `bundledIn` values in GraphQL results for optimistically
  indexed data items.
- Modified GraphQL query logic to require an ID for single block GraphQL
  queries. Previously queries missing an ID were returning an internal SQLite
  error. This represents a small departure from arweave.net's query logic which
  returns the latest block for these queries. We recommend querying `blocks`
  instead of `block` in cases where the latest block is desired.
- Adjusted Observer health check to reflect port change to 5050.

### Security

- Modified docker-compose.yaml to only expose Redis, PostgreSQL, and
  LocalStack ports internally. This protects gateways that neglect to deploy
  behind a firewall, reverse proxy, or load balancer.

## [Release 12] - 2024-06-05

### Added

- Added `/ar-io/admin/queue-data-item` endpoint for queuing data item
  headers for indexing before the bundles containing them are
  processed. This allows trusted bundlers to make their data items
  quickly available to be queried via GraphQL without having to wait for bundle
  data submission or unbundling.
- Added experimental support for retrieving contiguous data from S3. See
  `AWS_*` [environment variables documentation](docs/env.md) for configuration
  details. In conjuction with a local Turbo bundler this allows optimistic
  bundle (but not yet data item) retrieval.
- Add experimental support for fetching data from gateway peers. It can be
  enabled by adding `ario-peer` to `ON_DEMAND_RETRIEVAL_ORDER`. Note: do not
  expect this work reliably yet! This functionality is in active development
  and will be improved in future releases.
- Add `import_attempt_count` to `bundle` records to enable future bundle import
  retry optimizations.

### Changed

- Removed `version` from `docker-compose.yaml` to avoid warnings with recent
  versions of `docker-compose`
- Switched default observer port from 5000 to 5050 to avoid conflict on OS X.
  Since Envoy is used to provide external access to the observer API this
  should have no user visible effect.

## [Release 11] - 2024-05-21

### Added

- Added `arweave_tx_fetch_total` Prometheus metric to track counts of
  transaction headers fetched from the trusted node and Arweave network peers.

### Fixed

- Revert to using unnamed bind mounts due to cross platform issues with named
  volumes.

## [Release 10] - 2024-05-20

### Added

- Added experimental support for streaming SQLite backups to S3 (and compatible
  services) using [Litestream](https://litestream.io/). Start the service using
  the docker-compose 'litestream' profile to use it, and see the
  `AR_IO_SQLITE_BACKUP_*` [environment variables documentation](docs/env.md) for
  further details.
- Added `/ar-io/admin/queue-bundle` endpoint for queuing bundles for import
  before they're in the mempool. In the future, this will enable optimistic
  indexing when combined with a local trusted bundler.
- Added support for triggering webhooks when blocks are imported that match the
  filter specified by the `WEBHOOK_BLOCK_FILTER` environment variable.
- Added experimental support for indexing transactions and related data items
  from the mempool. Enable it by setting the `ENABLE_MEMPOOL_WATCHER` environment
  variable to 'true'.
- Made on-demand data caching circuit breakers configurable via the
  `GET_DATA_CIRCUIT_BREAKER_TIMEOUT_MS` environment variable. This allows gateway
  operators to decide how much latency they will tolerate when serving data in
  exchange for more complete data indexing and caching.
- Added `X-AR-IO-Hops` and `X-AR-IO-Origin` headers in preparation for future
  peer-to-peer data functionality.

### Changed

- Renamed cache header from `X-Cached` to `X-Cache` to mimic typical CDN
  practices.
- Upgrade to Node.js v20 and switch to the native test runner.

## [Release 9] - 2024-04-10

### Added

- Added experimental Farcaster Frames support enabling simple Areave based
  Frames with button navigation. Transaction and data item data is now served
  under `/local/farcaster/frame/<ID>`. `/local` is used as a prefix to indicate
  this functionality is both experimental and local to a particular the gateway
  rather than part of the global gateway API. Both GET and POST requests are
  supported.
- Added an experimental local ArNS resolver. When enabled it removes dependence
  on arweave.net for ArNS resolution! Enable it by setting `RUN_RESOLVER=true`,
  `TRUSTED_ARNS_RESOLVER_TYPE=resolver`, and
  `TRUSTED_ARNS_RESOLVER_URL=http://resolver:6000` in your `.env` file.
- Added a `CONTIGUOUS_DATA_CACHE_CLEANUP_THRESHOLD` environment variable
  that represents a threshold age in seconds to be compared with a contiguous
  data file age. If file is older than the amount of seconds set in the
  enviroment variable it will be deleted.
- Added an 'X-Cached' header to data responses to indicate when data is served
  from the local cache rather than being retrieved from an external source. This
  is helpful for interfacing with external systems, debugging, and end-to-end
  testing.
- Save hashes for unbundled data items during indexing. This enables reduction
  in data storage via hash based deduplication as well as more efficient
  peer-to-peer data retrieval in the future.

## [Release 8] - 2024-03-14

### Added

- Add GraphQL SQL query debug logging to support trouble shooting and
  performance optimization.
- Add support for indexing data items (not GraphQL querying) based solely on
  tag name (example use case: indexing all IPFS CID tagged data items).

### Changes

- Observer data sampling now uses randomized ranges to generate content hashes.
- Reference gateway ArNS resolutions are now cached to improve report
  generation performance.
- Contract interactions are now tested before posting using `dryWrite` to avoid
  submitting interactions that would fail.
- `/ar-io/observer/info` now reports `INVALID` for wallets that fail to load.

### Fixed

- Fix data caching failure caused by incorrect method name in getData\* circuit
  breakers.
- Fix healthcheck when ARNS_ROOT_HOST includes a subdomain.

## [Release 7] - 2024-02-14

### Added

- Add support for notifiying other services of transactions and data items
  using webhooks (see README for details).
- Add support for filter negation (particularly useful for excluding large
  bundles from indexing).
- Improve unbundling throughput by decoupling data fetching from unbundling.
- Add Envoy and core service ARM builds.

### Changed

- Improve resource cleanup and shutdown behavior.
- Don't save Redis data to disk by default to help prevent memory issues on
  startup for small gateways.
- Reduce the amount of data sampled from large files by the observer.
- Ensure block poa2 field is not cached to reduce memory consumption.

## [Release 6] - 2024-01-29

### Fixed

- Update observer to improve reliability of contract state synchronization and
  evaluation.

## [Release 5] - 2024-01-25

### Added

- Added transaction offset indexing to support future data retrieval
  capabilities.
- Enabled IPv6 support in Envoy config.
- Added ability to configure observer report generation interval via the
  REPORT_GENERATION_INTERVAL_MS environment variable (intended primarily for
  development and testing).

### Changed

- Updated observer to properly handle FQDN conflicts.
- Renamed most created_at columns to indexed_at for consistency and clarity.

### Fixed

- Updated LMDB version to remove Buffer workaround and fix occassional block
  cache errors.

## [Release 4] - 2024-01-11

### Added

- Added circuit breakers around data index access to reduce impact of DB access
  contention under heavy requests loads.
- Added support for configuring data source priority via the
  ON_DEMAND_RETRIEVAL_ORDER environment variable.
- Updated observer to a version that retrieves epoch start and duration from
  contract state.

### Changed

- Set the Redis max memory eviction policy to `allkeys-lru`.
- Reduced default Redis max memory from 2GB to 256MB.
- Improved predictability and performance of GraphQL queries.
- Eliminated unbundling worker threads when filters are configured to skip
  indexing ANS-104 bundles.
- Reduced the default number of ANS-104 worker threads from 2 to 1 when
  unbundling is enabled to conserve memory.
- Increased nodejs max old space size to 8GB when ANS-104 workers > 1.

### Fixed

- Adjusted paths for chunks indexed by data root to include the full data root.

## [Release 3] - 2023-12-05

### Added

- Support range requests ([PR 61], [PR 64])
  - Note: serving multiple ranges in a single request is not yet supported.
- Release number in `/ar-io/info` response.
- Redis header cache implementation ([PR 62]).
  - New default header cache (replaces old FS cache).
- LMDB header cache implementation ([PR 60]).
  - Intended for use in development only.
  - Enable by setting `CHAIN_CACHE_TYPE=lmdb`.
- Filesystem header cache cleanup worker ([PR 68]).
  - Enabled by default to cleanup old filesystem cache now that Redis
    is the new default.
- Support for parallel ANS-104 unbundling ([PR 65]).

### Changed

- Used pinned container images tags for releases.
- Default to Redis header cache when running via docker-compose.
- Default to LMDB header cache when running via `yarn start`.

### Fixed

- Correct GraphQL pagination for transactions with duplicate tags.

[Parquet and ClickHouse usage guide]: ./docs/parquet-and-clickhouse-usage.md
[PR 68]: https://github.com/ar-io/ar-io-node/pull/68
[PR 65]: https://github.com/ar-io/ar-io-node/pull/65
[PR 64]: https://github.com/ar-io/ar-io-node/pull/64
[PR 62]: https://github.com/ar-io/ar-io-node/pull/62
[PR 61]: https://github.com/ar-io/ar-io-node/pull/61
[PR 60]: https://github.com/ar-io/ar-io-node/pull/60