Dshackle Archive

Dshackle Archive copies JSON data from a blockchain to a plain files ("tables") archive (i.e., it’s the Extraction of data, as in ETL).

The archive is Avro or JSON files which contain blocks and transaction details fetched from blockchain nodes via their APIs. The resulting archive contain the JSON responses as is.

Features:

Extracts data from Bitcoin and Ethereum compatible blockchains.
Produces data in Avro or JSON, where one file could be a range of blocks or a separate file per each block
Archival can be scaled by using multiple blockchain nodes by using Dshackle Load Balancer.
Runs in:
- batch archive mode, when historical data is archived
- or in a streaming mode when only fresh data is added to the archive
Archive to:
- Filesystem
- S3
- Apache Pulsar
- Apache Kafka
Produces Notifications to:
- Filesystem
- Apache Pulsar
- Apache Kafka

The general idea it to use Dshackle Archive to copy data from a blockchain to a plain files, keeping the data structures as is, and then use traditional Big Data tools (Spark, Flink, Beam, etc.) to analyze the blockchain data.

Note	It takes several weeks to archive the whole blockchain. It’s a very IO intensive operation, but can be scaled by using multiple nodes.

Note	It uses Dshackle protocol to connect to the blockchain, not a plain HTTP/JSON RPC. A Dshackle instance is required to be running.

Usage

Command Line Options

Usage: dshackle-archive [OPTIONS] --blockchain <BLOCKCHAIN> --connection <HOST:PORT> <COMMAND>

Arguments:
  <COMMAND>  [possible values: stream, fix, verify, archive, compact]

Options:
  -b, --blockchain <BLOCKCHAIN>
          Blockchain

      --dry-run
          Do not modify the storage

  -c, --connection <HOST:PORT>
          Connection (host:port)

      --connection.notls
          Disable TLS

      --parallel <PARALLEL>
          How many API requests to make in parallel. Range: 1..512. Default: 16

      --notify.dir <NOTIFY_DIR>
          Write notifications as JSON line to the specified dir in a file <dshackle-archive-%STARTTIME.jsonl>

      --notify.pulsar.topic <PULSAR_TOPIC>
          Send notifications as JSON to the Pulsar to the specified topic (notify.pulsar.url must be specified)

      --notify.pulsar.url <PULSAR_URL>
          Send notifications as JSON to the Pulsar with specified URL (notify.pulsar.topic must be specified)

      --notify.kafka.topic <KAFKA_TOPIC>
          Send notifications as JSON to the Kafka to the specified topic (notify.kafka.url must be specified)

      --notify.kafka.url <KAFKA_URL>
          Send notifications as JSON to the Kafka brokers at the specified addresses, i.e. `kafka://localhost:9092` or a comma separated list of them (notify.kafka.topic must be specified)

      --auth.aws.access-key <ACCESS_KEY>
          AWS / S3 Access Key

      --auth.aws.secret-key <SECRET_KEY>
          AWS / S3 Secret Key

      --aws.endpoint <ENDPOINT>
          AWS / S3 endpoint url instead of the default one

      --aws.region <REGION>
          AWS / S3 region ID to use for requests

      --aws.s3.path-style
          Enable S3 Path Style access (default is false). Use this flag for a no-AWS service

      --aws.trust-tls
          Trust any TLS certificate for AWS / S3 (default is false)

      --stream.url <STREAM_URL>
          Publish stream data to a broker at the given URL. The scheme selects the backend:

          - `pulsar://HOST:PORT` — Apache Pulsar.

          - `kafka://HOST:PORT[,HOST:PORT]` — Apache Kafka, as a list of bootstrap brokers.

          Selecting a streaming target restricts the run to the `stream` command — `archive`, `fix`, `verify`, and `compact` are rejected at startup because topics are append-only.

      --stream.topics <STREAM_TOPICS>
          Prefix used to build the per-field topic names. Each field is published to `<prefix>-<field>` (e.g. `<prefix>-blocks`, `<prefix>-tx-json`). For Pulsar, include the full topic path up to the prefix, e.g. `persistent://public/default/archive-eth`; for Kafka it's a plain topic name, e.g. `archive-eth`

  -d, --dir <DIR>
          Target directory

      --continue
          [Stream Command] Continue from the last archived block; i.e., not the latest in blockchain

      --tail <TAIL>
          [Verify/Fix Commands] Use the latest T blocks instead of a range

  -r, --range <RANGE>
          Blocks Range (`N...M`)

      --range.chunk <RANGE_CHUNK>
          Range chunk size (default 1000)

  -t, --tables <TABLES>
          Types of tables to archive (comma-separated list of `blocks`, `txes`, `traces`). Default: blocks,txes

      --fields.trace <FIELDS_TRACE>
          List of data to include into tracing archive table (comma-separated list of `calls`, `stateDiff`). Default: calls,stateDiff; Used only if `traces` are included into the archived tables (see `--tables` option); Details: `calls` - debug_traceTransaction with `callTracer` tracing; `stateDiff` - debug_traceTransaction with `prestateTracer` tracing

      --fix.clean
          [Fix Command] Set to remove any existing data in whole chunk if any of tables is missing a block in the chunk or has broken values. Default is `false`, which deleted only tables with missing / corrupted data

      --compression <COMPRESSION>
          Compression algorithm to use for new output. For `--format=avro` it's the Avro file codec; for a streaming target it's the producer compression applied to every message. Default is `zstd`

          [possible values: snappy, zstd]

      --retry <RETRY>
          Retry policy for transient blockchain fetch failures.

          - `bounded` — give up after a fixed number of attempts (current behaviour for file targets). - `forever` — keep retrying indefinitely with exponential backoff. Required for ordered streaming targets where a missed record breaks the topic-order contract.

          Defaults: `forever` for streaming-ordered targets (Pulsar, Kafka), `bounded` otherwise.

          Possible values:
          - bounded: Give up after a fixed number of attempts. Default for file targets, where a failed fetch can be repaired later by the `fix` command
          - forever: Keep retrying indefinitely. Default for streaming-ordered targets (Pulsar) where a gap breaks the topic-order contract permanently

      --follow <FOLLOW>
          [Stream Command] Follow mode for new blocks: `latest` - follow the latest blocks (default); `finalized` - follow only finalized blocks

          [default: latest]
          [possible values: latest, finalized]

      --format <FORMAT>
          Output format. `avro` (default) writes one row-batched Avro file per (kind, range) under the historical layout. `json` writes one JSON file per field (block.json, tx-<HASH>.json, receipt-<HASH>.json, raw-<HASH>.hex, …) inside a directory per height. `compact` and `verify` are not supported with `json`

          Possible values:
          - avro: One Avro file per (kind, range). Historical layout — supports all commands
          - json: One JSON file per field under a per-height directory. `archive`, `stream`, and `fix` are supported; `compact` and `verify` are rejected at startup

          [default: avro]

      --metrics <HOST:PORT>
          Start a Prometheus-compatible metrics server on the given address (e.g., 127.0.0.1:8080). Metrics are served at http://HOST:PORT/metrics

      --metrics.await
          After the main command finishes, keep the metrics server running until one final scrape completes (or 60 seconds elapse). Useful for short-lived commands (fix, verify, compact) to ensure Prometheus collects the final metrics before the process exits

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Commands

archive - the main operation, copies data from a blockchain to archive
stream - append fresh blocks one by one to the archive
compact - merge individual block files into larger range files
fix - fix archive by making new archives for missing chunks
verify - verify that archive files contains required data and delete incomplete files

Stream as Tables

Continuously append fresh blocks one by one to the archive. In addition to the copying, Dshackle archive can be configured to notify an external system about new blocks in the archive.

Note that when it’s in streaming mode the archives are writen in a per-block basis. I.e., each block comes with in a separate group of files. One for the block itself, another one for all transactions in that block, and third for traces if selected. To merge the individual files into larger ranges use compact command.

To notify an external system, there are following options:

--notify.dir - write notifications as JSON line to the specified dir in a file <dshackle-archive-%STARTTIME.jsonl>
--notify.pulsar.url + --notify.pulsar.topic - send notifications as JSON to the specified Apache Pulsar topic
--notify.kafka.url + --notify.kafka.topic - send notifications as JSON to the specified Apache Kafka topic

Only one of those targets can be used at a time, and a broker needs both its URL and its topic. Anything else stops the archive at startup with an error, instead of quietly sending nothing.

For Kafka the URL is a list of bootstrap brokers, i.e. kafka://localhost:9092 or kafka://kafka-1:9092,kafka-2:9092 (the kafka:// prefix is optional). The topic must exist before the archive starts, unless the brokers are configured to auto-create topics.

Notifications are distributed over the topic partitions by heightStart, the first block of the notified range. For stream and archive that is the block itself, so all the notifications about one block go to the same partition and keep their order.

A notification that cannot be delivered stops the run with an error: otherwise the archive and the systems reading the topic would silently disagree about what has been archived.

See Notification format.

Stream as Records

When --stream.url passed Dshackle Archive publishes stream data to a broker at the given URL. Topic receives each individual record in a block. I.e., instead of writing a large Avro file like in "Stream as Tables" mode, this mode produces a new message for each transaction in a block.

The URL scheme selects the broker:

pulsar://HOST:PORT for Apache Pulsar
kafka://HOST:PORT for Apache Kafka, or a comma separated list of bootstrap brokers (kafka://kafka-1:9092,kafka-2:9092)

--stream.topics specifies the prefix used to build the per-field topic names. Each field is published to <prefix>-<field> (e.g. <prefix>-blocks, <prefix>-tx-json). For Pulsar the prefix includes the full topic path (e.g. persistent://public/default/ethereum); for Kafka it’s a plain topic name.

Every message of one block goes to the same partition, selected by block height, so a consumer sees the records of a block, and of any re-org replacement for it, in publish order.

Topics used if prefix specified as ethereum:

ethereum-blocks
ethereum-blocks-uncles
ethereum-tx-json
ethereum-tx-raw
ethereum-tx-receipts
ethereum-trace-calls
ethereum-trace-statediff

The JSON format is:

table - blocks, transactions or traces
field - the type of message, e.g. block, "uncles", tx, "raw", "receipt", "calls", statediff
blockchain - what was specified in --blockchain option, e.g. ETH or BTC
timestamp - Block timestamp as reported by the node, serialized as RFC 3339
height - Block height, as number
blockId - Block hash
parentId - Parent block hash
txCount - Number of transactions in the block
txIndex - Transaction index in block. Present on tx/trace records
txId - Transaction hash. Present on tx/trace records
uncleIndex - Uncle index. Present on Ethereum uncle messages only
value - Original data (JSON or HEX string for raw transaction)

Compact

Merge individual block files into larger range files.

Fix

Fixes the archive by checking if there are any missing blocks, and if so, it creates new archives for the missing blocks.

Verify

Verify that archive files contains required data and delete incomplete/corrupted files. The a fix command is supposed to run to download missing blocks.

Warning

This command is destructive, it deletes files from the archive.

Archive Size

Dshackle Archive copies and stored data as JSON responses from blockchain nodes the resulting archive is much larger that the node database size, which keeps data in a compact format. It uses Snappy compression for Avro files, which give a good compression ratio, but still the resulting archive is large.

Average size of a 1000 blocks range (w/o expensive JSON such as stateDiff and calls):

~300Mb for Ethereum
~400Mb for Bitcoin

And the whole archive (w/o expensive JSON such as stateDiff and calls):

~2.5Tb for Ethereum
~1.9Tb for Bitcoin

Avro structure and Java stubs: https://github.com/emeraldpay/dshackle-archive-avro
Dshackle load balancer: https://github.com/emeraldpay/dshackle

FAQ

How to organize the data gathering process?

First you need to archive the historical data, which may takes several week depending on how many and how fast nodes you have.
After finishing the initial archive, you run in the Streaming mode which append new blocks to the archive as they are mined.
Periodically (ex. once a day) you run Compaction to merge individual block files into larger range files.
Also, periodically (ex. once a day) you run a pair of Verify and Fix commands to ensure the integrity of the archive.

What are supported blockchains?

Dshackle requires only compatibility onj JSON RPC level, so technically it can work with any blockchain that uses similar API. I.e., it’s compatible with all major blockchains, including Bitcoin, Ethereum, Binance Smart Chain, Polygon, etc.

What blockchain API it uses?

It uses Dshackle protocol to connect to the blockchain, not a plain HTTP/JSON RPC. So a Dshackle instance is required to be running.

Dshackle is a Load Balancer for Blockchain APIs, and it can route requests to multiple nodes, which scales up the archival throughput.

How does Dshackle Archive ensure the integrity and accuracy?

Dshackle provides two commands to ensure the integrity of the archive:

first you run verify command, which checks the archive and deletes incomplete or corrupted files
then you run fix command, which copies the data again for the blocks deleted in the previous step

You can schedule the execution of these commands to run periodically, e.g. once a day. To avoid scanning the whole archive every time, you can specify a range to check, e.g. --tail 1000. The option above specifies that it should verify/fix only the last 1000 blocks. I.e., it goes backward from the current head block.

Archive Format

For a complete descriptions, schema and libs to access Avro files please refer to https://github.com/emeraldpay/dshackle-archive-avro

Block

Fields common between different blockchains

blockchainType - type of blockchain, as a definitions of what fields to expect. One of ETHEREUM or BITCOIN
blockchainId - actual blockchain id (ETH, BTC, etc)
archiveTimestamp - when the archive record was created. Milliseconds since epoch
height - block height
blockId - block hash
timestamp - block timestamp. Milliseconds since epoch
parentId - parent block hash
json - JSON response for that block

Ethereum specific fields

unclesCount - number of uncles for the current block
uncle0Json - JSON for first uncle (eth_getUncleByBlockHashAndIndex(0))
uncle1Json - JSON for second uncle (eth_getUncleByBlockHashAndIndex(1))

Bitcoin specific fields

none

Transaction

Fields common between different blockchains

blockchainType - type of blockchain, as a definitions of what fields to expect. One of ETHEREUM or BITCOIN
blockchainId - blockchain id (ETH, BTC, etc)
archiveTimestamp - when the archive record was created. Milliseconds since epoch
height - block height
blockId - block hash
timestamp - block timestamp. Milliseconds since epoch
index - index of the transaction in block
txid - hash or transaction id of the transaction
json - JSON response for that transaction
raw - raw bytes of the transaction

Ethereum specific fields

from - from address
to - to address
receiptJson - JSON response for eth_getTransactionReceipt

Bitcoin specific fields

none

Transaction Trace

Note	Applicable only for Ethereum-compatible blockchains.

Fields same as in Transaction:

blockchainType - type of blockchain, as a definitions of what fields to expect. One of ETHEREUM or BITCOIN
blockchainId - blockchain id (ETH, BTC, etc)
archiveTimestamp - when the archive record was created. Milliseconds since epoch
height - block height
blockId - block hash
timestamp - block timestamp. Milliseconds since epoch
index - index of the transaction in block
txid - hash or transaction id of the transaction

Trade data:

traceJson - JSON response for debug_traceTransaction({tracer: "callTracer"})
stateDiffJson - JSON response for debug_traceTransaction({tracer: "prestateTracer"})

Notification format

{
  "version":"https://schema.emrld.io/dshackle-archive/notify/v2",
  "ts":"2022-05-20T23:14:24.481327Z",
  "blockchain":"ETH",
  "type":"transactions",
  "run":"stream",
  "maturity": "finalized",
  "heightStart":14813875,
  "heightEnd":14813875,
  "location": {
    "type": "file",
    "url": "s3://my-bucket/blockchain-archive/eth/014000000/014813000/014813875.txes.avro"
  }
}

Where

version id of the current JSON format
ts timestamp of the archive event
blockchain blockchain
type type of data (transactions, blocks, or traces)
run mode in which the Dshackle Archive is run (archive, stream, copy or compact)
maturity block maturity level (latest or finalized); finalized is applicable to Ethereum PoS chains only
heightStart and heightEnd range of blocks covered by the notification
location where the data landed; an object distinguished by its own type field, see Location types

Location types

The shape of location depends on the target the archive writes to. The location.type defines the structure of the location object.

`file`

One row-batched file (the Avro layout):

{
  "type": "file",
  "url": "s3://my-bucket/eth/014000000/range-014813000_014813999.txes.avro"
}

`files`

Per-field files of a single height (ex. for the --format=json).

One notification is sent per kind per height. Each entry of files describes one block or one transaction, pointing to its individual files.

{
  "type": "files",
  "files": [
    {
      "txId": "0x40846886cf7b8...",
      "tx": "s3://my-bucket/eth/014000000/014813000/014813875/tx-0x40846886cf7b8....json",
      "raw": "s3://my-bucket/eth/014000000/014813000/014813875/raw-0x40846886cf7b8....hex",
      "receipt": "s3://my-bucket/eth/014000000/014813000/014813875/receipt-0x40846886cf7b8....json"
    }
  ]
}

Per-entry fields (only the produced ones are present)

txId - transaction id, on per-transaction entries
block - URL of the block JSON
uncles - URLs of the uncle JSONs, in uncle-index order
tx - URL of the transaction JSON
raw - URL of the raw transaction (hex)
receipt - URL of the transaction receipt JSON
calls - URL of the callTracer trace JSON
stateDiff - URL of the prestateTracer trace JSON

`pulsar`

Messages published to per-field topics of an Apache Pulsar broker (For the --stream.url target).

One notification is sent per kind per height; each message names its actual topic and is identified by its broker message id (ledgerId:entryId:partition[:batchIndex]):

{
  "type": "pulsar",
  "messages": [
    {
      "topic": "persistent://public/default/archive-eth-tx-json",
      "field": "tx-json",
      "txId": "0x40846886cf7b8...",
      "messageId": "125:4:-1"
    }
  ]
}

`kafka`

Records published to per-field topics of an Apache Kafka cluster (For the --stream.url target).

Same shape as pulsar, except a record is identified by its position within the topic, as partition:offset:

{
  "type": "kafka",
  "messages": [
    {
      "topic": "archive-eth-tx-json",
      "field": "tx-json",
      "txId": "0x40846886cf7b8...",
      "messageId": "2:1041"
    }
  ]
}

Community

Development Chat

Join our Discord chat to discuss development and ask questions:

Commercial Support

Want to support the project, prioritize a specific feature, or get commercial help with using Dshackle in your project? Please contact splix@emerald.cash to discuss the possibility.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Name		Name	Last commit message	Last commit date
Latest commit History 225 Commits
.github/workflows		.github/workflows
src		src
testdata		testdata
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
LICENSE		LICENSE
README.adoc		README.adoc
build.rs		build.rs

Navigation Menu

Uh oh!

Repository files navigation

Dshackle Archive

Usage

Command Line Options

Commands

Archive

Stream as Tables

Stream as Records

Compact

Fix

Verify

Archive Size

FAQ

How to organize the data gathering process?

What are supported blockchains?

What blockchain API it uses?

How does Dshackle Archive ensure the integrity and accuracy?

Archive Format

Block

Transaction

Transaction Trace

Notification format

Location types

`file`

`files`

`pulsar`

`kafka`

Community

Development Chat

Commercial Support

License

About

Releases

Packages

Used by

Contributors

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Dshackle Archive

Usage

Command Line Options

Commands

Archive

Stream as Tables

Stream as Records

Compact

Fix

Verify

Archive Size

Related projects:

FAQ

How to organize the data gathering process?

What are supported blockchains?

What blockchain API it uses?

How does Dshackle Archive ensure the integrity and accuracy?

Archive Format

Block

Transaction

Transaction Trace

Notification format

Location types

file

files

pulsar

kafka

Community

Development Chat

Commercial Support

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

`file`

`files`

`pulsar`

`kafka`