[#83] add: gRPC parity test suite (Zainod vs. Lightwalletd) by pacu · Pull Request #84 · zcash/integration-tests

pacu · 2026-04-08T00:28:18Z

Summary

Adds a test suite that runs Zainod and Lightwalletd side-by-side against
the same Zebrad node and compares their CompactTxStreamer gRPC responses.
Covers 21 test cases across 15 RPC methods. Integrates with the existing
BitcoinTestFramework and CI pipeline, and can be triggered from the
Lightwalletd repo via repository_dispatch.

Closes #83

Changes

New files

lightwallet-protocol/ — canonical proto source via git subtree from
zcash/lightwallet-protocol v0.4.0
qa/rpc-tests/test_framework/proto/ — generated Python gRPC stubs,
committed so CI only needs grpcio at runtime (not grpcio-tools)
scripts/generate_proto.sh — developer script to regenerate stubs after
a protocol version bump; fixes flat imports to relative automatically
qa/rpc-tests/grpc_comparison.py — test file
qa/zcash/grpc_comparison_tests.py — convenience runner
(uv run ./qa/zcash/grpc_comparison_tests.py)

Modified files

util.py / test_framework.py — Lightwalletd process lifecycle:
lwd_grpc_port, write_lwd_conf, start_lightwalletd,
wait_for_lwd_start, teardown
pyproject.toml — added grpcio and protobuf runtime dependencies
.github/workflows/ci.yml — added lightwalletd-interop-request
dispatch trigger, build-lightwalletd job, artifact download and
LIGHTWALLETD env var in test-rpc
README.md and doc/book/ — prerequisites, run instructions, and
writing guide updated

RPC methods tested

Method	Cases
`GetLightdInfo`	field-by-field (skipping implementation-specific fields)
`GetLatestBlock`	height and hash agree
`GetBlock`	header fields agree; both error on out-of-bounds
`GetBlockNullifiers`	header fields agree
`GetBlockRange`	forward and reverse; both error on out-of-bounds
`GetBlockRangeNullifiers`	forward and reverse
`GetTransaction`	raw bytes and height agree
`GetTaddressTxids`	full range, lower bound, upper bound
`GetTaddressBalance`	value agrees
`GetTaddressBalanceStream`	value agrees
`GetTreeState`	by height; both error on out-of-bounds
`GetLatestTreeState`	network, height, hash, tree roots agree
`GetSubtreeRoots`	Sapling and Orchard
`GetAddressUtxos`	sorted by (txid, index)
`GetAddressUtxosStream`	sorted by (txid, index)

Known divergences

Two behavioral differences between Zainod and Lightwalletd surfaced during
implementation. Both are worked around in the current tests but should be
investigated separately.

1. vtx in compact blocks

For blocks containing only transparent transactions (in our test chain:
all blocks are coinbase-only), Zainod returns an empty vtx while
Lightwalletd includes those transactions. Whether this is a bug, a protocol
interpretation difference, or expected behavior is unclear. GetBlock and
GetBlockRange tests currently compare header fields only (height, hash,
prevHash, time, chainMetadata).

2. gRPC error codes on out-of-bounds requests

Zainod returns OUT_OF_RANGE; Lightwalletd returns INVALID_ARGUMENT for
the same out-of-bounds inputs. Out-of-bounds tests assert only that both
sides raise a gRPC error, not that the codes match.

Out of scope

GetMempoolTx / GetMempoolStream — require wallet integration to
submit mempool transactions (TODO in test file)
Shielded transaction coverage — requires Zallet integration or a chain
cache with pre-existing shielded activity
GetSubtreeRoots with completed subtrees — each requires 2^16 outputs;
not feasible in a clean regtest chain (both sides return empty stream;
agreement is asserted)
SendTransaction, darkside mode

Test plan

Binaries available at ./src/ or via env vars (ZEBRAD, ZAINOD,
LIGHTWALLETD)
uv sync to pick up grpcio and protobuf dependencies
uv run ./qa/zcash/grpc_comparison_tests.py passes
uv run ./qa/zcash/grpc_comparison_tests.py --nocleanup passes and
<tmpdir>/lwd0/lwd.log contains no fatal errors
uv run ./qa/zcash/full_test_suite.py still passes
scripts/generate_proto.sh produces identical output to the
committed stubs
Push to this branch: CI pipeline passes with grpc_comparison.py
in the test-rpc shard output

Note: Currently there are things in Zaino that need fixing before this test can pass.

git-subtree-dir: lightwallet-protocol git-subtree-split: 23f0768ea4471b63285f3c0e9b6fbb361674aa2b

…et-protocol'

Closes zcash#83 Runs Zainod and Lightwalletd side-by-side against the same Zebrad node and compares their CompactTxStreamer gRPC responses. Covers 21 test cases across 15 RPC methods and integrates with the existing BitcoinTestFramework and CI pipeline. - `lightwallet-protocol/` — canonical proto source via `git subtree` from `zcash/lightwallet-protocol` v0.4.0 - `qa/rpc-tests/test_framework/proto/` — generated Python gRPC stubs committed so CI needs only `grpcio` at runtime - `scripts/generate_proto.sh` — regenerates stubs after a protocol version bump and fixes flat imports to relative - `util.py` / `test_framework.py` — Lightwalletd process lifecycle (`lwd_grpc_port`, `write_lwd_conf`, `start_lightwalletd`, `wait_for_lwd_start`, teardown) - `qa/rpc-tests/grpc_comparison.py` — test file - `qa/zcash/grpc_comparison_tests.py` — convenience runner - CI: `lightwalletd-interop-request` dispatch trigger, `build-lightwalletd` job, artifact download in `test-rpc` - Docs: README and book updated with prerequisites and run instructions 1. **`vtx` in compact blocks** — For blocks containing only transparent transactions, Zainod returns empty `vtx`; Lightwalletd includes them. Block comparison currently covers header fields only. 2. **gRPC error codes on out-of-bounds requests** — Zainod returns `OUT_OF_RANGE`; Lightwalletd returns `INVALID_ARGUMENT`. Tests assert only that both sides raise an error. - `GetMempoolTx` / `GetMempoolStream` (need wallet integration) - Shielded transaction coverage (need Zallet or chain cache) - Completed subtree roots (need 2^16 outputs per tree) - `SendTransaction`, darkside mode Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Adds a gRPC parity test suite that runs Zainod and Lightwalletd side-by-side against the same Zebrad node and compares CompactTxStreamer responses, plus vendored protocol sources and generated Python stubs to run in CI.

Changes:

Added grpc_comparison.py RPC test and a convenience runner to execute it via the existing rpc-tests harness.
Added Lightwalletd lifecycle support to the Python test framework (ports, config generation, start/wait/stop/teardown).
Vendored lightwallet-protocol proto sources and checked in generated Python gRPC/protobuf stubs, plus a script to regenerate them.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
scripts/generate_proto.sh	Regenerates Python protobuf/gRPC stubs from the vendored proto subtree (with import-rewrite post-processing).
qa/zcash/grpc_comparison_tests.py	Convenience runner that invokes the rpc-tests harness for the new gRPC parity test.
qa/rpc-tests/test_framework/util.py	Adds Lightwalletd binary resolution, gRPC port allocation, config writing, and process start/wait/teardown helpers.
qa/rpc-tests/test_framework/test_framework.py	Wires Lightwalletd startup/teardown into the core BitcoinTestFramework lifecycle.
qa/rpc-tests/test_framework/proto/__init__.py	Declares the generated stubs as a package.
qa/rpc-tests/test_framework/proto/service_pb2.pyi	Generated typing stubs for `service.proto`.
qa/rpc-tests/test_framework/proto/service_pb2.py	Generated protobuf runtime code for `service.proto`.
qa/rpc-tests/test_framework/proto/service_pb2_grpc.py	Generated gRPC client/server code for `CompactTxStreamer`.
qa/rpc-tests/test_framework/proto/compact_formats_pb2.pyi	Generated typing stubs for `compact_formats.proto`.
qa/rpc-tests/test_framework/proto/compact_formats_pb2.py	Generated protobuf runtime code for `compact_formats.proto`.
qa/rpc-tests/test_framework/proto/compact_formats_pb2_grpc.py	Generated gRPC glue for `compact_formats.proto`.
qa/rpc-tests/grpc_comparison.py	New parity test suite covering multiple `CompactTxStreamer` methods and error paths.
qa/pull-tester/rpc-tests.py	Registers `grpc_comparison.py` in the rpc-tests runner.
lightwallet-protocol/walletrpc/service.proto	Vendored canonical proto definitions (subtree).
lightwallet-protocol/walletrpc/compact_formats.proto	Vendored canonical proto definitions (subtree).
lightwallet-protocol/LICENSE	Vendored license for the protocol subtree.
lightwallet-protocol/CHANGELOG.md	Vendored protocol changelog (includes v0.4.0 notes).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Fix flat import in service_pb2.pyi (relative import was missing, unlike the .py counterpart); extend generate_proto.sh to rewrite imports in .pyi files as well - Add missing height assertion in test_get_taddress_txids_lower - Add per-element data+height assertions in test_get_taddress_txids_upper (previously only checked stream length) - Add missing script and height assertions in test_get_address_utxos_stream to match the coverage in test_get_address_utxos - Fix gRPC channel leak in _wait_for_indexers: wrap channel lifecycle in try/finally and close both channels on exit - Pad chain to 100 blocks before starting indexers: Zainod requires a minimum of 100 blocks; the three mining phases only produce 36 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

pacu · 2026-04-10T00:28:45Z

Addressed all Copilot review findings:

service_pb2.pyi flat import — fixed the missing relative import in the committed stub and extended generate_proto.sh to rewrite imports in .pyi files alongside .py and _grpc.py.
test_get_taddress_txids_lower missing height assert — added.
test_get_taddress_txids_upper only checking length — added per-element data and height assertions.
test_get_address_utxos_stream missing script and height — added to match the coverage in test_get_address_utxos.
gRPC channel leak in _wait_for_indexers — wrapped channel lifecycle in try/finally, both channels are now closed on exit.
grpcio/protobuf not in dependencies — already present in pyproject.toml from the original PR; non-issue. Import works correctly against both protobuf 6.x and 7.x.
Chain too short for Zainod (30 blocks) — added a padding step after the three mining phases to bring the chain to 100 blocks before calling setup_indexers(). prepare_chain() couldn't be used directly here since zebrad is a passive peer and only zcashd mines in this test.

…ddress config setmineraddress has been removed from this zcashd version, so the mining address must be set via zcash.conf; zcashd is restarted between phases. P2P block propagation is replaced by submitblock to sidestep known zcashd↔zebrad P2P issues (zebra#10329, #10332). Chain layout changes to match: Blocks 1–100 — Orchard coinbase (single zcashd restart with orchard_addr). Block 101 — z_sendmany confirmation: transparent + Sapling outputs in one tx. The Sapling tree is now correctly expected to be empty at Orchard-coinbase height and non-empty only after block 101. authproxy.py is fixed to accept both 'application/json' and 'application/json; charset=utf-8' so zcashd RPC calls no longer raise a content-type error.

pacu · 2026-04-23T19:52:10Z

@schell can you run copilot on this?

Work summary for the gRPC parity test:

stabilized qa/rpc-tests/grpc_comparison.py so it now passes reliably against Zebrad-backed Zainod + Lightwalletd
added a reusable two-stage cache (grpc_comparison_stage1 for the expensive builder-wallet setup, and grpc_comparison for the final Zebrad fixture) so normal reruns avoid rebuilding the chain
fixed the startup ordering so Zainod and Lightwalletd only connect after Zebrad has the restored/replayed chain loaded
isolated Zainod storage per test run to avoid stale DB reuse across runs
normalized known CompactBlock differences between Zainod and Lightwalletd so the parity checks focus on the meaningful shielded contents
documented the final fixture design and the path that led to it in the integration-tests book (doc/book/src/dev/grpc-comparison.md)

Current state:

uv run ./qa/zcash/grpc_comparison_tests.py passes from cache
uv run ./qa/zcash/grpc_comparison_tests.py --fresh regenerates the fixture and passes
the test remains in NEW_SCRIPTS for now

schell · 2026-04-23T19:55:10Z

@zodl-review

zodl-review · 2026-04-23T19:55:48Z

🤖 Claude Review

Summary

Adds a Python-based gRPC parity test suite comparing Zainod and Lightwalletd responses against a shared Zebrad backend, with two-stage caching, a lightwalletd CI build job, and a git-subtree-vendored copy of zcash/lightwallet-protocol v0.4.0 with generated Python stubs committed.

Risk Assessment

Medium — This is test infrastructure (no consensus or wallet-crypto code), so the blast radius is limited to CI. However, several design choices deserve scrutiny before merge:

Binary cache artifacts (zebrad_state.tar.gz, zcashd{0,1}_state.tar.gz) are committed to the repo. These will bloat history, can't be audited by reviewers, and any reproducibility claim is unverifiable. This is the largest concern.
The fixture relies on standalone zcashd (now deprecated per the i-am-aware-zcashd-will-be-replaced-by-zebrad-and-zallet-in-2025 flag embedded in util.py). Lifetime of this test is bounded.
_GRPC_ACTIVATION_HEIGHT = 2 places NU5/NU6 at height 2 while Sapling activates at 1 — consistent with the fixture narrative but worth re-checking against Zebra's regtest defaults before merging.

Key Observations

Good:

Normalization is explicitly documented (protoVersion, transparent-only coinbase vtx). The distinction between _normalize_compact_block (header-only) and _normalize_shielded_compact_block (full vtx with coinbase filter) is well-motivated.
Tests assert both implementations error on out-of-bounds without demanding matching status codes — correctly notes OUT_OF_RANGE vs INVALID_ARGUMENT divergence.
_assert_shielded_block_match asserts len(vtx) > 0 on both sides before comparing — guards against the tautological "both returned empty, therefore equal" failure mode that the rubric flags ("Tautological tests that only assert the change under test does what it was written to do").
Doc page grpc-comparison.md gives a clear "dead ends" narrative useful for future maintainers.
Separate zcashd0/zcashd1 builder split with explicit wallet restarts is a reasonable workaround for note-selection flakiness.

Needs attention:

Rubric violation — "No step-numbers in comments unless they trace a published spec": The Blocks 1–200 / Block 201 / Block 202... narrative in the module docstring and the # 201, # 203 trailing comments throughout _complete_chain_from_stage1 are exactly the "step 1/2/3" maintenance hazard the rubric calls out. Consider replacing magic-number comments with named constants (T_TO_SAPLING_HEIGHT = 201) or deriving from self.*_height fields.
Committed .tar.gz binary caches: these should live in CI cache / an artifact store, not git. A reviewer cannot verify that zebrad_state.tar.gz actually corresponds to the documented chain.
_GRPC_CACHE_VERSION = 7 with no explanation of what 1–6 were — a brief comment on bump policy would help.
test_get_taddress_txids_lower/_upper are labelled backwards: _lower uses height=tip, _upper uses height=1. Either the naming or the ranges are swapped.
test_get_lightd_info skips blockHeight equality — but the comment says it's checked. Re-read: it is asserted. Good, ignore.
.gitignore adds .claude and .codex — reasonable but probably belongs in a separate commit per "Atomic commits" rubric.
authproxy.py content-type fix is a good drive-by but unrelated to the PR scope; consider splitting.

Suggestions

Blocking: Move *.tar.gz caches out of the repo (CI cache / LFS / generated on first run from a pinned zcashd build). Binary blobs in git history are very hard to remove later.
Blocking (minor): Fix the _lower/_upper naming inversion in test_get_taddress_txids_*.
Nit: Replace the step-numbered block comments (# 201, # 203, ...) with named height constants or derive comments from self.*_height, per the rubric.
Nit: _skip_runtime_files is duplicated between _persist_framework_cache and _persist_stage1_cache; hoist to module scope.
Non-blocking. Split the authproxy.py content-type tweak and the .gitignore additions into their own commits per "Atomic commits; imperative specific headlines".
Non-blocking. Add a TODO pointing at how this suite should be re-homed once standalone zcashd is retired — the doc mentions it obliquely but doesn't plan the migration.
Nit: The print("Testing %s..." % label) loop would benefit from pass/fail timing; a flaky parity test is much easier to diagnose with per-check timestamps.

_{Automated review by zodl-review bot · Model: us.anthropic.claude-opus-4-7 · [Silent pilot — feedback welcome]}

Copilot

Pull request overview

Copilot reviewed 28 out of 33 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

scripts/generate_proto.sh:1

sed -i is not portable across GNU/Linux vs macOS/BSD sed (macOS requires an explicit backup suffix, e.g. -i ''). Consider switching to a portable in-place edit (e.g., perl -pi -e ...) or detecting the platform and using the appropriate sed -i invocation so developers can regenerate stubs on macOS without manual tweaks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-23T19:58:39Z

+    def _restore_framework_cache(self, cache_path):
+        with tarfile.open(os.path.join(cache_path, 'zebrad_state.tar.gz'), 'r:gz') as tf:
+            tf.extractall(self.options.tmpdir)
+
+    def _restore_stage1_cache(self, cache_path):
+        self._load_cached_metadata(cache_path)
+        for index in range(2):
+            with tarfile.open(os.path.join(cache_path, 'zcashd%d_state.tar.gz' % index), 'r:gz') as tf:
+                tf.extractall(self.options.tmpdir)


These cache restore paths call TarFile.extractall() directly. The framework already has tarfile_extractall(...) that uses Python 3.11.4+'s safe filter='data' mode; using it here would avoid tar path traversal issues if the cache tarballs are ever corrupted or replaced. Consider importing and using test_framework.util.tarfile_extractall (or duplicating the same safety behavior) for consistency with the rest of the framework.

Copilot · 2026-04-23T19:58:39Z

+            if os.path.exists(wallet_tgz_filename):
+                with tarfile.open(wallet_tgz_filename, "r:gz") as wallet_tgz_file:
+                    tarfile_extractall(wallet_tgz_file, os.path.join(to_dir, "wallet.dat"))


This changes behavior from failing fast when the per-node wallet cache is missing to silently continuing without the wallet. If downstream tests assume a wallet exists for a given persistent cache behavior, they'll now fail later with less actionable errors. If the intent is to make wallet optional only for certain cache behaviors, consider keeping the exception for cache modes that require a wallet, or at least emit a clear warning indicating the wallet cache was absent and the node will start without it.

Suggested change

if os.path.exists(wallet_tgz_filename):

with tarfile.open(wallet_tgz_filename, "r:gz") as wallet_tgz_file:

tarfile_extractall(wallet_tgz_file, os.path.join(to_dir, "wallet.dat"))

if not os.path.exists(wallet_tgz_filename):

raise Exception('Wallet cache missing for cache behavior %s, node %d' % (cache_behavior, i))

with tarfile.open(wallet_tgz_filename, "r:gz") as wallet_tgz_file:

tarfile_extractall(wallet_tgz_file, os.path.join(to_dir, "wallet.dat"))

daira · 2026-04-23T20:07:59Z

+- Zebrad and standalone `zcashd` do not build the fixture chain together over
+  P2P in this harness, so the test submits raw blocks explicitly.
+- Zainod must only start after Zebrad has loaded the full chain state, or it can
+  fail during initial indexing.


That seems like a bug in Zaino. It should wait for Zebra.

Move grpc_comparison.py back into NEW_SCRIPTS now that the parity test is stable enough for per-change CI. Also make --fresh skip the reusable stage-1 wallet cache so fresh runs rebuild from scratch instead of restoring a zcashd wallet cache that can leave wallet operations disabled while reindexing.

pacu · 2026-04-28T18:19:49Z

Updated this PR with the latest fixes from the gRPC comparison work:

Re-enabled grpc_comparison.py in the NEW_SCRIPTS set so it runs with the new tests on per-change CI.
Removed it from EXTENDED_SCRIPTS to avoid duplicate classification.
Fixed --fresh behavior so it now skips the reusable stage-1 wallet cache and rebuilds the full fixture from scratch.
This avoids restoring a cached standalone zcashd wallet state that can leave wallet operations disabled while reindexing.
Verified the rebuilt Zaino branch against the integration test:
- timeout 600s uv run ./qa/zcash/grpc_comparison_tests.py --fresh passes.
- timeout 600s uv run ./qa/zcash/grpc_comparison_tests.py passes.
Confirmed grpc_comparison.py is now present in NEW_SCRIPTS and absent from EXTENDED_SCRIPTS.

The cached path remains fast, and the fresh path now behaves as expected by fully rebuilding the fixture instead of resuming from a potentially stale stage-1 wallet cache.

All current issues are solved when this PR zingolabs/zaino#1067 lands in Zaino

pacu and others added 3 commits April 7, 2026 19:54

Squashed 'lightwallet-protocol/' content from commit 23f0768ea

2141976

git-subtree-dir: lightwallet-protocol git-subtree-split: 23f0768ea4471b63285f3c0e9b6fbb361674aa2b

Merge commit '214197623fb5c8148966be5f7ebe30c784bb12b5' as 'lightwall…

a0440fe

…et-protocol'

schell requested a review from Copilot April 8, 2026 01:51

Copilot started reviewing on behalf of schell April 8, 2026 01:52 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

pacu marked this pull request as draft April 10, 2026 00:33

pacu force-pushed the grpc-comparison-tests branch from 9222c97 to 91b3dca Compare April 14, 2026 15:32

pacu added 4 commits April 23, 2026 12:00

Add cached gRPC parity fixture for Zebra-backed comparison

034b52b

Refactor grpc comparison test flow for readability

db0fa82

add .claude and .codex to gitignore

eebe565

Document how the gRPC comparison test was stabilized

f0b873e

pacu marked this pull request as ready for review April 23, 2026 19:51

pacu requested a review from Copilot April 23, 2026 19:52

Copilot started reviewing on behalf of pacu April 23, 2026 19:53 View session

Copilot AI reviewed Apr 23, 2026

View reviewed changes