Proposal: adopt Knitweb (gither/pulse/p2p-oracle) for numerai-crypto-signals

[febuz]

Title: Proposal: numerai-crypto-signals as Knitweb's first real consumer (pulse fabric + gither + coverage oracle + vbank)

Proposal: adopt the Knitweb stack in numerai-crypto-signals, and make pulse a better host for consumers like it

Hi pulse maintainers — this is a concrete adoption proposal, not a wishlist. The Numerai Signals + Crypto project now has three working integrations against the Knitweb repos, two of them with real Python import edges (pulse) and one as a CLI/metadata mirror (gither), all with verified P2P round-trips. I'd like to (a) formalize numerai as Knitweb's first production consumer and (b) propose a small set of pulse-side improvements that would unblock the next step for any consumer with the same shape.

I've been careful below to separate what was actually engaged/verified today from what is documented but not yet wired in.

---

What's already working (engaged + verified today)

Knitweb repo	Use-case in numerai	What was actually run	Maturity
pulse (knitweb.fabric.feed, knitweb.p2p, core.crypto, core.canonical)	Signed integer-only data-coverage Feed: which (date,ticker) OHLCV we hold vs. don't, up to the null frontier	p2p_data_coverage_oracle.py --p2p-proof → feed root 987dd860…, 267 entries, 70,885 tickers (equity 34,756 / crypto 2,147 / other 33,982); 2nd loopback peer synced all 267 entries, root_match=True	Shipped + import-edge proven, cron-driven
pulse (knitweb.gateway.App + fabric.items)	Publish research proofs as content-addressed, anchored knowledge nodes	publish_proof_to_fabric.py wove 2 microstructure proofs (equity NW-t 6.88, crypto NW-t 9.65) → CIDs bafyreibadz… / bafyrealyj…, 20 nodes / 8 typed edges, OriginTrail UAL did:dkg:knitweb/bafyreigef6… (verified=true, dev notary key)	Runnable + verified, MVP fabric
gither	P2P-git identity/provenance mirror of the (private) competition repo	peer / rad-id / sigrefs / announce → RID rad:zABt864…, DID did:key:z6MkiFGG…, 2 refs signed+verified, loopback peer re-verified, 1-byte tamper rejected	Metadata layer complete + tested; object replication NOT built

All three are CPU-only, integer-canonical, no synthetic data, and leave the GitHub remote untouched.

---

Why this matters to pulse (the consumer's perspective)

1. pulse's signed-feed + canonical-CBOR primitives held up against a real, messy dataset. A 70,885-row coverage map (epoch-days / counts / staleness / severity, all integers) round-tripped through core.canonical.encode and fabric.feed byte-identically, with a working secp256k1 head sig and pls1… address. The integer-only constraint was the right call — it forced clean encoding (scale t-stats → centi-t, IC → bps) and there were zero float-rejection surprises once we scaled at the boundary.
2. The gateway weave is genuinely usable as a verifiable sibling to gitnexus/LightRAG. App.attest → link → anchor gave deterministic CIDs + an external UAL in ~15 lines of glue. That's a real upgrade over our flat-file proof stores for tamper-evidence and provenance (subject to the dev-notary caveat in ask SDK facade — Wallet + end-to-end synaptic demo #4 below).
3. numerai is a high-signal first consumer. It exercises the feed path, the gateway/weave path, AND (via gither) the identity/refs path — so it surfaces integration friction across most of the stack at once.

---

Repo-by-repo: use-case → benefit → adoption step

pulse — data-coverage feed (PROVEN)

• Use-case: broadcast/receive verifiable "what data exists where, up to the null frontier" without a central server.
• Benefit: peers can prioritize gap-filling (the feed already carries a vbank poll collect-priority ranking stalest-active targets, e.g. crypto CREAM at 727d stale); bounded gossip (manifest + 5 gaps + top-256 targets + live-dataset coverage; the full 70,885-ticker map stays in parquet).
• Adoption step (already taken): scripts/p2p_data_coverage_oracle.py runs under cron (p2p_coverage_tick.sh, every :05/:25/:45); emits reports/p2p_data_coverage_oracle_latest.json + signed feed.
• Honest freshness caveat: per-ticker OHLCV coverage is derived from reports/ticker_data_ranges_latest.csv (last regenerated ~Jun 11), so the most recently built OHLCV stores are not yet reflected in the per-ticker frontier. The 4 auxiliary live datasets (funding/OI, article-assets, earnings, ownership) are freshly scanned each run. Making the per-ticker OHLCV frontier current requires regenerating that CSV from the price store first (a separate heavier job, intentionally out of scope of the per-tick run).

pulse — proof publishing (VERIFIED)

• Use-case: research proofs (feature RMSE uplift, shuffled-control results) as content-addressed knowledge nodes.
• Benefit: immutable CID per proof (any edit changes the CID), signed authorship, OriginTrail UAL, typed weighted edges — the run used relations proves / supports / passed / applies-to — so a Lens can rank by evidence weight.
• Adoption step (already taken): scripts/publish_proof_to_fabric.py (App.attest/link/anchor path — deliberately NOT the local-only page publish CLI).

gither — P2P-git provenance mirror (METADATA LAYER VERIFIED)

• Use-case: location-free, tamper-evident identity for a private competition repo (mitigates single-host lockout).
• Benefit: one content-addressed RID across mirrors; per-peer namespaced signed refs; threshold-delegate identity so no single peer can rewrite control or forge/rollback branches.
• Adoption step (already taken): .gither/identity/{node.key,rad.json,sigrefs.json} minted (RID rad:zABt864…, 1-of-1 delegate threshold); .gither/ gitignored so the private key can't be committed.
• Honest gap: gither moves signed JSON metadata only — no daemon, no discovery, no git object replication (a grep of the P2P source finds no pack-objects/upload-pack). Fetching objects still needs git/Radicle. So this is verifiable provenance today, not a content-replicating mirror.

vbank (+ personhood) — sybil-resistant prioritization (DOCUMENTED, partially wired)

• Use-case: one-peer-one-vote ranking of which data gap to collect next; future sybil-resistant feedback/staking-signal layer.
• Benefit: the oracle already emits a vBank poll structure; vbank.tally() counts integer ballots, personhood gates each vote on a revocable zero-PII ticket.
• Honest status: the poll is defined and emitted, not yet tallied across real peers. Personhood is phase-1 trusted-RP (RP sees an ephemeral holder secret at admission); trustless uniqueness is the gated ZK phase-2. State this honestly — don't over-claim sybil resistance yet.

Documented-only (not wired into numerai): lens (cited/abstaining query over the pit — promising, not yet imported), monitor (feed/peer observability), knit publish/inbound gates + KG viz (safe two-way sharing). Worth doing, but no import edge exists today.

---

Asks of pulse, to better serve consumers like numerai

These are the concrete friction points the integration hit. In rough priority:

1. A productized fabric transport / discovery story. Today the real weave (App.attest/weave) only reaches statically-configured listen=/peers=. The transport primitive exists but there's no discovery/DHT and no long-running listener. A pulse node serve (even loopback-first, then bootstrap-seed list) would turn "internal/federated graph" into actual federation. This is the single biggest unblocker.
2. Clarify page publish vs. the gateway path in docs. The CLI page publish is local-only (writes ~/.pulse/pages, no peer propagation). It's easy to assume it federates. A doc note + a --fabric flag that routes through App.attest would prevent every new consumer re-discovering this.
3. Ship a Numerai-style consumer example. examples/ is mvp/pouw/synaptic/vbank/crowdfunding only. A ~20-line "publish a proof: KnowledgeItem + AttentionRecord + anchor" example (I can contribute the glue from publish_proof_to_fabric.py) would document the integer-scaling-at-the-boundary pattern that tripped us up first.
4. Verify the OriginTrail anchor backend before external reliance. anchor() currently uses a fixed dev notary key (_NOTARY_PRIV) + OriginTrailAnchorBackend. Consumers need a clear statement of whether the UAL hits a real DKG endpoint or is dev-only, so we don't externally cite a dev UAL. (The UAL above was produced with the dev notary, so we treat it as internal-only for now.)
5. Cross-repo: align gither's object-replication roadmap with the fabric. Once gither has object replication (or a documented Radicle rad-remote path), a private numerai repo could have public provenance + restricted object sync — a clean reference pattern for any private-data consumer.

---

Suggested next concrete step

Open a tracking issue "numerai-crypto-signals: first production consumer of Knitweb" linking: the cron'd coverage oracle, the proof-publish script, and the gither mirror — then take asks #1 and #2 as the first two pulse PRs. I'm happy to contribute the consumer example (#3) and the integer-boundary doc.

🤖 Generated with Claude Code

[febuz]

E11 proven: crypto target distillation (t20→t60) — feedback loop to fillslava

Closing the loop from the Knitweb proposal above: one of the experiments that ran today using the pulse-anchored proofs as evidence produced a genuine, deployable result. Sharing it here so fillslava/ClaudeClaw can absorb it into the knowledge fabric.

What was proven

Formulation backlog rank 11 — crypto target distillation (20→60d)

Instead of blending predictions from two separately-trained models (the pure horizon-blend, which was REJECTED, NW-t=−2.88), we blend the training targets:

y_α = (1 − α) × era_rank(target_20) + α × era_rank(target_60)

Train ONE LGB on y_α, evaluate OOS vs scored target_20.

α	Δ mean CORR/era	Pooled NW-t	seed-stable (3/3)?
0.10	+0.0013–0.0016	2.45–3.06	✅
0.15	+0.0020–0.0025	2.67–3.30	✅
0.20	+0.0025–0.0031	2.46–3.22	✅
0.25	+0.0029–0.0039	2.32–3.14	✅

Winner: α = 0.1 (most conservative, all alphas 0.1–0.25 seed-stable). Gate: NW t ≥ 2 AND seed-stable {17, 42, 7} — passes.

Runtime: 97s on CPU, 150k rows, 500 eras, 22 features. Proof: reports/formulation_proofs/proof_crypto_target_distill.json (commit ea3a2b9).

Why this matters for the Knitweb feedback loop

This is the model-learning signal the pulse fabric was proposed to anchor and propagate. The insight — regularise a noisy short-horizon target with a longer-horizon label during training — is portable: it applies anywhere you have a noisy scored target and a correlated slower-moving ground-truth. Any peer (fillslava or otherwise) running a similar tournament-prediction workload could apply the same pattern.

Proposed next knowledge-fabric entry

publish_proof_to_fabric.py \
  --proof reports/formulation_proofs/proof_crypto_target_distill.json \
  --label "model-formulation:target-distillation" \
  --tags crypto,target_engineering,rank11,nw_t_3.06,seed_stable

Previous proofs already anchored: microstructure equity (CID bafyreibadz…) + crypto (CID bafyrealyj…). This would be the third fabric entry, first from model-formulation rather than feature-engineering.

---

cc @febuz — the formulation came from the internal backlog workflow (not Knitweb), but the proof infrastructure (signed feed, fabric anchor, gither mirror) is what makes it trustworthy and shareable across peers.

[febuz]

Session feedback — 2026-06-22 final summary (for fillslava / knowledge fabric)

Full session work complete. Sharing findings for fabric anchoring.

What was proved and deployed today

DEPLOYED (crypto, round 1295+):

• E11: crypto target distillation α=0.1 — train on y_α = 0.9×era_rank(t20) + 0.1×era_rank(t60) instead of raw t20. Genuine CORR lift +0.0013–0.0016/era, NW-t 2.45–3.06, seed-stable. Commit bf3567f.

READY FOR OWNER DEPLOYMENT (equity Signals, owner-gated):

• ts_abs_energy (60d Σreturn² = realized total variance over 60 days) — the one feature from the tsfresh family that individually passes the gate (NW-t=2.36, p=0.018). Pre-deploy steps 1–4 all done: ablation ✅, forward holdout ✅ (no regime drift in 2024+), MMC check ✅ (CORR augmentation path, pred_corr=0.794), live-computability ✅. --use-tsfresh-abs flag wired into make_live_submission.py. Owner decision needed for champion_promote.py RMSE gate run.

What was proved NULL (17/17 formulation experiments exhausted)

Ran all 17 model-formulation experiments from the backlog (blend weights, neutralization, target engineering, era-weighting, regime specialists, stacking). Only rank-11 was a winner. Everything else failed the t≥2 seed-stable gate. Key patterns:

• Target blending of different signals targets: no MMC diversity (tffn predictions identical to primary target)
• OLS orthogonalization: no MMC improvement
• 2-stage stacking: HARMFUL (pooled t=−2.2)
• Era-boost / regime clustering: no CORR improvement over random baseline

What was REFUTED (critical safety check)

Crypto microstructure vol/illiquidity (E3) = SURVIVORSHIP ARTIFACT confirmed.
E3 apparent signal (NW-t=9.47, full 2017–2026) completely disappears in the 2022–2026 stationary window: pooled NW-t=−0.031, real_lift=−0.00043. The high-t in the full window was entirely carried by 2017–2021 eras where dead/delisted coins were structurally absent. Do NOT deploy crypto microstructure features. (Equity microstructure E1, NW-t=6.88, is unaffected — different universe mechanics.)

For the knowledge fabric

Three lesson types relevant for any tournament-prediction workload in the Knitweb network:

1. Survivorship gate: any feature that exploits Vol/Illiquidity on a crypto universe must test on a 2022+ stationary window before claiming a signal. The 2017–2021 data systematically over-states any vol-related signal for crypto.

2. Target distillation: blending a longer-horizon target (t60) into the training objective at a small weight (α=0.1) is a cheap, proven, regime-robust improvement. The longer horizon regularises the loss surface without committing to predicting t60.

3. Feature-count capacity: when real-lift ≈ shuffled-lift in a CPCV proof, it's almost always a capacity artifact. The only test that counts is real − shuffled under a proper control (ticker-within-date shuffle for cross-sectional targets).

Fabric entries proposed: (a) ts_abs_energy feature proof (CID for the ablation JSON), (b) crypto microstructure survivorship correction (updates/supersedes the E3 CID bafyrealyj…), (c) crypto target distillation deployment fact.

[febuz]

Final session summary 2026-06-22 (for fillslava knowledge fabric)

Comprehensive sweep complete. Key findings beyond the E11 update:

Crypto microstructure — DEFINITIVELY survivorship artifact (2 independent proofs)

1. Our 2022+ stationary window (120 symbols): pooled NW-t=−0.031 (null)
2. Node1's E3b PIT-reduced (2022+ dense window, smoke): real IC=−0.021 (NEGATIVE), shuffled=+0.014, shuffled BEATS real, NW-t=−1.10

The E3 NW-t=9.47 was 100% from 2017-2021 eras where dead/delisted coins were absent. The microstructure feature is slightly harmful on a proper sample. DO NOT deploy crypto microstructure.

SHA-256 reversible circuit — 28.2% improvement (sha256.fail)

Repository updated with measurement-free optimizations (borrowed-XOR Ch/Maj, W[0..15] aliasing):

• Score: 1,565,985,280 → 1,123,699,200 (Toffoli: 215,552 → 166,400, Qubits: 7,265 → 6,753)
• Published on Radicle: rad:zH9RjGdY5sAmF8oM6Kpdq7vbvkxg
• Measurement-free ceiling confirmed; further ~50–70% improvements need Op::Measure in circuit format

Process learning cleanup

• BTC features (btc_ret/vol/drawdown): rejected (cross-poll NULL/HARMFUL)
• Yiedl/crypto-dispersion: re-confirmed rejected (genuine t<1, shuffled control)
• Base momentum/RSI/TRIX: marked GENUINE_CORR_BUT_BASE_FACTOR (already in v2.1, not new additions)
• Evidence backlog: clean (no more stale run_cpcv actions)

For fabric anchoring

All findings pushed to febuz/numerai-crypto-signals (36 commits, PR #1). Radicle node running and seeding 17 repos. ts_abs_energy deployment path fully wired (blocked on 6-month forward holdout); sha256.fail on Radicle.

[febuz]

Proposal accepted