DB optimizations, cross-event enricher, new endpoints for Grafana& dashboards#15
Open
michalkucharczyk wants to merge 122 commits into
Open
DB optimizations, cross-event enricher, new endpoints for Grafana& dashboards#15michalkucharczyk wants to merge 122 commits into
michalkucharczyk wants to merge 122 commits into
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
…id full table scans
…ng tables Adds 10 migrations (004-013): - Hot columns on events (slot, core, submission_id) with partial indexes - 30s continuous aggregate + hierarchical rebuild of 1m/1h chain - event_services junction table with gas_used + service_stats_1m aggregate - core_stats_1m aggregate from events WHERE core IS NOT NULL - node_stats table (Status event extraction) + node_stats_1m aggregate - slot_convergence table for pre-computed block propagation stats - wp_tracking table for unique work package pipeline tracking
…ipeline - src/enricher.rs: NodeEventEnricher with cross-event correlation (core, service_ids, wp_hash) via submission/built/sending/request/reconstructing chain lookups. Per-node state with 60s TTL, 10K hard cap per map. - src/events.rs: Add slot(), core(), submission_id(), service_ids(), work_package_hash(), gas_per_service_item() field extractors. - src/server.rs: Fix EventId assignment (post-increment per JIP-3, Dropped advances by num not 1). Plug enricher between decode and db_batch.push. - src/batch_writer.rs: EventRecord tuple -> struct with enriched fields. flush_events now also writes to event_services and node_stats tables. - src/store.rs: Binary COPY expanded from 5 to 8 columns (slot, core, submission_id). Added store_event_services_batch and store_node_stats_batch COPY functions.
11 optimized endpoints under /api/grafana/* with automatic aggregate table selection (30s/1m/1h), time-bucketed timeseries, per-core/service stats, block convergence, node stats, DB metadata, and WP pipeline bottleneck analysis.
…racking SlotTracker: in-memory per-slot block propagation convergence with 3-phase lifecycle (10s initial flush, straggler UPSERTs, 60s eviction). Computes p50/p99/p100 percentiles from node timestamps. WpTracker: in-memory work package pipeline tracking through 7 stages (received→authorized→refined→report_built→guarantee_built→distributed, plus failed). Cross-node dedup via received_by counter. UPSERT flush every 5s with COALESCE/GREATEST merge. Both trackers wired into connection handler after enricher, with periodic flush task spawned in main.rs.
Update tart-global, tart-cores, and tart-node dashboards to use /api/grafana/* endpoints with start/end ISO 8601 time range params. Remove dropped panels (Health Score, raw events tables). Add 3 new dashboards: tart-grafana-services (per-service gas/WP stats), tart-grafana-blocks (block contents, convergence, pipeline), and tart-grafana-da (DA storage, peer counts, preimage stats).
Endpoint catalog for all 11 /api/grafana/* endpoints with query params, response schemas, and example curl commands. Dashboard recipes for 6 dashboards, Grafana variable templates, time range handling, and aggregate retention awareness.
psql runbook for verifying schema, data flow, compression, retention, and performance after deployment. Covers hot columns, continuous aggregates, hypertable sizes, and active query monitoring.
- Fix 1: grafana_stats merges LiveCounters (events/blocks per sec, slots, active nodes) - Fix 2: enricher cleanup task sweeps stale entries every ~2.5min - Fix 3: 8 enricher unit tests (WP chain, guarantee chain, segment chain, etc.) - Fix 4: 3 batch_writer unit tests (BlockExecuted gas, Authorized first-only, Status parsing) - Fix 5: retention-aware aggregate auto-select (30s→1m after 3d, 1m→1h after 30d) - Fix 6: dashboard group_by=node → group_by=node_id - Fix 7: wire /wp-funnel route in grafana.rs - Fix 8: rename misleading pg_micros → unix_micros in store.rs - Fix 9: return 400 for validation errors via map_sqlx_error helper
- 12 new integration tests covering timeseries, stats, cores, blocks, services, nodes, node-stats, db-stats, bottlenecks, and wp-funnel - Fix grafana_db_stats: hypertable_detailed_size() doesn't return hypertable_name column; chunk_compression_stats() needs aggregation - Add encode/encoded_size for WorkReportBuilt and GuaranteesDistributed - Add flush_trackers() to TelemetryServer for test use - Extend cleanup_test_data to truncate all pipeline tables - Add shared test helpers: event constructors, refresh_aggregates, flush_all
- Split event 105 (GuaranteeBuilt) into own match arm to increment guaranteed_by counter (was always 0) - Replace HashMap + cap_map(.clear()) with schnellru::LruMap for enricher chain maps — preserves correlations under load - Add per-node deduplication for received_by and guaranteed_by using HashSets to prevent double-counting on node reconnect - Add "node" as alias for "node_id" in group_by validation
LRU was over-engineered for this FIFO access pattern. Revert to HashMap with raised cap (10K → 50K) and partial eviction (oldest 25%) instead of the previous destructive .clear(). The existing TTL-based evict_stale (60s, every ~0.3s) keeps maps bounded in practice; the cap is just a safety valve for extreme bursts.
…WorkPackageFailed - Parameterize flush_slot_tracker age thresholds (was hardcoded 10s) so tests can bypass the age gate with Duration::ZERO - Add flush_trackers_for_test() to server for immediate test flushing - Add encoding for WorkPackageFailed event (was todo!() panic) - Add 7 unit tests for slot_tracker (SlotState, convergence) - Add 7 unit tests for wp_tracker (stages, idempotency, failure) - Add 8 Grafana integration tests (services, group_by core, convergence, wp failure pipeline, node-stats, cores detail, node disconnect, filters) - Fix wp_received_event() to embed submission_id in work_package_hash for unique DB keys across test work packages
- Demote per-flush batch_writer logs to TRACE (was spamming DEBUG) - Add periodic flush stats to batch_writer workers (events/s, avg/max flush time) at DEBUG level, logged every 5s - Promote MetricsTracker lifecycle messages to INFO (happen once) - Add wp_tracker flush summary at DEBUG (flushed/evicted/active) - Add enricher partial eviction debug log (before/after/removed) - Enable tower_http at DEBUG level for HTTP request/response logging
… path bugs Add authoring_ids/importing_ids correlation maps to propagate slot to Authored, AuthoringFailed, BlockVerified, BlockVerificationFailed, BlockExecuted, and BlockExecutionFailed events. Fix grafana_blocks_contents JSONB paths to match serde external tagging and correct field names.
…race Per-flush DB logs (COPY completed, batch committed, stats cache updated, node connections) and aggregator stats were flooding debug output. Demote to trace so debug level shows only periodic summaries.
…tion formatting Replace 8 per-worker debug lines (every 5s) with one aggregate line (every 10s) using shared atomics. Durations now use Rust's Duration display (14.183ms) instead of raw microseconds.
…ker entry creation - Add optional event_type query param to /api/grafana/blocks/convergence for per-phase convergence panels (Best Block, Importing, Finalization) - Fix slot_tracker to create entries for non-Authored events (11/12/40/43) that arrive before the Authored event for a slot - Add test assertions for event_type filtering on convergence endpoint
…d last_activity to /grafana/cores (Phase 0) Events endpoint: event_types now optional (returns all types when omitted), add node/core/wp_hash filters (using hot columns), add offset param, response wrapped in EventsSearchResponse with PaginationMeta (offset, limit, total, has_more). Add created_at field to EventRow. Cores endpoint: add last_activity field to CoreSummary via correlated subquery on wp_tracking (MAX(first_seen) per core). NULL for cores with no WP activity.
…ssed chunks Remove /api/analytics/block-propagation and /api/metrics/timeseries/grouped routes — verified unused by any UI component. Delete corresponding tests. Fix test isolation: use DELETE FROM (in DO block) instead of TRUNCATE TABLE for test cleanup. TimescaleDB compressed chunks were not cleared by TRUNCATE, causing stale data leaks between test binaries.
New endpoints: /grafana/failure-rates, /grafana/sync-timeline, /grafana/connections-timeline, /grafana/guarantees, /grafana/guarantees/by-guarantor, /grafana/wp-stats, /grafana/validators/cores, /grafana/network-health. Shared node_core_mapping() helper for guarantor/validators endpoints using guarantee_convergence table (observed behavior, not protocol assignment — documented in utoipa annotations). Each endpoint has: utoipa docs explaining question answered, data source, algorithm; documented response structs with field-level comments; integration test (inject events via TCP → flush → query → verify).
…ore metrics
Migration 021: add node_id, refine_gas_used, failure_reason, discard_reason
to wp_tracking + partial index for active WP queries.
Ingestion: populate node_id (event 94), refine_gas_used (event 101 costs sum),
failure_reason (event 92 reason field) in wp_tracker.
New endpoints:
- /grafana/wp-active: in-flight WPs with full envelope (summary, reached
funnel, stage duration percentiles, failure breakdown)
- /grafana/wp/{hash}: pipeline summary + raw event drilldown (1h retention)
- /grafana/wp/batch: multi-hash WP summary lookup
- /grafana/blocks/summary: block production totals + per-node authoring
- /grafana/cores/{id}/metrics: efficiency, latency percentiles, gas
Each with utoipa docs and integration tests.
Step 0 fixes from review:
- New /grafana/cores/{id}/validators endpoint — per-core guarantor list
from guarantee_convergence + nodes JOIN. Replaces legacy guarantors +
guarantors/enhanced. Utoipa docs with coverage limitation caveat.
- Register 32 Phase 0/3/4 response types in GrafanaApiDoc OpenAPI schema
- Add 6 gap-filling tests: events node/core filters, no-event-types
returns all, cores last_activity, wp/batch POST, core validators
…d/failed to reached counts /grafana/wp-active was filtering WHERE distributed_at IS NULL AND failed_at IS NULL which returned 0 WPs (pipeline completes in <15ms on testnet). Legacy endpoint had no such filter — returned all recent WPs. Remove filter to match legacy behavior. Add distributed and failed to WpReachedCounts (UI pipeline funnel reads these). Update test assertions and utoipa docs.
…hase 5) New endpoint queries ingested_raw_events (1h retention) with JSONB extraction for three execution phases: authorization (Authorized), refinement (Refined), accumulation (BlockExecuted). Returns per-phase stats (count, total_gas, avg_gas, avg_time_ns) plus per-service gas breakdown from accumulation.
…ces at ingestion The /grafana/execution endpoint was querying JSONB from ingested_raw_events (1h retention) at query time. This moves timing extraction to ingestion time in the event_services table (7-day retention), eliminating expensive JSONB cross-joins and extending history from 1h to 7 days. - Migration: add elapsed_ns, load_ns columns to event_services - Batch writer: extract timing for types 47, 95, 101 alongside gas - Store: expand COPY binary format from 5 to 7 fields - Endpoint: rewrite grafana_execution from 4 JSONB queries to 2 indexed queries - Response: add avg_load_ns to ExecutionPhaseStats and ServiceExecutionRow - Tests: unit tests for timing extraction, e2e test with precise value assertions
if_not_exists silently kept the old 7-day policy from migration 001, causing the table to consume excessive disk space.
…on, refinement, accumulation) The by_service array in /grafana/execution previously only included accumulation (type 47) data. event_services already has per-service rows for refinement (type 101) and authorization (type 95), so this expands the query to include all three phases with a new `phase` field on each row.
Continuous aggregates (_1m, _1h) only had TimescaleDB's default bucket index. Grafana queries filtering by event_type were doing sequential scans after bucket range narrowing — ~800ms on 1024-node DB for 24h queries. Adds (event_type, bucket DESC) on all 28 aggregate tables, plus (service_id, bucket), (node_id, bucket) on service/node aggregates, (first_distributed_at) on assurance_convergence, and partial core indexes on raw count tables for core-filtered queries.
New endpoints /api/grafana/validator-profiling and /api/grafana/validator-profiling-timeseries that answer "which nodes are slow or failing?" by computing per-node average latency at each WP pipeline stage from wp_tracking. Includes slowdown_factor (node avg vs network avg) and failure_rate. No new tables or migrations needed.
All UI migration phases are complete — the frontend now uses /api/grafana/* endpoints exclusively. Remove 30+ dead legacy routes from api.rs, 31 store methods from store.rs (~6700 LOC of heavy JSONB SQL), the --disable-legacy-endpoints CLI flag, legacy cache warming, and 43 associated tests. Surviving endpoints (node details, network, metrics, jam, health, websocket, grafana) remain intact.
At 1024-node volume, 1h retention with daily schedule accumulates hundreds of GB before cleanup. 5-minute schedule keeps disk usage bounded at ~5 minutes of ingestion.
…ort params, Grafana panels
- Response is now { network_avg_total_ms, nodes: [...] } for baseline rendering
- `limit` param caps returned nodes (network avg still reflects all)
- `sort=asc|desc` controls order (default desc = slowest first)
- Grafana Nodes dashboard: top 10 slowest + fastest side-by-side barcharts
- Removed Ticket Activity panel, changed Selected Events to stacked bars
…tor pipeline - Add WHERE (distributed_at IS NOT NULL OR failed_at IS NOT NULL) to only include completed WPs — fixes sort=asc returning null avg_total_ms nodes - Fix null sort ordering (nulls always last regardless of direction) - Rename all labels/docs from "validator pipeline" to "guarantor pipeline" since wp_tracking tracks the guarantor-side processing path
…hing, and preimage transfers Adds histogram-based latency tracking for the three DA subsystems that previously only had volume counting. Follows the proven da_tracker pattern with LatencyFlow helper struct, 23-bucket CONVERGENCE_BOUNDS histograms, and 10s periodic flush. Tracked flows per subsystem: - Bundle (140-153): shard req/resp, full req/resp, reconstruction CPU, e2e recovery - Segment (160-178): shard req/resp, full req/resp, reconstruction CPU - Preimage (190-199): requestor, responder New endpoints: /api/grafana/bundle-latency, /segment-latency, /preimage-latency New tables: bundle_latency_hist, segment_latency_hist, preimage_latency_hist New dashboard sections: 3 rows, 6 panels in tart-grafana-da.json
The network uses the full bundle path (148→153) rather than the shard path (140→145), so the panels showed empty. Updated to display both paths so whichever the network uses will appear.
…timing Failed WPs produce meaningless timing (near-zero or negative due to clock skew) that pollutes "fastest guarantors" charts. Add FILTER (WHERE distributed_at IS NOT NULL) to all 6 stage AVG columns in both summary and timeseries endpoints. wp_count/failures/failure_rate still count all completed WPs.
slowdown_factor was being rendered as a second bar series on the same axis as avg_total_ms, creating confusing tiny bars at the base.
… dashboard Duplicates the Pipeline Total panel from tart-cores.json but without core filter — shows network-wide guarantor pipeline latency (P50/P95) over time via /bottlenecks-timeseries endpoint.
…ry debugging Individual --disable-* flags for each subsystem (enricher, slot-tracker, wp-tracker, convergence, da-tracker, event-counter, ws-broadcast, db-writes, metrics-tracker, onchain, cache-warmer) plus --disable-all-trackers and --ingest-only convenience combos. All flags support env vars via clap.
michalkucharczyk
force-pushed
the
mku-grafana-endpoints
branch
from
31619ac to
0a45eb2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a complete observability layer to TART: a cross-event enricher that correlates context across the JAM pipeline, 16 Grafana HTTP API endpoints for JIP-3 telemetry, 9 on-chain endpoints from JAM RPC, database schema extensions for aggregation and tracking, 10 provisioned Grafana dashboards, and query performance improvements.
1. Query performance improvements
These commits sit between #14's ingestion work and the Grafana endpoints — they make the existing REST API viable under load:
MetricsTracker— replaces heavy self-JOIN queries for real-time stats with a rolling-window in-memory trackerLiveCounters— replaces heavy real-time endpoint queries with atomic counters updated at ingestion timecreated_attotimestamp(the TimescaleDB partition key) to avoid full table scans2. Database schema & aggregation
events:slot,core,submission_id— populated at ingestion, eliminating JSONB queries for Grafanaevent_stats_30s,event_stats_1m,event_stats_1h,core_stats_1mwp_tracking(work-package pipeline),slot_convergence(block propagation timing)event_services(service × event with gas)node_stats— extracted Status fields per nodeonchain_cores,onchain_services,onchain_validatorsfor RPC-sourced data3. Enricher (
src/enricher.rs)Per-node stateful correlation engine:
core,service_ids,submission_id,wp_hash,slotfrom root events (WorkPackageReceived) to ~30 downstream eventsMetricsTrackerguaranteed_byfield mapping, per-node dedup, JSONB path extraction4. Grafana API — JIP-3 telemetry endpoints
16 endpoints under
/api/grafana/(src/grafana.rs,src/grafana_store.rs,src/grafana_types.rs):/timeseries/stats/cores/summary/cores/{core}/blocks/convergence/blocks/contents/services/services/timeseries/nodes/nodes/stats/nodes/stats/aggregate/db-stats/bottlenecks/wp-funnel/events/event-typesKey features:
$__intervalvalues snapped to nearest valid bucket width (6s-aligned sub-minute support){a,b,c}variable expansion forevent_typesandserviceparamsservice_idas hex — consistent0x...formatting viaDbServiceIdnewtype/api/docs/openapi.json5. Grafana API — on-chain endpoints
9 endpoints under
/api/grafana/onchain/(src/onchain_stats.rs,src/onchain_store.rs,src/onchain_types.rs):Data sourced from JAM RPC (not JIP-3 telemetry). Polling ingestion stores snapshots in
onchain_*tables; derived metrics (gas rates, throughput) computed at query time.6. Trackers
src/slot_tracker.rs): Per-slot convergence — when each node first reports each block event. Computes min/max/median convergence times. Parameterized flush interval.src/wp_tracker.rs): Work-package pipeline stages (received → built → guaranteed → audited → reported) with per-stage timing.Both flush periodically to their respective DB tables and log stats.
7. Event type metadata (
src/event_type_meta.rs)/api/grafana/event-types, used by dashboards for variable dropdowns and legend labels8. Grafana dashboards (10 provisioned JSONs)
All use the Infinity plugin (JSON mode) pointed at
tart-backend:8080:Dashboard features:
$node,$core,$service,$intervalwith chaining[JIP-3]and[On-Chain]labels on all panel titles to distinguish data sourcestart-playground.json10. Tests & docs
tests/grafana_tests.rs, ~1,486 lines): E2E coverage for all Grafana endpoints — inserts events, verifies aggregation, checks response shapesdocs/grafana-guide.md(endpoint specs, query types, dashboard recipes, aggregate retention),/api/docs/openapi.jsonTest plan