Iterative Validation for Large Datasets

[joneubank]

Motivation

The @overture-stack/lectern-validation package currently requires every record across all related schemas to be fully loaded into memory before validation can begin. For large datasets this becomes a hard limit on the size of data sets that can be validated. The problem is compounded by the fact that TSV data is expanded into JavaScript objects before validation runs, so actual memory usage significantly exceeds raw file size.

Solution

Introduce a streaming validation API: stateful validator objects that accept records one at a time, validate each record immediately as it is submitted, and maintain only the information required to detect schema- and dictionary-level constraint violations across the full dataset. Once all records have been submitted, a final report is produced. At no point is the full dataset held in memory.

Field- and record-level validation (data types, required fields, codeLists, regex) is stateless — existing validation functions already handle this correctly and run immediately on each submitted record. The memory challenge comes from cross-record and cross-schema constraints (unique, uniqueKey, foreignKey), which by definition require information about more than one record. To support these in a streaming context, two new low-level index-only components are introduced:

• CrossRecordValidator - builds a compact hash index as records are submitted and uses it at report time to detect unique and uniqueKey violations. Does not perform field- or record-level validation.
• CrossSchemaValidator - builds a set of referenced field values across schemas as records are submitted and uses it at report time to detect foreignKey violations. Does not perform field-, record-, or uniqueness validation.

These are composed into two higher-level validators that provide the primary developer interface:

• SchemaValidator - runs field- and record-level validation on each submitted record and delegates cross-record tracking to a CrossRecordValidator. Produces a combined report at the end.
• DictionaryValidator - runs field- and record-level validation on each submitted record, delegates cross-record tracking to per-schema CrossRecordValidator instances, and delegates cross-schema tracking to a shared CrossSchemaValidator. Produces a combined report at the end.

All four components are exported so consumers can compose them independently. Per-record errors are returned directly from submit() and never stored internally. Cross-record and cross-schema violation counts are maintained as running totals; report() returns aggregate stats only.

Summary of Work

The implementation is organized into three areas:

1. Test infrastructure - A new private packages/data-generator package providing reusable test dictionaries and programmatic data generators needed to validate correctness and measure performance at scale. This is a prerequisite for all other work.

2. Breaking type changes - matchingRecords on unique error types changes from number[] to string[] (caller-supplied IDs replace positional indices). This is a semver-major breaking change that must be documented and communicated to consumers before the new validators ship.

3. Streaming validator implementation - The four new validator objects, their internal data structures (DataSetHashMap, SchemaDataReference), and the lifecycle API contract (submit() / report()).

Tasks

Test Infrastructure

• Create Data Generator Package #452
• Author Example Dictionaries for Lectern Performance Testing #453

Performance Testing and Verification Plan

• Performance Tests
  • Document performance test plan
  • Reproduce and document the memory limitation with a benchmark
  • Correctness test: streaming validators produce equivalent results to existing batch validators on same input
  • Memory profile: validate that peak memory is bounded to index size, not record count
  • FK ordering test: validate behaviour when parent schema records arrive after child records

Feature Implementation

• Documentation — document the streaming validation API Design: lifecycle contract, caller ID uniqueness responsibility, FK completeness caveat, semver bump and migration guide

• Breaking Type Changes — change matchingRecords from number[] to string[] on unique error types; update affected callers and tests; write migration guide for external consumers

• CrossRecordValidator

• CrossSchemaValidator

• SchemaValidator

• DictionaryValidator