Skip to content

Benchmark cold, warm, and incremental materialization CPU #759

Description

@flyingrobots

Parent: #758
Related implementation: #738

Actor story

As a git-warp maintainer, I need reproducible cold, warm, and incremental materialization measurements so that an optimization is proven against the same causal workload and later changes cannot quietly restore full replay or redundant CAS work.

Required work

  • Define a versioned, deterministic causal corpus generated from a fixed seed.
  • Keep corpus generation outside timed regions while reporting its logical size and topology.
  • Run cold materialization, warm git-cas reuse, and one bounded suffix update as distinct scenarios.
  • Capture process user/system CPU, wall time, peak RSS/heap, Git operations, git-cas evidence, and output cardinality.
  • Include warmups and repeated samples; report median and dispersion.
  • Execute base and head on the same runner/toolchain with order bias controlled.
  • Emit a stable JSON result schema suitable for CI comparison and historical publication.

Acceptance criteria

  • The corpus is deterministic and versioned.
  • Cold, warm, and incremental scenarios assert identical semantic results where expected.
  • Timed regions and excluded setup are documented.
  • CPU and wall time are reported separately.
  • Warm reuse proves that git-cas materialization state is actually hit.
  • A checked-in calibration baseline and reviewed regression budget exist.
  • Repeated local and CI runs demonstrate that the chosen threshold exceeds ordinary noise without masking material regression.
  • Tests reject malformed, partial, or semantically incomplete result records.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:runtimePrimary work area: runtime.area:testingPrimary work area: testing.area:toolingPrimary work area: tooling.legend:perfMethod legend perf.priority:asapImmediate release pressure.status:availableOpen and available for prioritization; not blocked or actively in progress.type:proof-storyChild issue scoped to one actor story or proof contract.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions