Correct local model memory evidence

docs/optimization-log-2026-07-12.md

@@ -71,9 +71,13 @@ Hypothesis: a baked Phi Mini model could answer selected short-output families
 with zero scored Fireworks tokens while remaining inside the 4 GB, two-vCPU,
 and per-task deadline limits.
 
-Four pinned GGUF artifacts were tested with `llama-cpp-python==0.3.33`, two
-llama.cpp threads, 1,024-token context, swap disabled, and a
-4,000,000,000-byte systemd memory limit:
+The Participant Guide specifies a 4 GB RAM, two-vCPU grading environment and
+describes 2B-3B four-bit models as safe sizing guidance; it does not impose a
+3B parameter eligibility ceiling, define GB as decimal or binary, or require
+unused memory headroom. The project-prescribed Docker reproduction uses
+`--memory=4g`. Four pinned GGUF artifacts were tested with
+`llama-cpp-python==0.3.33`, two llama.cpp threads, 1,024-token context, swap
+disabled, and corrected smokes under that hard 4 GiB cgroup limit:
 
 - Phi-3 Mini Q4, SHA-256
   `8a83c7fb9049a9b2e92266fa7ad04933bb53aa1e85136b7b30f1b8000ff2edef`,
@@ -88,15 +92,46 @@ llama.cpp threads, 1,024-token context, swap disabled, and a
   `88f5f428d64ea04332eaf3dc19d05ad4df3e04847ad9d2b0e3c947233f361a56`,
   2,224,487,808 bytes.
 
-The first three artifacts repeatedly reached the kernel-rounded memory ceiling
-of 3,999,997,952 bytes. That is survival evidence, not safe deployment
-headroom. Merely widening the internal 3B parameter gate would incorrectly
-certify those runs. Q4_K_S Phi-4 was the strongest quality arm on the original
-short-family calibration (14/15: factual 5/5, sentiment 5/5, NER 4/5, no
-runtime failure), but was rejected on memory.
-
-IQ4_XS reduced peak memory to roughly 2.49-2.51 GB, leaving about 1.49 GB of
-measured headroom. It preserved exact sentiment accuracy on the ten original
+- Phi-3 Mini Q4: 18.107-second startup, 2.678-second generation,
+  4,231,827,456-byte peak, and no `memory.events` max or OOM event;
+- Phi-3.5 Mini Q4_K_S: 17.624-second startup, 5.432-second generation,
+  4,294,967,296-byte peak, 1,189 max events, and zero OOM events;
+- Phi-4 Mini Q4_K_S: 16.582-second startup, 1.667-second generation,
+  4,294,967,296-byte peak, 813 max events, and zero OOM events;
+- Phi-4 Mini IQ4_XS: 12.950-second startup, 2.656-second generation,
+  2,486,353,920-byte peak, and no max or OOM event.
+
+All four artifacts therefore loaded and generated individually without an OOM
+under this limit. Reaching `memory.max` proves pressure and no measured margin,
+not incompatibility: max events are reclaim/limit-hit evidence, while the OOM
+counter remained zero. These smokes do not by themselves certify the exact
+submitted image, full hybrid process, or ten-minute batch lifecycle.
+
+Q4_K_S Phi-4 was also the strongest quality arm on the original short-family
+calibration (14/15: factual 5/5, sentiment 5/5, NER 4/5, no runtime failure).
+It was then retested from a dropped page cache under the stricter
+4,000,000,000-byte limit. The smoke reached model-ready in 13.150 seconds,
+generated its one-token answer in 2.260 seconds, peaked at the kernel-rounded
+3,999,997,952-byte limit, and recorded 965 max events but zero OOM or OOM-kill
+events. A broader
+40-case sentiment-v2 run under the same limit scored 35/40 under the strict
+local proxy, including 19/20 standard-label cases, with zero runtime failures,
+invalid outputs, or timeouts. Mean latency was 4.869 seconds, p95 was 10.084
+seconds, the maximum was 15.092 seconds, and the full service completed in
+214.58 seconds. Its cgroup recorded 535 max events by model-ready, 1,038 at
+completion (503 after load), and zero OOM or OOM-kill events. The output binds
+dataset fingerprint
+`d56881134781fa960aa0dd2d80d45583e9b9405e4ed6bdbb0d9c31d4b6dbf6df`,
+run fingerprint
+`11e753e72814ed33ef513ec108369d8fb3a8047f5747c3fb5c4d7a45c4c447be`,
+and artifact SHA-256
+`1d8d83daa0e622fba686e1aa8fdce8dbc54f4690b03242f7825104f5a6c5a862`.
+This proves that the candidate can survive and answer repeatedly under either
+common interpretation of 4 GB on this host; it does not prove exact-image
+startup or timing.
+
+IQ4_XS reduced peak memory to roughly 2.49 GB, about 1.81 GB below the corrected
+hard limit. It preserved exact sentiment accuracy on the ten original
 unique calibration/consumed-holdout cases. A new 40-case balanced sentiment
 calibration then varied instruction placement, passage length, neutral/mixed
 nuance, and binary/three-way/four-way custom label vocabularies. IQ4_XS exceeded
@@ -108,18 +143,23 @@ removed.
 
 An early WSL wrapper pinned logical CPUs 0 and 1, later found to be sibling
 hyperthreads of one physical core. Those latency results were discarded. The
-final IQ4_XS corpus and isolated-prompt failures used no affinity and retained
-the backend's two-thread limit. The improved probe also established that this
-WSL user service has no delegated CPU controller: it records an unlimited
-cgroup CPU quota and cpuset `0-7`. Consequently these timings are diagnostic,
-not exact-image two-vCPU promotion evidence. The memory controller was active
-and its current-process peak/limit measurements are retained.
-
-Decision: reject every tested Phi artifact and make no scored-runtime or image
-change. No weight, llama.cpp dependency, local route, or broader parameter
-exception is packaged. The useful retained work is measurement-only: nested
-cgroup memory/CPU discovery, repeatable task-family filtering, and the broader
-sentiment calibration corpus. The sealed `holdout-v2` was not consumed.
+final IQ4_XS corpus, isolated-prompt failures, and corrected Q4_K_S runs used no
+affinity and retained the backend's two-thread limit. The improved probe also
+established that this WSL user service has no delegated CPU controller: it
+records an unlimited cgroup CPU quota and cpuset `0-7`. Consequently these
+timings are diagnostic, not exact-image two-vCPU promotion evidence. Likewise,
+the 13.150-second cold-cache smoke covers checkpoint identity, worker spawn,
+and load inside the service, not an externally measured Docker pull-to-ready
+interval. The memory controller was active and its current-process peak/limit
+and event measurements are retained.
+
+Decision: make no scored-runtime or image change yet. Phi-4 Mini Q4_K_S remains
+viable for an exact-image, narrow-route follow-up, but is not promoted from
+these development-host measurements. No tested weight, llama.cpp dependency,
+local route, or broader parameter exception is currently packaged. The useful
+retained work is measurement-only: nested cgroup memory/CPU discovery,
+repeatable task-family filtering, and the broader sentiment calibration corpus.
+The sealed `holdout-v2` was not consumed.
 
 ## Remaining high-value unknown
 

eval/benchmark_local_routes.py

@@ -83,6 +83,10 @@ class LocalRouteObservation:
     backend_configuration: dict[str, object]
     generation_configuration: dict[str, object]
     hardware: dict[str, object]
+    cgroup_memory_events_source: str | None
+    cgroup_memory_events_at_load: dict[str, int]
+    cgroup_memory_events: dict[str, int]
+    cgroup_memory_events_delta: dict[str, int]
     answer: str
     expected_answer: object
     correct: bool
@@ -129,6 +133,34 @@ def _metadata_mapping(value: object) -> dict[str, object]:
     return {str(key): item for key, item in value.items()}
 
 
+def _counter_mapping(value: object) -> dict[str, int]:
+    if not isinstance(value, Mapping):
+        return {}
+    counters: dict[str, int] = {}
+    for key, item in value.items():
+        if (
+            isinstance(key, str)
+            and key
+            and isinstance(item, int)
+            and not isinstance(item, bool)
+            and item >= 0
+        ):
+            counters[key] = item
+    return counters
+
+
+def _memory_event_evidence(
+    backend: object,
+) -> tuple[str | None, dict[str, int], dict[str, int], dict[str, int]]:
+    source = getattr(backend, "cgroup_memory_events_source", None)
+    return (
+        source if isinstance(source, str) else None,
+        _counter_mapping(getattr(backend, "cgroup_memory_events_at_load", {})),
+        _counter_mapping(getattr(backend, "cgroup_memory_events", {})),
+        _counter_mapping(getattr(backend, "cgroup_memory_events_delta", {})),
+    )
+
+
 def _generation_configuration(
     max_tokens: int | None,
     temperature: float,
@@ -271,6 +303,12 @@ def run_shadow_benchmark(
                 temperature=temperature,
             )
         except Exception as exc:
+            (
+                memory_events_source,
+                memory_events_at_load,
+                memory_events,
+                memory_events_delta,
+            ) = _memory_event_evidence(backend)
             error_latency = exc.latency_ms if isinstance(exc, LocalBackendError) else None
             error_peak_memory = (
                 exc.peak_memory_bytes if isinstance(exc, LocalBackendError) else None
@@ -311,6 +349,10 @@ def run_shadow_benchmark(
                     backend_configuration=configuration,
                     generation_configuration=generation_configuration,
                     hardware=hardware,
+                    cgroup_memory_events_source=memory_events_source,
+                    cgroup_memory_events_at_load=memory_events_at_load,
+                    cgroup_memory_events=memory_events,
+                    cgroup_memory_events_delta=memory_events_delta,
                     answer="",
                     expected_answer=case.expected_answer,
                     correct=False,
@@ -377,6 +419,12 @@ def run_shadow_benchmark(
             (time.perf_counter() - started) * 1000.0,
             generation.latency_ms,
         )
+        (
+            memory_events_source,
+            memory_events_at_load,
+            memory_events,
+            memory_events_delta,
+        ) = _memory_event_evidence(backend)
         observations.append(
             LocalRouteObservation(
                 task_id=case.task.id,
@@ -410,6 +458,10 @@ def run_shadow_benchmark(
                 backend_configuration=configuration,
                 generation_configuration=generation_configuration,
                 hardware=hardware,
+                cgroup_memory_events_source=memory_events_source,
+                cgroup_memory_events_at_load=memory_events_at_load,
+                cgroup_memory_events=memory_events,
+                cgroup_memory_events_delta=memory_events_delta,
                 answer=answer,
                 expected_answer=case.expected_answer,
                 correct=correct,
@@ -552,6 +604,10 @@ def _load_failure_observations(
             backend_configuration=configuration,
             generation_configuration=generation_configuration,
             hardware=hardware,
+            cgroup_memory_events_source=None,
+            cgroup_memory_events_at_load={},
+            cgroup_memory_events={},
+            cgroup_memory_events_delta={},
             answer="",
             expected_answer=case.expected_answer,
             correct=False,

eval/manifests/track1-v1.json

@@ -1,5 +1,5 @@
 {
-  "benchmark_shared_implementation_sha256": "6a3bc2ec8c93edd7d99a7dfdc62e9f8505c6013e44ccbe500a73ab61413c7274",
+  "benchmark_shared_implementation_sha256": "1cff15c7f81f54adeeee32348f171a5a419c9cced775dfe8d38edfc8ddf9c79e",
   "datasets": {
     "calibration": {
       "answer_contract_sha256": "4cbbf8355c7a6b9b1980c98895e600202d98230f4f856314b8319a4c2ca3f55b",
@@ -54,10 +54,10 @@
       "task_count": 40
     }
   },
-  "fallthrough_source_tree_sha256": "cb9e0183c14048841079bb7077115868389c958b1c30e11970d8cd9554fc8312",
+  "fallthrough_source_tree_sha256": "03c005044658fd0aec0a3bfd78f12eeb777361338c48f82551481762e10132d8",
   "producer_implementation_sha256": {
-    "local_shadow_observations": "c7b2ec662aac277ff5c724eed477a88afe44c0b9feb8a00e65343d0193e05c7e",
-    "route_observations": "1fa7aad37791a0a497baefd24f42343050568edf07f53b520279a6284821e874"
+    "local_shadow_observations": "0cdef5e5af37a37aafa995fd8a13f24d52fb14a665d596951a9773e74490806b",
+    "route_observations": "3c5fa9e7188e2a0d6eeabcca1b3351d0187445d3c5c181aeb59d7ed22bb3d4de"
   },
   "promotion_rule": {
     "maximum_exact_one_sided_p": 0.1,
@@ -87,7 +87,7 @@
     "policy": "transparent-local-proxy-scorer-v3"
   },
   "sources": {
-    "eval/benchmark_local_routes.py": "6623923e16b8da33976aa3a42dea9a1c2f2cc337a00b46dffac994d8e9b71cb8",
+    "eval/benchmark_local_routes.py": "9c451db074fdcbdc78c80d7410e0f1a446ca04a2d48bbb8c334fd3bed8a6782a",
     "eval/common.py": "f022e701f59e3ca38beea31216628f3a2f09b2532453f81860a83b09f68ca6cf",
     "eval/compare_routes.py": "d9ad67865779a2dd27c29bb955b9d0e0fededc7e0ce361b88b2c7392fd2add2e",
     "eval/constraints.py": "fbd5df20f19f290b1a2513e76f9c4fb88650c60cd15d9c99e64fc1a6804e29f4",

eval/manifests/track1-v2.json

@@ -1,5 +1,5 @@
 {
-  "benchmark_shared_implementation_sha256": "6a3bc2ec8c93edd7d99a7dfdc62e9f8505c6013e44ccbe500a73ab61413c7274",
+  "benchmark_shared_implementation_sha256": "1cff15c7f81f54adeeee32348f171a5a419c9cced775dfe8d38edfc8ddf9c79e",
   "datasets": {
     "calibration": {
       "answer_contract_sha256": "4cbbf8355c7a6b9b1980c98895e600202d98230f4f856314b8319a4c2ca3f55b",
@@ -54,10 +54,10 @@
       "task_count": 80
     }
   },
-  "fallthrough_source_tree_sha256": "cb9e0183c14048841079bb7077115868389c958b1c30e11970d8cd9554fc8312",
+  "fallthrough_source_tree_sha256": "03c005044658fd0aec0a3bfd78f12eeb777361338c48f82551481762e10132d8",
   "producer_implementation_sha256": {
-    "local_shadow_observations": "c7b2ec662aac277ff5c724eed477a88afe44c0b9feb8a00e65343d0193e05c7e",
-    "route_observations": "1fa7aad37791a0a497baefd24f42343050568edf07f53b520279a6284821e874"
+    "local_shadow_observations": "0cdef5e5af37a37aafa995fd8a13f24d52fb14a665d596951a9773e74490806b",
+    "route_observations": "3c5fa9e7188e2a0d6eeabcca1b3351d0187445d3c5c181aeb59d7ed22bb3d4de"
   },
   "promotion_rule": {
     "maximum_exact_one_sided_p": 0.1,
@@ -87,7 +87,7 @@
     "policy": "transparent-local-proxy-scorer-v3"
   },
   "sources": {
-    "eval/benchmark_local_routes.py": "6623923e16b8da33976aa3a42dea9a1c2f2cc337a00b46dffac994d8e9b71cb8",
+    "eval/benchmark_local_routes.py": "9c451db074fdcbdc78c80d7410e0f1a446ca04a2d48bbb8c334fd3bed8a6782a",
     "eval/common.py": "f022e701f59e3ca38beea31216628f3a2f09b2532453f81860a83b09f68ca6cf",
     "eval/compare_routes.py": "d9ad67865779a2dd27c29bb955b9d0e0fededc7e0ce361b88b2c7392fd2add2e",
     "eval/constraints.py": "fbd5df20f19f290b1a2513e76f9c4fb88650c60cd15d9c99e64fc1a6804e29f4",

scripts/smoke_test_llama_cpp.py

@@ -98,6 +98,16 @@ def _generation_configuration(args: argparse.Namespace) -> dict[str, object]:
     }
 
 
+def _refresh_memory_event_report(
+    report: dict[str, Any],
+    backend: LlamaCppBackend,
+) -> None:
+    report["cgroup_memory_events_source"] = backend.cgroup_memory_events_source
+    report["cgroup_memory_events_at_load"] = dict(backend.cgroup_memory_events_at_load)
+    report["cgroup_memory_events"] = dict(backend.cgroup_memory_events)
+    report["cgroup_memory_events_delta"] = dict(backend.cgroup_memory_events_delta)
+
+
 def run_smoke(args: argparse.Namespace) -> tuple[int, dict[str, Any]]:
     identity_started = time.perf_counter()
     identity_error: BaseException | None = None
@@ -166,6 +176,10 @@ def run_smoke(args: argparse.Namespace) -> tuple[int, dict[str, Any]]:
         "generation_peak_memory_bytes": None,
         "observed_peak_memory_bytes": None,
         "memory_measurement": "container_cgroup_peak_or_worker_process_peak_rss_bytes",
+        "cgroup_memory_events_source": None,
+        "cgroup_memory_events_at_load": {},
+        "cgroup_memory_events": {},
+        "cgroup_memory_events_delta": {},
         "error_code": None,
         "error_type": None,
         "error": None,
@@ -206,6 +220,7 @@ def run_smoke(args: argparse.Namespace) -> tuple[int, dict[str, Any]]:
         report["model_load_success"] = True
         report["backend_configuration"] = backend.configuration
         report["hardware"] = backend.hardware
+        _refresh_memory_event_report(report, backend)
         report["run_fingerprint"] = experiment_fingerprint(
             backend=backend.backend_name,
             model_label=model_label,
@@ -228,6 +243,7 @@ def run_smoke(args: argparse.Namespace) -> tuple[int, dict[str, Any]]:
             max_tokens=args.max_tokens,
             temperature=0.0,
         )
+        _refresh_memory_event_report(report, backend)
         report["generation_latency_ms"] = generation.latency_ms
         report["prompt_tokens"] = generation.prompt_token_count
         report["tokens_generated"] = generation.token_count
@@ -256,6 +272,7 @@ def run_smoke(args: argparse.Namespace) -> tuple[int, dict[str, Any]]:
         )
         return 0, report
     except Exception as exc:
+        _refresh_memory_event_report(report, backend)
         if isinstance(exc, LocalBackendError):
             report["generation_latency_ms"] = exc.latency_ms
             report["generation_peak_memory_bytes"] = exc.peak_memory_bytes