Skip to content

test: verify installed Decisions end to end before calibration #125

Description

@timeleft--

02 — Install merged main locally and verify Decisions before calibration

Parent

#123

What to build

Prove the installed Decisions path works before calibration. After ticket 01 merges, install that upstream main revision locally, explicitly configure Decisions with an initial Noul question and threshold 0.5, and verify real OpenRouter review and durable recall through the installed MCP server. The initial settings are test inputs, not calibrated defaults. A suitable initial question is: Does this candidate meet the supplied Trust Gate policy for promotion as a useful institutional memory?

Acceptance criteria

  • Confirm ticket 01's implementation is on upstream main, record the exact commit, and install that revision locally using the project's supported installation/update process. Record the installed executable and version/revision; do not substitute an editable development checkout for installation acceptance.
  • Explicitly configure decisions, OpenRouter, ~typesafe/jev-latest, the initial Noul question, and threshold 0.5 using the normal configuration and nested trust_gate_decisions_config. Read the authorized credential through trust_gate_api_key_file; never print it.
  • Run doctor and start the installed MCP server with a dedicated test identity and disposable test data repository. Verify the effective policy, model, endpoint, configuration, and credential source.
  • Through a real MCP client session, save a synthetic quality memory, invoke propose_truth, and verify Jev approval, persisted Decisions provenance, and visibility through governed recall and get_thought.
  • Exercise both approval and rejection through the real MCP flow and verify that each Verdict matches its recorded probability and configured threshold. Verify rejected records are absent from governed recall. Use isolated threshold-boundary test configurations if the uncalibrated question does not naturally produce both outcomes; do not confuse this plumbing check with quality calibration. Restore 0.5 afterward and record the initial question and observed probabilities.
  • Restart the installed test server and re-read the approved memory and provenance, proving persistence across process restart.
  • Using an isolated invalid-credential test configuration, exercise a real failed review and verify fail-closed behavior with no promotion and no fallback. Restore the valid test configuration and verify a successful request afterward.
  • Report the installed commit, configuration excluding secrets, returned model identity when available, expected versus actual outcomes, and sanitized MCP evidence. A green CI run alone does not complete this task.
  • Diagnose and route confirmed defects through the existing Hermes repair workflow; after fixes merge, reinstall the corrected main revision and repeat affected acceptance checks. Do not declare completion with known failing required behavior.
  • Keep acceptance records disposable and avoid modifying the pinned vendor source checkout or introducing synthetic records into the institutional data repository.

Blocked by

Execution ownership

The Captain coordinates this local acceptance autonomously on the operator's host; Hermes crew handles implementation and repairs. This is executable agent work, not a human-only gate. The parent epic is not complete until the installed runtime passes these checks. Dispatch starts only after the operator's command to run the epic.

The operator has authorized using the local OpenRouter key file for these real tests. Its host-specific location is supplied in the private Captain handoff, not embedded as a worker-container path assumption. If it is missing, unreadable, or rejected, report the concrete result without exposing the credential. Do not replace provider state or request a new key without that evidence.

Activity

  1. timeleft-- commented on Sep 29, 2026

    @timeleft--
    MemberAuthor

    Installed/live acceptance complete.

    Implementation and repair revisions:

    Secret-free configuration:

    • policy: decisions
    • provider: openrouter
    • model alias: ~typesafe/jev-latest
    • endpoint: https://openrouter.ai/api/alpha/decisions
    • Noul question: Does this candidate meet the supplied Trust Gate policy for promotion as a useful institutional memory?
    • threshold: 0.5
    • credential supplied only through trust_gate_api_key_file; value was never printed.

    Real installed MCP evidence:

    • Quality candidate: probability 0.90, approved/promoted, governed recall and get_thought visible, Decisions provenance persisted.
    • Server restart: the approved record and provenance remained readable.
    • Vague candidate: probability 0.04, rejected, absent from governed recall.
    • Invalid credential: real HTTP 401, failed closed, no promotion, no fallback.
    • Valid credential restored: real request succeeded; probability 0.30, rejected at threshold 0.5.
    • Returned model identity: typesafe/jev-1.13-20260917.

    Confirmed defect and repair:

    • The original egress disclosure incorrectly called the transmitted metadata “redacted.”
    • PR fix: describe Trust Gate egress metadata as selected fields, not redacted #130 corrected code, diagnostics, docs, and tests to say the full prompt, full candidate body, and complete selected metadata fields are sent; agent_id and metadata.extra are excluded.
    • Exact-head validation for PR fix: describe Trust Gate egress metadata as selected fields, not redacted #130: Ruff passed; 108 affected trust-gate tests passed locally; both JJ CI matrices, CodeQL, and semantic-title checks passed after rerunning a GitHub-rate-limited JJ download.
    • Fresh install from repaired main: doctor reported the corrected disclosure and credential source without secrets.
    • Fresh real MCP/Jev request after repair: model typesafe/jev-1.13-20260917, probability 0.78, approved/promoted at 0.5, governed recall visible with persisted Decisions provenance.

    All acceptance data remained under disposable /private/tmp storage; no institutional data repository or pinned vendor checkout was modified.

  2. timeleft-- commented on Sep 29, 2026

    @timeleft--
    MemberAuthor

    Installed runtime acceptance and the post-repair live Jev check are complete. Closing this dependency so calibration issue #126 can begin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions