Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@ jobs:
run: /tmp/typeference diff examples/helio --against dist --emit-ard --publisher-domain helio.example
- name: Eval harness dry run (no API key, no network)
run: /tmp/typeference eval examples/helio --scenarios evals/scenarios --out /tmp/eval-dry-run
- name: Playground builds (js/wasm bridge and example packer)
run: |
GOOS=js GOARCH=wasm go build ./cmd/typeference-wasm
go run ./cmd/playground-pack -root .. -out /tmp/examples.json
working-directory: go
# NOTE: BETH pack/score is intentionally NOT gating CI yet. The harness is
# unproven end to end (no real judged run has been collected), and an
# offline pack+score observes nothing — a green scorecard there would be
Expand Down
41 changes: 41 additions & 0 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Builds the browser playground (web/playground) and deploys it to GitHub
# Pages on every push to main. The playground is the unmodified Go compiler
# built for js/wasm; see docs/decisions/0010-browser-playground.md.
name: pages

on:
push:
branches: [main]
workflow_dispatch:

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: pages
cancel-in-progress: true

jobs:
deploy:
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: stable
cache-dependency-path: go/go.sum
- name: Build playground (wasm + examples + wasm_exec.js)
run: make playground VERSION=${GITHUB_SHA::12}
- uses: actions/configure-pages@v5
with:
enablement: true
- uses: actions/upload-pages-artifact@v3
with:
path: web/playground
- id: deployment
uses: actions/deploy-pages@v4
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,7 @@ beth-runs/
.vs/
__pycache__/
*.pyc
# Generated playground assets (make playground)
web/playground/typeference.wasm
web/playground/wasm_exec.js
web/playground/examples.json
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,16 @@ First tagged release. Everything before this version was unversioned development
files with expected-behavior rubrics, LLM-judged adherence scoring, and a
dry-run default that emits exact request payloads with no network calls
(ADR-0008). Starter corpus under `evals/`.
- **Browser playground** under `web/playground` — the unmodified Go compiler
built for `js/wasm` against an in-memory filesystem, with a dependency-free
UI: live recompilation, artifact browser, embedding graph, resolved-bundle
view, and shareable links. Deployed to GitHub Pages on push to `main`; the
Helio example reproduces the committed `dist/` digest exactly (ADR-0010).
Its **Equivalence tab** is a BETH operator console: packs scenario × surface
cells in the browser with the real `equivalence pack` code, collects
responses by copy/paste, exports a deterministic run `.tar.gz` for local
scoring, and visualizes the resulting `scorecard.json` — no credential ever
touches the page (ADR-0011).
- `typeference version` command in both implementations.
- Architecture decision records under `docs/decisions/`.
- Tag-driven release workflow shipping per-platform archives of the Go CLI with
Expand Down
11 changes: 10 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ LDFLAGS := -s -w -X main.version=$(VERSION)
BINDIR := bin

.PHONY: all build build-go build-dotnet test test-go test-dotnet conformance \
selfhost selfhost-check fmt vet clean release-binaries
selfhost selfhost-check fmt vet clean release-binaries playground

all: build test

Expand Down Expand Up @@ -56,6 +56,15 @@ vet:
cd go && $(GO) vet ./...
cd go && test -z "$$(gofmt -l .)"

# Browser playground: the unmodified Go compiler built for js/wasm plus a
# static, dependency-free UI. Everything lands in web/playground; serve that
# directory with any static file server. wasm_exec.js is copied from the
# building toolchain so it always matches the wasm binary's Go version.
playground:
cd go && GOOS=js GOARCH=wasm $(GO) build $(GOFLAGS) -ldflags "$(LDFLAGS)" -o ../web/playground/typeference.wasm ./cmd/typeference-wasm
cp "$$($(GO) env GOROOT)/lib/wasm/wasm_exec.js" web/playground/wasm_exec.js
cd go && $(GO) run ./cmd/playground-pack -root .. -out ../web/playground/examples.json

clean:
rm -rf $(BINDIR)

Expand Down
21 changes: 21 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,27 @@ skills:

Use profiles for reusable organizational, domain, or team defaults that should participate in composition without producing their own target bundle.

## Try it in your browser

The **[playground](https://buchk.github.io/TypeFerence/)** runs the real Go
compiler — built for WebAssembly, internals untouched — entirely in your tab.
Edit typed source, watch the compiled Codex/Copilot/Cursor/neutral artifacts,
the embedding graph, and the diagnostics update live. The status-bar digest is
the determinism guarantee made interactive: the Helio example reproduces the
digest of this repository's committed `dist/` exactly, and the self-hosting
example recompiles the repository's own root `AGENTS.md` byte for byte. There
is no backend; nothing you type leaves the browser
([ADR-0010](docs/decisions/0010-browser-playground.md)).

The playground's **Equivalence** tab walks the full BETH loop (ADR-0009)
without ever asking for a credential: it packs scenario × surface cells with
the real `equivalence pack` code, you collect responses by copy/paste from
real hosts (BETH's operator model), it exports the assembled run as a
deterministic `.tar.gz`, you score locally — keys stay in your terminal — and
dropping the resulting `scorecard.json` back onto the page renders adherence,
agreement, and every divergence
([ADR-0011](docs/decisions/0011-playground-live-runs.md)).

## Quick start

Requires Go 1.24+ and nothing else. From clone to compiled artifacts:
Expand Down
83 changes: 83 additions & 0 deletions docs/decisions/0010-browser-playground.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# 0010 — Browser playground: the Go compiler as WebAssembly

## Status

Accepted.

## Context

Evaluating TypeFerence required installing a toolchain: clone the repository,
build the Go binary (or the .NET solution), then run `validate` / `build`
against an example. That is a high price for the first five minutes of
curiosity, and it keeps the project's two strongest properties abstract:

- **Determinism.** "Byte-identical artifacts" is a claim in the README until a
person watches the same digest come out of two different environments.
- **Composition.** The embedding/promotion model is easiest to understand by
editing a profile and watching the compiled `AGENTS.md` change.

The project already pays for two independent implementations kept
byte-identical by a conformance suite (ADR-0005). That investment makes a
browser port nearly free: Go compiles to `js/wasm` out of the box, and the Go
implementation's only dependency is `gopkg.in/yaml.v3` (ADR-0003), which is
pure Go.

## Decision

Ship a static, zero-install playground at `web/playground`, deployed to GitHub
Pages on every push to `main` (`.github/workflows/pages.yml`).

1. **The compiler is not forked, wrapped, or reimplemented.** A new
`go/cmd/typeference-wasm` entry point builds the existing `internal/`
packages for `GOOS=js GOARCH=wasm` and exposes one function,
`TypeFerence.compile`, that runs the same `compile.Validate` →
`compile.Build` → `compile.HashDirectory` pipeline as the CLI.
2. **File I/O is satisfied, not removed.** Go's `js/wasm` syscall layer
delegates to a global Node-style `fs` object. `web/playground/memfs.js`
implements that API over an in-memory tree, so the compiler's ordinary
`os` calls work unchanged inside the browser tab.
3. **The UI has zero dependencies.** Plain HTML/CSS/JS: an editor with
lightweight syntax coloring, an artifact browser, the embedding graph
(including computed structural-interface satisfaction), the resolved-bundle
view, and a share link (gzip + base64url in the URL fragment). Nothing the
user types leaves the tab; there is no backend.
4. **Digests reproduce the repository's own builds.** The in-memory source
directory is named after the loaded example and the ARD publisher domain
matches the repository's build commands, so the Helio example produces the
exact digest of the committed `dist/`, and the maintainer example
reproduces the repository-root `AGENTS.md` byte for byte.
5. **Generated assets are not committed.** `typeference.wasm`,
`examples.json`, and `wasm_exec.js` are produced by `make playground`;
`wasm_exec.js` is copied from the building toolchain's `GOROOT` so it can
never skew from the wasm binary's Go version. CI builds the wasm bridge and
packer on every push so the playground cannot silently rot.

## Consequences

- Anyone can evaluate TypeFerence — including the determinism guarantee — from
a link, with nothing installed and no data leaving their machine.
- The playground is a demonstration surface, not a supported runtime. The CLI
remains the product; the playground intentionally exposes no command surface
beyond compile-on-edit.
- The wasm binary is ~5 MB (~1.8 MB compressed), an acceptable one-time load
for a code playground.
- `memfs.js` implements the private contract between Go's syscall layer and
its JavaScript host. A Go major release could change that contract; because
`make playground` builds binary and shim support files from one toolchain,
such a break surfaces at build/CI time, not silently in production.

## Alternatives considered

- **A TypeScript reimplementation of the compiler for the browser.** A third
implementation to keep conformant forever, and it would demonstrate nothing
about the shipped compilers. Rejected outright; the whole value of the
playground is that it runs the real one.
- **A hosted compile API.** Requires a server, introduces cost, latency, and a
privacy story ("your agent definitions are uploaded to..."), and undermines
the static-artifact ethos of the project. The wasm build is strictly better.
- **Compiling the C# reference implementation with .NET's wasm support.**
Works, but produces a far larger payload and a heavier runtime; the Go
implementation exists precisely to be the small, static, portable one
(ADR-0007).
- **Recorded demo (GIF/asciinema) in the README.** Cheap, but not interactive
and proves nothing — a recording of a digest is not a digest you produced.
86 changes: 86 additions & 0 deletions docs/decisions/0011-playground-live-runs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# 0011 — Playground equivalence console: no secrets in the browser

## Status

Accepted and implemented (the playground's Equivalence tab). Extends
ADR-0010.

## Context

The playground (ADR-0010) shows what TypeFerence compiles but not what the
compiled definition *does*. The natural extension — fire the compiled agent
at real models and compare — is also the project's central question: BETH
(ADR-0009) exists because the same declared intent, run on different hosts,
may or may not behave equivalently.

A first implementation did this with bring-your-own-key adapters: paste an
Anthropic/OpenAI/Gemini/GitHub Models key into the page, requests go directly
from the browser to the provider, keys never persist unless opted in, and a
CSP pins `connect-src` to the four provider hosts. It worked, and it was
still wrong. Asking visitors to paste provider credentials into a hosted page
is exactly the anti-pattern this project's trust posture argues against: the
"keys never leave your browser" claim is unverifiable without auditing the
served JavaScript on every load, and one compromised deploy would turn the
page into a key harvester. A project whose pitch is provenance and
fail-closed trust should not train users to paste secrets into web pages.

There is also no real OAuth escape hatch: no target provider offers
third-party OAuth for API access, GitHub Copilot has no public API (and the
token exchange unofficial clients use violates GitHub's terms), and a backend
proxy would route other people's keys and prompts through project
infrastructure — strictly worse.

## Decision

The playground never asks for a secret. The Run surface becomes a **BETH
operator console** that keeps every credential in the user's own terminal:

1. **Pack in the browser.** The wasm bridge grows a `pack` entry point over
the existing `eval.Pack`, laying out scenario × surface cells for the
compiled definition (scenario files ship with the examples).
2. **Collect by copy and paste.** Each cell renders as a card: copy the
prompt, run it in a real host (Claude Code, Copilot, Cursor, a raw API
call — the operator's choice, with the operator's own keys), paste the
response back. This is not a workaround; a human operator collecting one
attested response per cell is BETH's defined collection model (ADR-0009).
3. **Export the run.** The console assembles the pasted responses into the
run-directory layout and downloads it as a `.tar.gz` (dependency-free tar
writer plus the browser's native gzip `CompressionStream`).
4. **Score locally.** The console prints the exact
`typeference equivalence score <run> --live` command; the judge key lives
in the local environment, where keys belong.
5. **Visualize the scorecard.** Dropping the resulting `scorecard.json` back
onto the page renders adherence per surface, cross-surface agreement, and
every divergence. Reading a local file requires no trust.

## Consequences

- The full BETH loop — author, compile, pack, collect, score, inspect —
becomes walkable from a browser tab, and the only step that touches a
credential happens in the user's terminal against the provider's own CLI
or API.
- The interim BYOK adapters (`providers.js`) and key UI were deleted; the
verified per-provider transport knowledge they encoded (CORS behavior, SSE
wire formats) is preserved in this branch's history should a legitimate
need return. The page's CSP now pins `connect-src` to its own origin.
- The export is a plain ustar + gzip archive written with a fixed mtime, so
identical runs produce byte-identical archives; the CLI's workspace-digest
check verified the browser-written bytes exactly during development.
- The console must not blur illustration into measurement: it emits and
consumes BETH's own run-directory and scorecard shapes rather than
inventing a parallel result format.
- Manual paste-back limits collection volume. That is BETH's existing,
documented trade-off (one observation per surface, operator-attested), not
a new one.

## Alternatives considered

- **Bring-your-own-key direct calls** (built, then rejected): unverifiable
trust ask; normalizes pasting API keys into web pages; one bad deploy from
being harmful.
- **OAuth/SSO to providers**: not offered for third-party API access by any
target provider.
- **A backend proxy**: centralizes other people's keys and prompts; worse on
every axis this project cares about.
- **Keeping BYOK behind an "advanced" toggle**: still trains the behavior;
the honest console makes the same demo possible without it.
2 changes: 2 additions & 0 deletions docs/decisions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,5 @@ Format: `NNNN-short-title.md` with sections **Status**, **Context**, **Decision*
| [0007](0007-release-distribution.md) | Release distribution: static binaries, no installer |
| [0008](0008-eval-harness-scope.md) | Behavioral eval harness: scope and design |
| [0009](0009-behavioral-equivalence-harness.md) | BETH: the behavioral equivalence test harness |
| [0010](0010-browser-playground.md) | Browser playground: the Go compiler as WebAssembly |
| [0011](0011-playground-live-runs.md) | Playground equivalence console: no secrets in the browser |
48 changes: 48 additions & 0 deletions docs/design-notes/typed-context-shapes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Design note: typed context shapes

Status: idea captured 2026-07-13, not designed, not scheduled. A future ADR
decides; this note only pins the problem and the hook points so the thought
survives.

## Problem

v3 disciplines composition: profiles, capabilities, skills, and overrides are
typed, versioned, promoted with ambiguity checks, and carry provenance. But
`contextFiles` are opaque markdown. Everything the type system squeezed out of
instruction files can silently migrate into referenced context files —
org charts, timelines, policies, and de-facto behavioral overrides living as
free prose that no compile-time check can see. The sprawl TypeFerence exists
to eliminate does not disappear; it relocates to the one untyped surface.

## Gesture at a solution

1. **Governable context types.** A small starter registry of shapes a context
file can declare itself to be — e.g. `cast-of-characters` (roster: names,
roles, authority), `timeline` (ordered events; **decision points** as
first-class entries on a timeline, not a separate type), `glossary`,
`policy` (addressable clauses). Declaring a type is optional; *using* one
is strict: the compiler validates the file against the shape and fails
closed on drift, same as every other v3 check.
2. **Extensible registry, strict instances.** Do not enumerate the world's
context types — runtime-adjacent shapes will keep appearing. Make adding a
type cheap (a shape schema is itself a versioned resource a source tree
can carry), but make conformance to a chosen type non-negotiable. Strict
about use, liberal about the universe.
3. **Overrides belong in YAML, not prose.** If an agent deviates from an
embedded profile's behavior, that override should be a declared member on
the typed resource — baked into the compiled skill/bundle with provenance —
never a paragraph in a context file that quietly contradicts what the
profile promoted. Possible compile-time stance: typed context files simply
have no vocabulary for behavioral instruction, which is what makes them
governable.

## Hook points when this becomes real

- Loader/validation: `resource.validateShape` and friends already own
referenced-file checks; typed context validation slots beside them.
- Spec: a "Typed context" section; canonical form questions (ordering,
formatting) get the ADR-0004 treatment so digests stay stable.
- Conformance: new fixtures per shape, including failure fixtures.
- ARD: each context type maps naturally to a media type on published entries.
- Self-hosting: `agents/maintainer/context/*` is the first test corpus —
ADR-0006 already catalogs where its prose is doing type-system work.
Loading
Loading