Sandbox templates: content-keyed records, derived-image strategies, template forks, GC - #16
Merged
Merged
Conversation
Templates are keyed by content — unique (provider, base_image, requirements_hash) — with status building -> available -> failed. POST /templates answers 202 and the caller polls; the build runs as a background task against the TemplateCapability a provider may implement (implementations land next). Concurrent identical creates resolve through the unique index: the losing racer re-reads the winner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TemplateCapability input is (base_image, setup_script) — no live host: the strategy synthesizes FROM <base> plus the script as a bake-time root step, builds with the docker CLI, and the resulting ref is the handle. docker keeps the local tag, docker-sbx builds into the daemon sandboxd reads templates from, and exe pushes to a registry exe.dev can pull (exe.dev has no snapshot verb — the image is its snapshot; registry credentials are EXE_TEMPLATE_REGISTRY/_USERNAME/_PASSWORD). Providers stay dumb: the handle rides host.image and create_vm is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HostCreate accepts template — an id or a requirements hash — and the resolution is explicit image > template handle > provider default, with the resolved handle riding host.image into the unchanged provision path. A hash matching several bases prefers the available ones, newest first. Naming a template that is not available fails 409 with the status (and last_error for failed builds); drukbox never builds on miss — the caller owns when to build. A template request is a customization, so it never claims a warm pool host, and each fork stamps last_used_at for GC. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Templates cost money independent of hosts, so python -m templates.janitor gets its own cron entry. Three sweeps: an unfinished build older than TEMPLATE_BUILD_TIMEOUT_MINUTES becomes failed (the only exit for a build whose process died); failed templates keep their diagnostics for TEMPLATE_FAILED_RETENTION_HOURS, then go; available templates unleased for TEMPLATE_UNUSED_TTL_DAYS age out through the strategy's delete -- which is also how a superseded hash dies, since edits mint a new one. Deletes re-validate under the row lock so a fresh lease or status change spares the row, hosts-janitor style. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
create_template replaces materialize_template on TemplateCapability and
its three providers, and the DockerCLI attribute is docker_cli. The docs
and comments this branch added are rewritten to ASD-STE100 shape: short
sentences, active voice, one name per concept ("persistent" replaces
"durable"; "template" replaces "artifact").
exe.dev assumes a public image, and templates land in a private
registry, so create_vm now passes --registry-auth when the host image
lives on the configured template registry — and only then, so the
credentials never reach another registry (exe.dev docs: private-image).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The requirements hash is script-only, so one hash can match two base images on the same provider — hash leasing had to guess with a newest-available tie-break. The guess is gone: template is a UUID, the lookup is exact, and the wire boundary validates the type. Callers do not lose content addressing — POST /templates is idempotent by content and always returns the current record with its ID. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repost-as-lookup suggestion blurred the contract: POST /templates is the only path that builds, and naming a template on POST /hosts never creates one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reaping is one concern, and every resource type adding its own cron entry is operator sprawl. The reapers stay in their packages; the janitor package is the single cron entry that runs both. The pool maintainer stays its own command because it creates resources instead of reaping them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The registry setting is not template-specific in behavior — registry auth applies to any image on that registry — so the name drops the template. The three GC settings follow the LEASE_DEFAULT_TTL precedent: no unit suffix, value in seconds (TEMPLATE_BUILD_TIMEOUT=3600, TEMPLATE_FAILED_RETENTION=86400, TEMPLATE_UNUSED_TTL=1209600). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Settings defaults and UUID parsing are pydantic's contract, not ours. The provider-error build test fell to the crash test, which guards the same except arm plus the breadth of the catch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The column holds the provider image that lands in host.image, so it is named image — built from base_image, produces image. Handle named the mechanism. templates/deps.py held one pass-through factory; it now lives beside the router (hosts/deps.py keeps its file because it owns real app-state wiring). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ashes The provider surface no longer borrows the service's word: TemplateCapability declares build_template_image -> image ref and delete_template_image — a provider author builds and deletes images, only the service creates template records. derive_image_tag and create_build_context replace the noun-phrase function names. setup_script_hash replaces requirements_hash: the column stores sha256(setup_script) and now says so next to its input field. UnknownTemplateError replaces TemplateReferenceError, mirroring UnknownProviderError on the same route. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The module synthesizes a Dockerfile and drives the DockerCLI, so its home is the package that owns them. At the providers top level it inverted the layering: framework-level code importing one concrete provider. exe and docker-sbx already import from providers.docker for the CLI itself; the derived-image import now reads the same way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
We were growing a docker client — subprocess verbs, stderr string matching, a temp-file dance to keep env values off argv. aiodocker already is one. DockerAPI keeps the same seam and method names; the transport is the Engine API, with daemon resolution at CLI parity (DOCKER_HOST, the active docker context, socket auto-detection). The client is created lazily on first use because providers are constructed in sync code and an aiohttp session must be born on the running loop. What the swap deletes: stderr parsing (404s map to the typed not-found errors), docker login (push carries per-call credentials in the X-Registry-Auth header — the global credential store is never touched), the env-file workaround (env rides the API body over the socket, and newline values now just work), and the docker binary in the image. The build context is a gzipped in-memory tar; the Engine builds it with the classic builder, which is enough for the synthesized three-line Dockerfiles. Smoke-tested live against a real daemon: version, derived image build, remove. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_translated hid the exception mapping behind a with-statement and a not-found flag parameter, and its broad ValueError/OSError arm converted genuine bugs into transport errors. Each method now carries its own narrow try/except in the codebase's boundary idiom; the no-daemon ValueError is translated in _client, the one place it can happen, and 404 branches exist only where not-found has a meaning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The built ref is called image everywhere the checklist reaches — Template.image, build_template_image, host.image — so the DockerAPI params, images.py locals, and derive_image_name stop saying tag. The only tag left is aiodocker's own build kwarg at the boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The abandoned-build sweep was a per-row locked loop for a pure status flip — now one guarded UPDATE ... RETURNING, the pool-claim pattern. The two delete sweeps and their sweep-spec tuple table collapse into a single reapable(settings) predicate that lives in the service: the janitor selects with it and delete(expired_only=True) re-checks the same predicate under the row lock, mirroring delete_host. The reap_status/reap_before parameter protocol is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OpenAPI contract check gains the four /templates operations, and the black-box suite gains the full template lifecycle against the real daemon: 202 build, poll to available, fork a host whose image is the template's, delete both. The image-validation step that asserted docker --version dies with the binary it verified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the drukbox half of the sandbox-needs design (ENG-877, ENG-878, ENG-879, ENG-880 — children of ENG-527). Boot-and-bake providers (ENG-881) come in a later PR.
Verification: ruff, ruff format, pyright, pytest (447) all green per commit.