Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Bug report
description: Report a reproducible env0 bug.
title: "[Bug]: "
labels:
- bug
body:
- type: textarea
id: what-happened
attributes:
label: What happened?
description: Describe the observed behavior and what you expected instead.
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Reproduction steps
description: Include commands, task name, service, and relevant output.
placeholder: |
1. Run ...
2. See ...
validations:
required: true
- type: input
id: version
attributes:
label: env0 version or commit
placeholder: "0.1.0 or git SHA"
- type: textarea
id: validation
attributes:
label: Validation already run
description: List relevant commands, for example scripts/smoke_dev.sh or a package pytest command.
39 changes: 39 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: Feature request
description: Propose an env0 runtime, mock-service, seed, or tooling improvement.
title: "[Feature]: "
labels:
- enhancement
body:
- type: textarea
id: problem
attributes:
label: Problem
description: What agent-testing or mock-environment workflow does this unblock?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposal
description: Describe the API, seed, tooling, or documentation change.
validations:
required: true
- type: dropdown
id: area
attributes:
label: Area
options:
- mock-gmail
- mock-gcal
- mock-gdoc
- mock-gdrive
- mock-slack
- dev tooling
- Docker/base image
- documentation
- other
- type: textarea
id: validation
attributes:
label: Validation plan
description: Which smoke, conformance, or Docker checks should cover this change?
13 changes: 13 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
## Summary

-

## Validation

- [ ] `scripts/smoke_dev.sh`
- [ ] Changed package tests, for example `cd packages/environments/mock-gdrive && uv run --extra dev pytest tests -q`
- [ ] Docker/base-image checks, if relevant: `docker/build-base.sh` and `PULL_BASE=0 scripts/smoke_docker_examples.sh`

## Notes

-
55 changes: 55 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
name: CI

on:
pull_request:
push:
branches:
- main

jobs:
repo-smoke:
name: Repo smoke
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@v5
- name: Run repo smoke
run: scripts/smoke_dev.sh

conformance:
name: Conformance (${{ matrix.package }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
package:
- mock-gmail
- mock-gcal
- mock-gdoc
- mock-gdrive
- mock-slack
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@v5
- name: Run conformance suite
working-directory: packages/environments/${{ matrix.package }}
run: uv run --extra dev pytest tests/test_conformance.py -q

docker-smoke:
name: Docker base and examples
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Build base image
run: docker/build-base.sh
- name: Smoke example task images
run: PULL_BASE=0 scripts/smoke_docker_examples.sh
27 changes: 27 additions & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Code Of Conduct

env0 is an open-source project for deterministic mock environments and agent
testing. Participation in this project should be professional, respectful, and
focused on improving the software.

## Expected Behavior

- Be direct and constructive in technical discussions.
- Assume good intent, but keep feedback grounded in reproducible facts.
- Respect maintainers' decisions about repo boundaries, security, and release
quality.
- Do not share private credentials, customer data, or private account exports in
issues, pull requests, discussions, or logs.

## Unacceptable Behavior

- Harassment, threats, discrimination, or personal attacks.
- Publishing private information or credentials.
- Spam, abusive automation, or intentionally disruptive behavior.
- Pressuring maintainers to disclose security details before a fix is ready.

## Enforcement

Maintainers may edit, hide, or remove content and may restrict participation for
behavior that violates this code. Security-sensitive issues should be reported
through `SECURITY.md`, not public issues.
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# env0

[![CI](https://github.com/benchflow-ai/env0/actions/workflows/ci.yml/badge.svg)](https://github.com/benchflow-ai/env0/actions/workflows/ci.yml)

env0 is the first-party mock-environment runtime for agent testing. It provides
stateful, deterministic mock services for local development, seed contracts,
API-parity checks, dev tooling, and a shared Docker base image.
Expand Down Expand Up @@ -146,6 +148,8 @@ still contain small service maps and must be kept in sync when adding services.
- [API validation playbook](docs/api-validation-playbook.md)
- [Parity audit](docs/parity-audit/README.md)
- [Validated workflows](docs/validated-workflows.md)
- [Contributing](CONTRIBUTING.md)
- [Security policy](SECURITY.md)

## Example Tasks

Expand Down
29 changes: 29 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Security Policy

## Reporting A Vulnerability

Please report suspected vulnerabilities privately by emailing security@benchflow.ai.
Include the affected service, commit or version, reproduction steps, impact, and
any logs that do not contain credentials.

Do not open public issues for vulnerabilities involving credential exposure,
authorization bypass, container escape, or access to private task payloads.

## Scope

Security reports are in scope for:

- mock service APIs under `packages/environments/mock-*`
- local launcher and devhub control paths
- task image payload isolation under `/var/lib/task`
- Docker base image build scripts and runtime defaults
- fixture capture scripts that handle live provider credentials

Reports about downstream benchmark scoring policy are usually out of scope for
this repo unless they expose an env0 runtime vulnerability.

## Credential Hygiene

Do not commit OAuth tokens, API keys, real account exports, provider credential
files, or private customer data. Fixture captures must be reviewed and sanitized
before publication.
8 changes: 1 addition & 7 deletions config.toml
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
# Project configuration — single source of truth for service ports and paths.
# All consumers (cli.py, backend/harbor.py, Dockerfiles, gws wrapper, vite) read from here.
# All env0 control scripts, Docker generation, devhub, and mock service CLIs read from here.
#
# [mock-*] sections: environment services
# gws_service: maps to gws CLI service name for the wrapper script.
# mock-gmail -> "gmail" -> MOCK_GMAIL_URL
# mock-gcal -> "calendar" -> MOCK_GCAL_URL
# Omit for services without a gws CLI counterpart (e.g., mock-slack).
#
# [clawsbench] section: dashboard port and output directory

[runtime]
version = 1

Expand Down Expand Up @@ -40,7 +38,3 @@ gws_service = "docs"
port = 9005
db_path = "/data/slack.db"
env_var = "MOCK_SLACK_URL"

[clawsbench]
port = 9050
output_dir = ".local/clawsbench-output"
2 changes: 1 addition & 1 deletion docker/generate_dockerfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ def generate(dry_run: bool = False) -> str:
result = result.replace("{{MOCK_ENV}}", env_block(cfg))

if dry_run:
print(result)
sys.stdout.write(result)
else:
OUT_PATH.write_text(result)
print(f"Generated {OUT_PATH}")
Expand Down
4 changes: 4 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ validation.
conformance status.
- [Validated workflows](validated-workflows.md) — commands that have been run
against this checkout and the intended preconditions for heavier commands.
- [Contributing](../CONTRIBUTING.md) — repo boundaries, validation matrix, and
pull request expectations.
- [Security policy](../SECURITY.md) — private vulnerability reporting and
credential hygiene.

## Boundaries

Expand Down
27 changes: 14 additions & 13 deletions docs/parity-audit/AUDIT_RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,19 @@ Historical source: initial parity audit generated 2026-03-27 by a 3-agent
audit council.

This file is kept as the initial env0 parity baseline. Notebook files have
been ported into this directory with `mock-*` path/name updates, but should be
rerun before treating their results as current gates.
been ported into this directory with `mock-*` path/name updates, but the current
release gate is the package conformance suites under
`packages/environments/mock-*/tests/test_conformance.py`.

## Summary

| Environment | Correctness | Completeness | Readability | Status |
|-------------|-------------|--------------|-------------|--------|
| mock-gmail | NEEDS FIX | MOSTLY COMPLETE | CLEAR | Needs refresh |
| mock-gcal | PASS | MOSTLY COMPLETE | CLEAR | OK |
| mock-gdoc | NEEDS FIX | COMPLETE | CLEAR | Needs refresh |
| mock-gdrive | NEEDS FIX | INCOMPLETE | NEEDS POLISH | Needs refresh |
| mock-slack | FAIL | MOSTLY COMPLETE | NEEDS POLISH | Needs refresh |
| Environment | Current release gate | Fixture count | Notes |
|-------------|----------------------|---------------|-------|
| mock-gmail | `uv run --extra dev pytest tests/test_conformance.py -q` | 35 | Current conformance suite passes. |
| mock-gcal | `uv run --extra dev pytest tests/test_conformance.py -q` | 31 | Current conformance suite passes. |
| mock-gdoc | `uv run --extra dev pytest tests/test_conformance.py -q` | 6 | Current conformance suite passes with documented skips. |
| mock-gdrive | `uv run --extra dev pytest tests/test_conformance.py -q` | 42 | Current conformance suite passes with documented skips. |
| mock-slack | `uv run --extra dev pytest tests/test_conformance.py -q` | 57 | Current conformance suite passes with documented skips. |

## Initial Must-Fix Items

Expand Down Expand Up @@ -50,10 +51,10 @@ Current env0 real golden fixture count:
- Slack: 57 fixtures
- Total: 171 golden fixtures

## env0 Refresh Needed
## Maintenance Items

- Verify all paths/names use current `mock-*` contracts.
- Verify fixture coverage maps under each `packages/environments/mock-*`.
- Add missing error response tests.
- Keep fixture coverage maps under each `packages/environments/mock-*` in sync
with route additions.
- Add missing error response tests when new error fixtures are captured.
- Add pagination tests where APIs support pagination.
- Re-run conformance suites after any fixture refresh.
5 changes: 3 additions & 2 deletions docs/parity-audit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,9 @@ The notebooks are included as starting points:
- `gdrive-parity-audit.ipynb`
- `slack-parity-audit.ipynb`

Treat them as review artifacts until each one is rerun in the env0 repo.
Before using a notebook as a gate, verify it:
Treat them as historical review artifacts. The current release gate is the
package conformance suite for each mock service. Before using a notebook as a
gate, verify it:

- load fixtures from `packages/environments/mock-*/tests/fixtures`
- read `mock_coverage.json`
Expand Down
4 changes: 2 additions & 2 deletions docs/parity-audit/gcal-parity-audit.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"**Discovery document:** https://www.googleapis.com/discovery/v1/apis/calendar/v3/rest\n",
"**Audit date:** 2026-03-26\n",
"\n",
"This notebook validates the API parity between the `mock-gcal` mock environment and the real Google Calendar REST API v3. It loads the endpoint spec, golden fixtures captured from a real Google Calendar account (`mediar.acc1@gmail.com`), and compares response shapes against the mock server using `fastapi.testclient.TestClient`.\n",
"This notebook validates the API parity between the `mock-gcal` mock environment and the real Google Calendar REST API v3. It loads the endpoint spec, golden fixtures captured from a real Google Calendar account (`fixture-user@example.com`), and compares response shapes against the mock server using `fastapi.testclient.TestClient`.\n",
"\n",
"**Data sources:**\n",
"- `tests/fixtures/gcal_api_spec.json` -- 38 endpoints from the official Calendar API\n",
Expand Down Expand Up @@ -49,7 +49,7 @@
"Endpoints with golden fixture: 25\n",
"Endpoints with tests: 38\n",
"Fixture files on disk: 29\n",
"Fixtures captured from: mediar.acc1@gmail.com\n",
"Fixtures captured from: fixture-user@example.com\n",
"Last capture date: 2026-03-27\n"
]
}
Expand Down
4 changes: 2 additions & 2 deletions docs/parity-audit/gmail-parity-audit.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"**Discovery document:** https://gmail.googleapis.com/$discovery/rest?version=v1 \n",
"**Audit date:** 2026-03-26\n",
"\n",
"This notebook validates the API parity between the `mock-gmail` mock environment and the real Gmail REST API v1. It loads the endpoint spec, golden fixtures captured from a real Gmail account (`mediar.acc1@gmail.com`), and compares response shapes against the mock server using `fastapi.testclient.TestClient`.\n",
"This notebook validates the API parity between the `mock-gmail` mock environment and the real Gmail REST API v1. It loads the endpoint spec, golden fixtures captured from a real Gmail account (`fixture-user@example.com`), and compares response shapes against the mock server using `fastapi.testclient.TestClient`.\n",
"\n",
"**Key concepts:**\n",
"- **Golden fixtures:** JSON responses captured from the real Gmail API that serve as the ground-truth reference for each endpoint's response structure.\n",
Expand Down Expand Up @@ -53,7 +53,7 @@
"Endpoints with golden fixture: 23\n",
"Endpoints with tests: 62\n",
"Fixture files on disk: 33\n",
"Fixtures captured from: mediar.acc1@gmail.com\n",
"Fixtures captured from: fixture-user@example.com\n",
"Last capture date: 2026-03-27\n"
]
}
Expand Down
4 changes: 4 additions & 0 deletions env0-mobile/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,7 @@ env0-mobile/
The root `tasks/` directory remains the small public/env0 reference set. Use
`tasks-eval` as the stable eval denominator, `tasks-train` as the broad training
pool, and `tasks-train-mini` as the compact training subset.

Manifest labels use service tags, not directory-family tags. For example,
directories that begin with `multi-` carry the service tags they exercise
(`gmail`, `gcal`, `gdrive`, `gdoc`, `slack`) instead of a separate `multi` tag.
2 changes: 1 addition & 1 deletion env0-mobile/tasks-eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ The task contents are real directories, not symbolic links, to keep the
evaluation set self-contained and easy to inspect inside env0. Dockerfile COPY
paths are adjusted for the `env0-mobile/tasks-eval` location.

Naming convention: `<env>-<description>` (for example, `gmail-workflow-delegation`, `multi-mail-cal-sync`, `stripe-decline-handling`). Categories such as `safety` are conveyed via tags in `task.toml`, not the directory name.
Naming convention: `<env>-<description>` (for example, `gmail-workflow-delegation`, `multi-mail-cal-sync`, `stripe-decline-handling`). Categories such as `safety` are conveyed via tags in `task.toml`, not the directory name. The `multi-` prefix is a directory-family convention; manifests use the underlying service tags rather than a separate `multi` tag.

| Task | Tags |
|------|------|
Expand Down
2 changes: 1 addition & 1 deletion env0-mobile/tasks-eval/_manifests/env-0.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# env-0 Environment-plane manifest (benchflow >= v0.6).
#
# One manifest serves the whole benchmark — the same pattern as benchflow's
# internal clawsbench dogfood benchmark: per-task images build FROM the shared
# env-0 task runtime: per-task images build FROM the shared
# base, bake their seed data at build time, and ship no service-starting
# entrypoint. benchflow's ManifestEnvironment reads this file, probes each
# service's CLI with `--help`, and starts ONLY the services actually installed
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts
RUN chown agent:agent /logs/agent /logs/artifacts

# NOTE: Daytona doesn't run CMD/ENTRYPOINT. gmail is started via
# harbor_runner.py's AGENT_START hook (see _add_direct_mode_hook).
# the env-0 runner AGENT_START hook.
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts
RUN chown agent:agent /logs/agent /logs/artifacts

# NOTE: Daytona doesn't run CMD/ENTRYPOINT. gmail is started via
# harbor_runner.py's AGENT_START hook (see _add_direct_mode_hook).
# the env-0 runner AGENT_START hook.
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts
RUN chown agent:agent /logs/agent /logs/artifacts

# NOTE: Daytona doesn't run CMD/ENTRYPOINT. gmail is started via
# harbor_runner.py's AGENT_START hook (see _add_direct_mode_hook).
# the env-0 runner AGENT_START hook.
Loading
Loading