Skip to content

Commit 3dfc865

Browse files
authored
Merge pull request #459 from posit-dev/feat-termshow-ci
feat: reproducible scripted recordings in CI, with `termshow check --json` and a workflow recipe
2 parents e39d5ab + 2377891 commit 3dfc865

8 files changed

Lines changed: 445 additions & 41 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -411,7 +411,7 @@ Beyond `init`, `build`, and `preview`, Great Docs includes a full suite of quali
411411
| `great-docs termshow render` | Render SVG frames without a full site build |
412412
| `great-docs termshow export` | Export a recording as one animated SVG for READMEs |
413413
| `great-docs termshow validate` | Check a recording and its script for mistakes |
414-
| `great-docs termshow check` | Re-run scripted steps and report recordings whose output has changed |
414+
| `great-docs termshow check` | Re-run scripted steps and report recordings whose output has changed (`--json` for CI) |
415415
| `great-docs termshow assets` | Copy the web player for pages built without Quarto |
416416
| `great-docs termshow import-cast` | Import an asciinema `.cast` file |
417417
| `great-docs termshow import-tape` | Convert a VHS `.tape` file to a script with `steps:` |

‎docs/user_guide/41-terminal-recordings.qmd‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1161,6 +1161,10 @@ The `record` section sets up the shell:
11611161
- `cwd` is the directory to start in, relative to the script.
11621162
- `env` sets environment variables. The shell starts with a plain `$ ` prompt and no history
11631163
file, so nothing you record ends up in your own history.
1164+
- `inherit_env` (`false`) passes on your whole environment. By default the shell only gets a few
1165+
variables from it, such as `PATH`, `HOME`, and `LANG`, so the recording doesn't pick up
1166+
whatever your shell profile, editor, or CI service has set. That's what keeps a recording
1167+
made on your machine the same as one made in CI. List anything else the steps need in `env`.
11641168
- `typing_delay` (0.05 seconds) and `typing_jitter` (0.3) set the typing speed and how much it
11651169
varies. `delay` on a `type`, `run`, or `key` step overrides the speed for that step.
11661170
- `timeout` (10 seconds) is how long to wait for a prompt or a pattern. `timeout` on a `run` or
@@ -1245,6 +1249,101 @@ the next marker isn't. A `marker` between commands compares each command's outpu
12451249
next one pushes it away, and a taller terminal (`record.rows`) helps with commands that print a
12461250
lot. The check runs at the recording's size, whatever the script's `record` section says now.
12471251

1252+
### Keeping demos up to date in CI
1253+
1254+
This GitHub Actions workflow does two jobs. On every pull request it checks the demos, and the
1255+
job fails if one has drifted, with the diff in its log. Once a week, or when you run it by hand,
1256+
it records the drifted demos again and opens a pull request with the new recordings, so a
1257+
release that changes your tool's output leads to a demo update you only have to review:
1258+
1259+
```{.yaml filename=".github/workflows/demos.yml"}
1260+
name: Terminal demos
1261+
1262+
on:
1263+
pull_request:
1264+
schedule:
1265+
- cron: "0 6 * * 1"
1266+
workflow_dispatch:
1267+
1268+
jobs:
1269+
check:
1270+
if: github.event_name == 'pull_request'
1271+
runs-on: ubuntu-latest
1272+
steps:
1273+
- uses: actions/checkout@v4
1274+
- uses: actions/setup-python@v5
1275+
with:
1276+
python-version: "3.12"
1277+
- run: pip install great-docs .
1278+
- run: great-docs termshow check docs/assets/*.termshow.yml
1279+
1280+
update:
1281+
if: github.event_name != 'pull_request'
1282+
runs-on: ubuntu-latest
1283+
permissions:
1284+
contents: write
1285+
pull-requests: write
1286+
steps:
1287+
- uses: actions/checkout@v4
1288+
- uses: actions/setup-python@v5
1289+
with:
1290+
python-version: "3.12"
1291+
- run: pip install great-docs .
1292+
- run: great-docs termshow check docs/assets/*.termshow.yml --update
1293+
# For recordings that a README shows as an animated SVG
1294+
- run: |
1295+
for recording in docs/assets/*.termshow; do
1296+
great-docs termshow export "$recording"
1297+
done
1298+
- uses: peter-evans/create-pull-request@v7
1299+
with:
1300+
branch: update-terminal-demos
1301+
commit-message: Record the terminal demos again
1302+
title: Update the terminal demos
1303+
body: "`great-docs termshow check --update` found demos whose output has changed."
1304+
```
1305+
1306+
`pip install great-docs .` installs your package along with Great Docs, so the steps run the
1307+
version of your tool in the pull request. The `update` job can only open a pull request if the
1308+
repository allows it: turn on **Allow GitHub Actions to create and approve pull requests** under
1309+
**Settings → Actions → General**. Both jobs fail if a step fails, such as a command that isn't
1310+
installed on the runner.
1311+
1312+
A recording made on your Mac and checked on a Linux runner can differ in ways that have nothing
1313+
to do with your tool: `sh` is a different shell on each, and paths and some commands' output
1314+
differ too. Two things help. Set `record.shell` to `bash --norc` in each script, so the same
1315+
shell runs everywhere. And let the `update` job make the recordings, so the ones you commit come
1316+
from the same kind of machine that checks them.
1317+
1318+
To use the results in another tool, or in a job summary, add `--json`. The output is one JSON
1319+
object, and the exit status is the same as without it:
1320+
1321+
```bash
1322+
great-docs termshow check docs/assets/*.termshow.yml --json > drift.json
1323+
```
1324+
1325+
```json
1326+
{
1327+
"version": 1,
1328+
"summary": {"up_to_date": 0, "drifted": 1, "updated": 0, "failed": 0},
1329+
"results": [
1330+
{
1331+
"script": "docs/assets/install-guide.termshow.yml",
1332+
"recording": "docs/assets/install-guide.termshow",
1333+
"status": "drifted",
1334+
"compared": 3,
1335+
"differences": [
1336+
{"at": "marker 'Verify'", "diff": ["--- install-guide.termshow, at marker 'Verify'", "..."]}
1337+
]
1338+
}
1339+
]
1340+
}
1341+
```
1342+
1343+
`status` is `up_to_date`, `drifted`, `updated` (with `--update`), or `failed`. A failed result
1344+
has an `error` message instead of `differences`, and a `screen` with the screen's text when a
1345+
step timed out.
1346+
12481347
## Importing Existing Recordings
12491348

12501349
### From asciinema

‎docs/user_guide/42-termshow-file-formats.qmd‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -223,13 +223,17 @@ them as written when it saves.
223223
| `rows` | `24` | Terminal height |
224224
| `cwd` | the script's directory | Working directory, relative to the script's directory |
225225
| `env` | (none) | Environment variables to set. Values may be strings, numbers, or `true`/`false` |
226+
| `inherit_env` | `false` | Start the shell with the whole current environment, not just the variables listed below |
226227
| `typing_delay` | `0.05` | Seconds between typed characters |
227228
| `typing_jitter` | `0.3` | How much each delay varies, as a fraction of it (0 to 1) |
228229
| `timeout` | `10` | Seconds to wait for the prompt or a `wait_for` pattern |
229230
| `prompt_pattern` | `\$ $` | Regular expression (Python syntax) that matches the shell's prompt |
230231

231-
The shell starts with `PS1="$ "`, `TERM=xterm-256color`, and no history file. `env` can change
232-
any of these. `COLUMNS` and `LINES` are removed so the terminal size is `cols` by `rows`.
232+
Only `PATH`, `HOME`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TMPDIR`, `TZ`, and `VIRTUAL_ENV` are
233+
taken from the current environment, unless `inherit_env` is `true`. `SHELL` is set to the shell
234+
being recorded. The shell then gets `PS1="$ "`, `TERM=xterm-256color`, and no history file, and
235+
`env` can change any of these. With `inherit_env`, `COLUMNS` and `LINES` are still removed so
236+
the terminal size is `cols` by `rows`, and `SHELL` is left as it is.
233237

234238
Each step is one of these words or single-action mappings:
235239

‎great_docs/_term_player/schemas/termshow-script.schema.json‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -408,10 +408,15 @@
408408
"type": "string"
409409
},
410410
"env": {
411-
"description": "Environment variables to set on top of the current environment.",
411+
"description": "Environment variables to set for the shell.",
412412
"type": "object",
413413
"additionalProperties": { "type": ["string", "number", "boolean"] }
414414
},
415+
"inherit_env": {
416+
"description": "Start the shell with the whole current environment. By default it only gets PATH, HOME, USER, LOGNAME, LANG, LC_*, TMPDIR, TZ, and VIRTUAL_ENV, so the recording doesn't depend on what your shell, editor, or CI service has set.",
417+
"type": "boolean",
418+
"default": false
419+
},
415420
"typing_delay": {
416421
"description": "Seconds between typed characters.",
417422
"type": "number",

‎great_docs/_term_player/steps.py‎

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
import re
1515
import select
1616
import shlex
17+
import shutil
1718
import sys
1819
import time
1920
from collections.abc import Callable
@@ -61,6 +62,9 @@
6162
"HISTFILE": "",
6263
"BASH_SILENCE_DEPRECATION_WARNING": "1",
6364
}
65+
# Kept from the current environment unless `record.inherit_env` is set, so a recording doesn't
66+
# depend on what your own shell, editor, or CI service has set. `LC_*` variables are kept too.
67+
KEPT_ENV = ("PATH", "HOME", "USER", "LOGNAME", "LANG", "TMPDIR", "TZ", "VIRTUAL_ENV")
6468
# Removed from the environment, since they'd override the terminal size
6569
_SIZE_VARS = ("COLUMNS", "LINES")
6670

@@ -81,6 +85,22 @@ def key_sequence(name: str) -> str | None:
8185
return None
8286

8387

88+
def shell_env(settings: RecordSettings, environ: dict[str, str] | None = None) -> dict[str, str]:
89+
"""The environment the shell starts with.
90+
91+
Only `KEPT_ENV` and `LC_*` are taken from `environ` (the current environment), unless
92+
`settings.inherit_env` is set. `DEFAULT_ENV` and the script's `record.env` go on top.
93+
"""
94+
environ = dict(os.environ if environ is None else environ)
95+
if settings.inherit_env:
96+
env = {k: v for k, v in environ.items() if k not in _SIZE_VARS}
97+
else:
98+
env = {k: v for k, v in environ.items() if k in KEPT_ENV or k.startswith("LC_")}
99+
env.update(DEFAULT_ENV)
100+
env.update(settings.env)
101+
return env
102+
103+
84104
def screen_text(emulator: TerminalEmulator) -> str:
85105
"""The screen's text, with trailing spaces removed from each row and blank rows at the end
86106
left out."""
@@ -108,6 +128,7 @@ class RecordSettings:
108128
rows: int = 24
109129
cwd: Path | None = None
110130
env: dict[str, str] = field(default_factory=dict)
131+
inherit_env: bool = False
111132
typing_delay: float = 0.05
112133
typing_jitter: float = 0.3
113134
timeout: float = 10.0
@@ -324,9 +345,10 @@ def start(self) -> None:
324345
argv = shlex.split(self.settings.shell)
325346
except ValueError as e:
326347
raise StepsError(f"can't read record.shell: {e}") from None
327-
env = {k: v for k, v in os.environ.items() if k not in _SIZE_VARS}
328-
env.update(DEFAULT_ENV)
329-
env.update(self.settings.env)
348+
env = shell_env(self.settings)
349+
if not self.settings.inherit_env:
350+
# Programs that start a subshell use the one being recorded
351+
env.setdefault("SHELL", shutil.which(argv[0]) or argv[0])
330352
winsize = struct.pack("HHHH", self.settings.rows, self.settings.cols, 0, 0)
331353

332354
pid, fd = pty.fork()

‎great_docs/cli.py‎

Lines changed: 74 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -4614,7 +4614,16 @@ def _has_steps(script: str) -> bool:
46144614
)
46154615
@click.option("--update", is_flag=True, help="Re-record the recordings that have drifted")
46164616
@click.option("--shell", type=str, default=None, help="Run the steps in this shell instead")
4617-
def term_check(scripts: tuple[str, ...], ignore: tuple[str, ...], update: bool, shell: str | None):
4617+
@click.option(
4618+
"--json", "as_json", is_flag=True, help="Print the results as JSON, for CI and other tools"
4619+
)
4620+
def term_check(
4621+
scripts: tuple[str, ...],
4622+
ignore: tuple[str, ...],
4623+
update: bool,
4624+
shell: str | None,
4625+
as_json: bool,
4626+
):
46184627
"""Check that recordings made from steps still match what the steps produce.
46194628
46204629
Each script's steps are run again, as `record --steps` would, and the
@@ -4629,10 +4638,15 @@ def term_check(scripts: tuple[str, ...], ignore: tuple[str, ...], update: bool,
46294638
Exits with status 1 if any recording has drifted or a step failed. With
46304639
--update, recordings that have drifted are recorded again instead.
46314640
4641+
With --json, the results are printed as one JSON object instead, with
4642+
the same exit status. Progress still goes to stderr.
4643+
46324644
Examples:
46334645
great-docs termshow check docs/assets/*.termshow.yml
46344646
great-docs termshow check docs/assets/install.termshow --ignore 'v\\d+\\.\\d+'
4647+
great-docs termshow check docs/assets/*.termshow.yml --json > drift.json
46354648
"""
4649+
import json
46364650
import re
46374651

46384652
from ._term_player.check import CheckError, check_script
@@ -4647,54 +4661,80 @@ def term_check(scripts: tuple[str, ...], ignore: tuple[str, ...], update: bool,
46474661
f"{pattern!r} isn't a valid regular expression ({e})", param_hint="--ignore"
46484662
) from None
46494663

4650-
drifted = failed = 0
4664+
results = []
46514665
for script in scripts:
46524666
script_path = Path(script)
46534667
if script_path.name.endswith(".termshow"):
46544668
script_path = script_path.with_name(script_path.name + ".yml")
46554669
click.echo(f"Checking {script_path}", err=True)
4670+
entry = {"script": script_path.as_posix(), "recording": None}
46564671
try:
46574672
result = check_script(script_path, ignore=ignore, shell=shell)
4658-
except CheckError as e:
4659-
click.echo(f"✗ {script_path}: {e}")
4660-
failed += 1
4661-
continue
4662-
except StepsError as e:
4663-
click.echo(f"✗ {script_path}: {e}")
4664-
if e.screen:
4665-
click.echo("\nThe screen at the time:")
4666-
for line in e.screen.splitlines():
4667-
click.echo(f" │ {line}")
4668-
failed += 1
4673+
except (CheckError, StepsError) as e:
4674+
entry |= {"status": "failed", "error": str(e)}
4675+
if isinstance(e, StepsError) and e.screen:
4676+
entry["screen"] = e.screen
4677+
results.append(entry)
4678+
if not as_json:
4679+
_echo_check_failure(entry)
46694680
continue
46704681

4671-
screens = f"{result.snapshots} screen{'' if result.snapshots == 1 else 's'}"
4672-
if not result.drifted:
4673-
click.echo(f"✓ {result.recording_path} is up to date ({screens} compared)")
4674-
continue
4675-
drifted += 1
4676-
click.echo(f"✗ {result.recording_path} has drifted:\n")
4677-
for difference in result.differences:
4678-
for line in difference.lines:
4679-
color = {"+": "green", "-": "red", "@": "cyan"}.get(line[:1])
4680-
if line.startswith(("+++", "---")):
4681-
color = None
4682-
click.secho(f" {line}", fg=color)
4683-
click.echo()
4684-
if update:
4682+
entry |= {
4683+
"recording": result.recording_path.as_posix(),
4684+
"status": "drifted" if result.drifted else "up_to_date",
4685+
"compared": result.snapshots,
4686+
"differences": [{"at": d.where, "diff": d.lines} for d in result.differences],
4687+
}
4688+
if result.drifted and update:
46854689
_write_termshow(result.recording, result.recording_path)
4686-
click.echo(f"↻ Recorded {result.recording_path} again")
4687-
4688-
if len(scripts) > 1:
4690+
entry["status"] = "updated"
4691+
results.append(entry)
4692+
if not as_json:
4693+
_echo_check_result(entry)
4694+
4695+
counts = {
4696+
status: sum(r["status"] == status for r in results)
4697+
for status in ("up_to_date", "drifted", "updated", "failed")
4698+
}
4699+
if as_json:
4700+
click.echo(json.dumps({"version": 1, "summary": counts, "results": results}, indent=2))
4701+
elif len(scripts) > 1:
4702+
changed = counts["drifted"] + counts["updated"]
46894703
click.echo(
4690-
f"\n{len(scripts) - drifted - failed} of {len(scripts)} up to date"
4691-
+ (f", {drifted} {'updated' if update else 'drifted'}" if drifted else "")
4692-
+ (f", {failed} couldn't be checked" if failed else "")
4704+
f"\n{counts['up_to_date']} of {len(scripts)} up to date"
4705+
+ (f", {changed} {'updated' if update else 'drifted'}" if changed else "")
4706+
+ (f", {counts['failed']} couldn't be checked" if counts["failed"] else "")
46934707
)
4694-
if failed or (drifted and not update):
4708+
if counts["failed"] or counts["drifted"]:
46954709
raise SystemExit(1)
46964710

46974711

4712+
def _echo_check_failure(entry: dict) -> None:
4713+
click.echo(f"✗ {entry['script']}: {entry['error']}")
4714+
if entry.get("screen"):
4715+
click.echo("\nThe screen at the time:")
4716+
for line in entry["screen"].splitlines():
4717+
click.echo(f" │ {line}")
4718+
4719+
4720+
def _echo_check_result(entry: dict) -> None:
4721+
recording = entry["recording"]
4722+
if entry["status"] == "up_to_date":
4723+
screens = f"{entry['compared']} screen{'' if entry['compared'] == 1 else 's'}"
4724+
click.echo(f"✓ {recording} is up to date ({screens} compared)")
4725+
return
4726+
click.echo(f"✗ {recording} has drifted:\n")
4727+
for difference in entry["differences"]:
4728+
for line in difference["diff"]:
4729+
color = {"+": "green", "-": "red", "@": "cyan"}.get(line[:1])
4730+
if line.startswith(("+++", "---")):
4731+
color = None
4732+
click.secho(f" {line}", fg=color)
4733+
click.echo()
4734+
if entry["status"] == "updated":
4735+
click.echo(f"↻ Recorded {recording} again")
4736+
4737+
46984738
@click.command("assets")
46994739
@click.argument("dest", type=click.Path(file_okay=False))
47004740
def term_assets(dest: str) -> None:

0 commit comments

Comments
 (0)