Skip to content

feat(rewrite): rewrite multiline blocks - #3319

Open
aeppling wants to merge 5 commits into
developfrom
feat/rewrite-multiline-blocks
Open

feat(rewrite): rewrite multiline blocks#3319
aeppling wants to merge 5 commits into
developfrom
feat/rewrite-multiline-blocks

Conversation

@aeppling

@aeppling aeppling commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Multi-line Bash blocks (a shape coding agents emit constantly) were treated as one command: the rtk prefix landed on the first line and every following line ran raw and unfiltered. This PR splits blocks at the newline tokens the quote-aware lexer emits and rewrites each line through the existing single-line path. Implements the line-by-line rewrite proposed in #1243.

$ rtk rewrite 'git status
# check history
  grep -rn foo src'
rtk git status
# check history
  rtk grep -rn foo src
  • Lines continued by a trailing && / || / | / |& are joined and rewritten as one logical command (git status &&git log -3rtk git status && rtk git log -3), exactly what develop produced for the single-line form.
  • Quoted multi-line strings (multi-line commit messages) keep rewriting as one prefixed command, as develop always did.
  • Blank lines, comment lines, indentation, and CRLF separators are preserved verbatim.
  • Conservative by design — the whole block passes through untouched (native permission handling sees the original) whenever a line can't be attested: control flow keywords, heredocs (including << split across a \<NL> continuation), grouping/arithmetic spanning lines, swallowed newlines (quote state opened by an apostrophe in a comment), a continuation operator hidden behind a trailing comment, $'…' containing \' (the one ANSI-C case where the lexer's view diverges from bash), and |& stderr pipes.

How this was built

This PR supersedes #3188 (@revopsrocks), taken over after the author became unresponsive; their three commits are cherry-picked with authorship preserved.

  1. feat(rewrite): rewrite each line of multi-line Bash blocks #3188 as submitted — line-by-line rewrite + two adversarial review rounds by the original author (swallowed newlines, cross-line arithmetic, array literals/function bodies, comment-stripped independence checks, ANSI-C bail).
  2. Maintainer review fixes (review on feat(rewrite): rewrite each line of multi-line Bash blocks #3188, applied here):
    • narrowed the $' bail to the actual hazard (\' inside a $'…' span) so an unrelated echo $'a\tb' line no longer disables the block;
    • cross-line &&/||/| lists are now joined and rewritten instead of declined (develop already handled the joined form);
    • comment_start recognizes # after operator bytes (|&;()), so git log |# keepgrep -f patterns.txt passes through instead of rebuilding a guarded pipeline;
    • the heredoc/$(( gate runs on the bash-faithful continuation join (\<NL> → nothing) before the space-join normalization erases the operator.
  3. Differential fuzz round (below) — found one genuine regression vs develop (quoted multi-line strings declined) and fixed it: zero emitted newline tokens + balanced quote state at EOF proves the block is one logical command, which is rewritten whole; unbalanced quote state (the apostrophe-comment hazard) still declines.

Differential fuzz validation

971-case corpus (real per-ecosystem commands, compounds, multiline blocks, cross-line continuations, control flow, deep quoting, heredoc/redirect matrix, permission-bypass attempts, plus the #3188 adversarial cases), run 3-way through the real rtk rewrite hook path — develop vs this PR vs #2781 — with every rewrite validated by bash -n:

binary rewrote invalid rewrites crashes
develop 598/971 (61.6%) 0 0
this PR 684/971 (70.4%) 0 0
#2781 (2c89559) 717/971 (73.8%) 1 0
  • 105 blocks newly rewritten; 208 upgraded from partial to full (of 579 cases both rewrote, develop had prefixed only the first line in 208 — silent incomplete coverage the percentage never showed).
  • 19 deliberately declined that develop "rewrote": apostrophe-in-comment blocks (develop's output left lines 2+ unfiltered anyway), degraded comment-in-continuation joins, no-op trims.
  • Pipe-consumer protection verified: zero cases where any stage feeding a pipe gained an rtk prefix; cross-line pipes join into the same rewrite_compound machinery (final-stage-only, pipeline_final_command_is_safe, grep -f rejected, |& raw).
  • feat(rewrite): rewrite commands bundles #2781's coverage lead is partly mis-rewrites its own bash -n check can't see; this PR declines all of them: cat <\<EOFcat < <EOF (the corpus's only syntax error), $(\(1+2)) arithmetic corrupted into command substitution, rtk spliced inside a $'…' literal's value, grep -f rebuilt as the final stage of a pipeline behind a |# comment, and function bodies prefixed (changes x=$(foo) captured output).

Permission verdicts

Verified live against Claude Code allow/deny rules (.claude/settings.json): a block auto-allows (exit 0) only when every line independently matches an allow rule (#1213 aggregation); one unmatched line downgrades to ask; a deny match on any line wins outright; and a block whose newline was swallowed by quote state can never auto-allow — the same condition that blinds split_for_permissions forces the rewrite to decline, and exit 0 is impossible without a rewrite. split_for_permissions and the multiline splitter share the same newline-emitting tokenizer, so a verdict is always computed for exactly the lines being rewritten.

Test plan

  • cargo fmt --all && cargo clippy --all-targets && cargo test --all (2609 passed, clippy clean)
  • 971-case 3-way differential fuzz (above), every rewrite bash -n-validated
  • ~45 unit tests on the multiline path covering the rewrite and every passthrough gate

Follow-ups (out of scope here)

  • Replace the hand-rolled attestation checks with a real Bash grammar (tree-sitter-bash or a pure-Rust parser) — the apostrophe-comment cluster (18 declined cases, a very common agent shape) needs comment-aware parsing to rewrite safely. Being discussed with maintainers; startup-latency and binary-size budgets to be measured first.
  • Statement-level rewriting when a different line holds a redirect/substitution (feat(rewrite): rewrite commands bundles #2781's remaining legitimate coverage lead).
  • Check the fuzz corpus + runner into the repo as the regression suite for the above.

Ref #1243. Supersedes #3188. Related: #2781.

revopsrocks and others added 4 commits July 30, 2026 13:47
Coding agents routinely emit multi-line Bash blocks of sequential
commands. The rewriter tokenized newlines as plain whitespace, so a
block was treated as one command: the rtk prefix landed on the first
line and every following line ran raw and unfiltered.

Split multi-line input at the newline tokens the quote-aware lexer
emits (newlines inside quoted strings are never split points) and
rewrite each line through the existing single-line path. Blank lines,
comment lines, indentation, and CRLF separators are preserved verbatim.

Per-line rewriting only applies when every line is an independent
command. The whole block passes through untouched when:
- a shell keyword opens control flow (for/if/while/case/...)
- a list or pipeline continues across the line break (&&, ||, |, |&
  at a line edge)
- a subshell or group spans lines
- the block contains a heredoc (existing gate)

Ref #1243
Hardening from adversarial review of the multi-line rewrite:

- If any newline byte was swallowed by quote state (raw \n/\r count vs
  emitted newline tokens), pass the whole block through. The lexer has
  no comment awareness, so an apostrophe in a # comment opens quote
  state and hides subsequent lines; rewriting such a block would act on
  lines no permission verdict accounted for. Passthrough hands the
  original command to native permission handling. Quoted multi-line
  strings (commit messages) forgo their rewrite as the safe trade.
- Bail when (( or )) sits at a line edge: arithmetic spanning lines
  must not get a command prefix spliced into arithmetic context.
- Strip trailing unquoted comments before the independence checks:
  'git log | # keep pipeline' continues the pipeline across the
  newline even though the line ends in comment text, and rewriting
  the next line would rebuild a pipeline whose final stage the
  pipeline-safety guard may specifically reject (e.g. grep -f).
- Bail when a line's unquoted ()/{} don't balance: array literals
  (arr=(one), function bodies (foo() {), and groups span lines, so
  surrounding lines are not independent commands. Subsumes the
  previous single-char edge checks.
- Bail on any block containing $'...': inside ANSI-C quoting bash
  treats \' as a literal quote that does not close the string, but
  the lexer thinks it does — emitting a split point bash would never
  honor, the inverse of the swallowed-newline case the count check
  catches.
@aeppling
aeppling marked this pull request as draft July 30, 2026 12:23
@aeppling
aeppling marked this pull request as ready for review July 30, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants