feat(rewrite): rewrite multiline blocks - #3319
Open
aeppling wants to merge 5 commits into
Open
Conversation
Coding agents routinely emit multi-line Bash blocks of sequential commands. The rewriter tokenized newlines as plain whitespace, so a block was treated as one command: the rtk prefix landed on the first line and every following line ran raw and unfiltered. Split multi-line input at the newline tokens the quote-aware lexer emits (newlines inside quoted strings are never split points) and rewrite each line through the existing single-line path. Blank lines, comment lines, indentation, and CRLF separators are preserved verbatim. Per-line rewriting only applies when every line is an independent command. The whole block passes through untouched when: - a shell keyword opens control flow (for/if/while/case/...) - a list or pipeline continues across the line break (&&, ||, |, |& at a line edge) - a subshell or group spans lines - the block contains a heredoc (existing gate) Ref #1243
Hardening from adversarial review of the multi-line rewrite: - If any newline byte was swallowed by quote state (raw \n/\r count vs emitted newline tokens), pass the whole block through. The lexer has no comment awareness, so an apostrophe in a # comment opens quote state and hides subsequent lines; rewriting such a block would act on lines no permission verdict accounted for. Passthrough hands the original command to native permission handling. Quoted multi-line strings (commit messages) forgo their rewrite as the safe trade. - Bail when (( or )) sits at a line edge: arithmetic spanning lines must not get a command prefix spliced into arithmetic context.
- Strip trailing unquoted comments before the independence checks:
'git log | # keep pipeline' continues the pipeline across the
newline even though the line ends in comment text, and rewriting
the next line would rebuild a pipeline whose final stage the
pipeline-safety guard may specifically reject (e.g. grep -f).
- Bail when a line's unquoted ()/{} don't balance: array literals
(arr=(one), function bodies (foo() {), and groups span lines, so
surrounding lines are not independent commands. Subsumes the
previous single-char edge checks.
- Bail on any block containing $'...': inside ANSI-C quoting bash
treats \' as a literal quote that does not close the string, but
the lexer thinks it does — emitting a split point bash would never
honor, the inverse of the swallowed-newline case the count check
catches.
aeppling
marked this pull request as draft
July 30, 2026 12:23
aeppling
marked this pull request as ready for review
July 30, 2026 16:31
pszymkowiak
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Multi-line Bash blocks (a shape coding agents emit constantly) were treated as one command: the
rtkprefix landed on the first line and every following line ran raw and unfiltered. This PR splits blocks at the newline tokens the quote-aware lexer emits and rewrites each line through the existing single-line path. Implements the line-by-line rewrite proposed in #1243.&&/||/|/|&are joined and rewritten as one logical command (git status &&⏎git log -3→rtk git status && rtk git log -3), exactly what develop produced for the single-line form.<<split across a\<NL>continuation), grouping/arithmetic spanning lines, swallowed newlines (quote state opened by an apostrophe in a comment), a continuation operator hidden behind a trailing comment,$'…'containing\'(the one ANSI-C case where the lexer's view diverges from bash), and|&stderr pipes.How this was built
This PR supersedes #3188 (@revopsrocks), taken over after the author became unresponsive; their three commits are cherry-picked with authorship preserved.
$'bail to the actual hazard (\'inside a$'…'span) so an unrelatedecho $'a\tb'line no longer disables the block;&&/||/|lists are now joined and rewritten instead of declined (develop already handled the joined form);comment_startrecognizes#after operator bytes (|&;()), sogit log |# keep⏎grep -f patterns.txtpasses through instead of rebuilding a guarded pipeline;$((gate runs on the bash-faithful continuation join (\<NL>→ nothing) before the space-join normalization erases the operator.Differential fuzz validation
971-case corpus (real per-ecosystem commands, compounds, multiline blocks, cross-line continuations, control flow, deep quoting, heredoc/redirect matrix, permission-bypass attempts, plus the #3188 adversarial cases), run 3-way through the real
rtk rewritehook path — develop vs this PR vs #2781 — with every rewrite validated bybash -n:rtkprefix; cross-line pipes join into the samerewrite_compoundmachinery (final-stage-only,pipeline_final_command_is_safe,grep -frejected,|&raw).bash -ncheck can't see; this PR declines all of them:cat <\⏎<EOF→cat < <EOF(the corpus's only syntax error),$(\⏎(1+2))arithmetic corrupted into command substitution,rtkspliced inside a$'…'literal's value,grep -frebuilt as the final stage of a pipeline behind a|# comment, and function bodies prefixed (changesx=$(foo)captured output).Permission verdicts
Verified live against Claude Code allow/deny rules (
.claude/settings.json): a block auto-allows (exit 0) only when every line independently matches an allow rule (#1213 aggregation); one unmatched line downgrades to ask; a deny match on any line wins outright; and a block whose newline was swallowed by quote state can never auto-allow — the same condition that blindssplit_for_permissionsforces the rewrite to decline, and exit 0 is impossible without a rewrite.split_for_permissionsand the multiline splitter share the same newline-emitting tokenizer, so a verdict is always computed for exactly the lines being rewritten.Test plan
cargo fmt --all && cargo clippy --all-targets && cargo test --all(2609 passed, clippy clean)bash -n-validatedFollow-ups (out of scope here)
Ref #1243. Supersedes #3188. Related: #2781.