Classifier cannot apply “created during current task” cleanup allowance because tool results are omitted

[NarryG]

Summary

The default policy allows deleting files created during the current task, but the classifier transcript excludes tool-result messages. As a result, the classifier cannot see evidence that a target was absent before the task or was created by a prior build/install command, causing false-positive cleanup blocks.

Environment

• pi-automode: 1.11.0
• OMP: 18.0.5
• OS: Windows 11
• Classifier: openai-codex/gpt-5.6-luna
• No custom allow/deny rules

The transcript behavior still exists on current main/1.13.0.

Real occurrence

The session:

1. Observed that node_modules did not exist.
2. Ran npm ci, creating node_modules.
3. Listed node_modules, confirming it contained task-created validation dependencies.
4. Attempted to remove those validation artifacts.

The classifier blocked the deletion:

Deletes the local node_modules directory, which may have existed before the session and was not specifically named by the user.

This conflicts with the built-in allowance:

Local development operations inside the working tree: ... deleting files created during the current task.

Minimal reproduction

Given session entries containing:

• user instruction to install dependencies and clean temporary artifacts
• assistant tool call npm ci
• tool result stating node_modules was newly created
• assistant read node_modules
• tool result confirming only task-created contents

buildClassifierTranscript() returns only:

User: Install the locked dependencies, run tests, and clean temporary validation artifacts.
ToolCall bash: { "command": "npm ci" }
ToolCall read: { "path": "node_modules" }

Both tool results—and therefore all artifact-provenance evidence—are missing.

Root cause

collectTranscriptEntries() accepts user messages and assistant tool calls but ignores tool-result messages. The classifier therefore cannot reliably evaluate the “created during the current task” exception.

Suggested direction

Either:

1. Include bounded, recent, explicitly untrusted tool-result evidence relevant to filesystem provenance and command outcomes; or
2. Maintain a deterministic per-session ledger of files/directories known to have been created by successful tool calls.

Tool-result text should remain clearly marked as untrusted data, not instructions.

[czottmann]

Thanks for the report, @NarryG. I thought about it and decided against adding new functionality because even though the proposed action is clear cut, that's only one problem solved, one action defined.

But there are many languages people work with, and I try to keep the plugin working for everyone. Every new action added requires clear semantics: shell contexts, workspaces and Windows paths need strict fail-closed rules; task boundaries must be defined. That's a game of whack-a-mole that can't be won. 😉

That said, I believe a new allow rule would do the trick.

Proposed workaround

Add a narrow classifier allow rule for deleting node_modules directories inside the working tree.

{
  "autoMode": {
    "allow": [
      "$defaults",
      "Deleting node_modules directories inside the working tree is allowed when the action only removes those directories and has no other side effects."
    ]
  }
}

Set this rule in autoMode.allow, not in permissions.allow. (A permissions.allow match bypasses classifier policy, including classifier hard_deny rules.)

For a user-wide rule, add it to ~/.pi/agent/extensions/pi-automode/config.json. For only one project, add it to .pi/automode.local.json in that project.

Keep $defaults in the list. Without it, the custom list replaces all built-in allow rules.

This rule treats node_modules as a regenerable dependency artifact. It should fix your reported cleanup case without exposing raw tool results to the classifier — but it does not solve provenance for arbitrary generated files.

Hope this helps!

[NarryG]

For what it's worth, this problem (tool calls being obscured) affects a lot more than this specific edge-case! I've seen pi-automode being overly aggressive in general because it doesn't know what has happened earlier in a session.

The node example was just one specific repro, I completely understand not wanting to make solutions for specific build toolchains, but the intention here is a generalized rule that tool results being obscured, obscures the classifier from knowing what was created by earlier tool calls.

[czottmann]

Reopening this because after pondering it some more, I agree you're on to something, @NarryG. While I'm still not certain that a sufficiently great implementation is doable, I think the current status quo can be improved. Let's discuss this…

---

The idea to have the classifier receive earlier tool results as historical context is good. Still, my point stands, there are so many different tools, and we can't anticipate everything.

The root cause you named is correct. collectTranscriptEntries() currently includes user instructions and assistant tool calls. But it intentionally excludes tool results.

What is feasible: Providing better evidence

Pi exposes structured data for completed tool calls:

• toolCallId
• toolName
• isError
• content
• details

The extension can match results to earlier tool calls through toolCallId. It can then add bounded result data to the classifier context.

The extension might not need to understand each tool, though. It could provide a generic result envelope and let the classifier interpret the evidence.

For example:

ToolResult:
{
  "toolName": "bash",
  "toolCallId": "…",
  "isError": false,
  "content": "bounded text excerpt"
}

Not feasible: Guaranteeing general provenance

Arbitrary bash(…) and custom tool results cannot prove the complete effects of a tool call.

The extension cannot reliably determine:

• which files the tool created;
• whether the output is correct or complete;
• whether background processes caused other changes;
• whether a path existed before the call;
• whether another process changed the path at the same time.

isError: false means only that the tool reported no error. It does not prove specific side effects.

A general created-artifact ledger would therefore create a false safety guarantee. A narrow ledger can work for tools with strict and known semantics.

For example, the extension can track some write(…) operations. This approach does not solve the reported npm ci case.

Idea

We could add a bounded outcome transcript that contains completed results from the active, compacted branch, toolName, toolCallId, and isError. Optionally includes a small text excerpt.

Data boundary risk

Tool results can contain secrets or file contents. A configured classifierModel can use a different provider. The change can therefore send data to a provider that did not previously receive it.

Before implementation, we'd need to define one data policy:

• Always send result text to the classifier.
• Maybe:
  • Send result text only within the same provider and data boundary.
  • Send result text only after explicit opt-in.

Structured metadata has less risk. However, metadata alone probably does not solve the overall issue.

Additional policy problem

The current policy uses two different boundaries:

• The allow rule refers to files created during the current task.
• The soft-deny rule refers to files that existed before the session started.

The extension does not define or store a task boundary. Additional result context reduces false positives, but it does not resolve this ambiguity.