feat(tool): support cache-friendly on-demand tool loading

[iluv7]

Problem

AgentScope's ToolGroup uses reset_tools to switch active tool groups dynamically, reducing the number of tools visible to the model in each call. reset_tools does not return or append the newly activated tools' schemas itself. It only updates the activation state and returns the activated groups' usage instructions as a normal tool result.

The complete schemas are added separately when AgentScope prepares the next model call:

assistant calls reset_tools(...)
  -> ResetTools.call() updates activated_groups
  -> reset_tools returns group usage instructions
  -> the tool result is appended to conversation history
  -> the agent starts the next reasoning iteration
  -> Agent._prepare_model_input() rebuilds tools from activated_groups
  -> the next model request contains different top-level tool schemas

The relevant implementation points are:

• ResetTools.call() updates activated_groups and returns group usage instructions, not tool schemas;
• Agent._prepare_model_input() regenerates tools from the active groups;
• Toolkit._get_available_tools() collects basic tools, tools from active groups, and MCP tools.

AgentScope eventually constructs the following input:

{
    "messages": messages,
    "tools": tools,
}

Although tools appears after messages in this Python dictionary, that does not mean tool definitions are placed at the end of the model's effective input. Each provider parses and organizes these fields independently.

The appended reset_tools result is cache-friendly by itself. The cache boundary is introduced by the separate reconstruction of the top-level tools before the next request. The core issue is therefore:

Calling reset_tools changes activated_groups. Before the next model call, AgentScope rebuilds the top-level tools from that state, changing the input prefix seen by the provider and reducing prompt/KV cache reuse.

Why this affects caching

Prompt caching relies on a continuous, identical input prefix. Appending new content to the end of the conversation is cache-friendly:

old: stable prefix + history
new: stable prefix + history + new message + tool result

Changing tool definitions may instead introduce a difference before the conversation history:

old: tools[A]    + system + history
new: tools[A, B] + system + history

Anthropic's documentation explicitly describes its cache hierarchy as:

tools -> system -> messages

Changing tool definitions invalidates the tools, system, and messages caches. Therefore, with Anthropic, the current reset_tools behavior creates an explicit cache boundary.

References:

• Prompt caching
• Tool use with prompt caching

Proposed approach

With on-demand tool loading, newly discovered tool information can be appended to the conversation history without changing the top-level tools. I suggest keeping the current behavior while adding a provider-independent stable mode:

tool_loading_mode: Literal["native", "stable"] = "native"

• native: keep using ToolGroup + reset_tools to expose real tools dynamically;
• stable: expose only a fixed search_tools + execute_tool pair at the top level.

fixed tools
├─ search_tools(query)
└─ execute_tool(tool_name, arguments)

end of conversation history
├─ search_tools call
├─ search result: tool name, description, and complete schema
├─ execute_tool call
└─ real tool result

search_tools searches AgentScope's complete tool registry and appends the candidates as a tool result. execute_tool resolves the real tool by name and reuses the existing Toolkit execution path.

Tool recall in long contexts

As a conversation grows, earlier tool search results move toward the middle of the context, where the model may use their schemas less reliably. Context compaction may also remove these results entirely. The stable mode should therefore not depend on the model remembering discovered tools indefinitely.

• search_tools should support repeated keyword search and direct loading by tool name;
• when the tool name or its arguments are uncertain, the model can search again so that the relevant schema appears near the end of the history;
• if execute_tool argument validation fails, it should return a clear hint telling the model to search again and retry;
• context compaction can preserve lightweight state such as discovered tool names, while complete schemas are retrieved again when needed.

Repeated searches only append content to the conversation and do not modify the existing cached prefix.

The top-level tools therefore remain unchanged before and after discovery:

old request: search_tools + execute_tool
new request: search_tools + execute_tool

This approach only changes how tools are exposed to the model. It does not change AgentScope's existing tool execution logic and does not require support for any provider-specific deferred-loading protocol. The default remains native for backward compatibility.

This is my current understanding of the problem and a possible optimization direction. I may have missed something or misunderstood part of the behavior, so feedback and discussion are very welcome.

[DavdGao]

@iluv7 Great point. When designing the new tool_choice argument, we did notice this issue.

The OpenAI Response API supports mask-based tool choice — it sends the full JSON schema upfront but controls which tools are active via masks during inference, preserving the KV cache. This is actually why we evolved tool_choice from a simple string enum to the current struct design. This approach aligns naturally with ResetTools semantics, and to some extent we've been watching whether this paradigm gains broader API adoption.

That said, mask-based caching is currently only supported by the Response API, which is why calling ResetTools still breaks the cache for other providers. I'd personally consider it the more principled approach — though it comes with a real tradeoff: all tool schemas are loaded into the prompt upfront, including tools that may never be invoked, and that prompt cache cost is unavoidable.

Bringing it back to your proposal — I find the idea genuinely interesting. Delegating tool discovery to the agent itself is a thoughtful application-layer solution. That said, I have a couple of concerns:

• Injecting JSON schemas into tool execution results isn't fundamentally different from placing them in the system prompt, and its effect on model behavior is uncertain — training has likely baked in strong assumptions about where and how tool schemas appear in context, so this shift is worth validating carefully.
• Beyond that, in scenarios where all required tools come from the dynamic set, correctness becomes entirely dependent on search_tool quality and query precision; without global tool awareness, the agent may fail to locate the right tool at a critical step, and the invocation frequency could become non-trivial, adding meaningful latency.

That said, I think this direction is worth pursuing. Let me sketch a more complete design, grounded in two principles: the agent should have deterministic awareness of its global tool inventory — a stable understanding of its capability boundaries — and deterministic awareness of currently active tool schemas at runtime.

Assuming the total tool count is fixed and tools are pre-organized into named groups (dynamic registration is a separate discussion), the tools parameter continues to carry the "basic" group as before. Since the overall tool set is fixed, ResetTools's own schema doesn't change either, so it lives in the basic group and stays cache-stable. When ResetTools is called, in addition to returning its normal result, we append a system-level notification immediately after:

<system-reminder>
Active tool group: [xxx]. Newly activated tools (+N schemas):
[{"type": "function", "function": {...}}, ...]
</system-reminder>

This avoids any dependency on search_tool quality, gives the agent explicit awareness of its capability boundaries so it can cleanly reject out-of-scope tasks rather than issuing repeated failed searches, and keeps the static portion cache-friendly. The two obvious problems are that ResetTools is a full reset rather than a delta, so repeatedly switching groups causes already-active schemas to be re-emitted and bloats the context; and once context compression kicks in, the agent loses track of what it currently has equipped.

To address both, we could replace ResetTools with incremental primitives — ToolGroupActivate and ToolGroupDeactivate — where each ToolGroupActivate call only injects schemas for newly activated tools. And whenever context compression occurs, the post-summary context should always be augmented with the current active group state and the full schemas of all active tools, so the agent's tool awareness survives compression.

It's worth being upfront about what this demands from the model:

• It needs to correctly infer its current tool loadout from an accumulated sequence of activation and deactivation events — that temporal state-tracking ability is really the load-bearing assumption the whole scheme rests on.
• Beyond that, stepping outside the tools API also means giving up the structured output validation guarantees that API providers offer.
• Some providers may actively validate that any tool call the model emits is declared in the tools field — and that check would fail here. This is actually where the execute_tool primitive in your proposal starts to show its value.

---

很好的观察。在设计新的 tool_choice 参数时，我们确实注意到了这个问题。

OpenAI Response API 支持基于掩码的工具选择——它一次性发送完整的 JSON schema，但在推理过程中通过掩码控制哪些工具处于激活状态，从而保留 KV cache。这也是我们将 tool_choice 从简单的字符串枚举升级为当前结构体设计的原因。这种方式在设计上与 ResetTools 的语义天然契合——某种程度上，我们也在观望这一范式能否获得更广泛的 API 支持。

不过，掩码缓存目前仅 Response API 支持，这也是为什么对其他 provider 调用 ResetTools 仍然会破坏缓存。我个人认为这是更本质的解法——尽管它也有代价：所有工具 schema 都会预先写入 prompt，包括那些可能始终不会被调用的工具，这部分 prompt cache 的开销是实打实存在的。

回到你的提案——我觉得这个思路很有意思，把工具发现的能力委托给 agent 自身，是一种很好的应用层解法。但我有几点顾虑：

首先，将 JSON schema 后置到工具执行结果中，本质上和放在 system prompt 里的差异并不大，但对模型行为的实际影响目前还不确定：

1. 模型训练对工具 schema 的位置和组织方式有较强的预设，这个变化的影响值得验证。
2. 其次，在所需工具完全来自动态工具集的场景下，效果完全依赖 search_tool 的质量和 query 的准确度；
3. 缺乏全局工具认知的情况下，agent 在某个关键步骤很可能找不到合适的工具，且调用频次可能相当高，带来额外的延迟。

这个方向值得深入探索。我想在此基础上提出一个更完整的设计思路，核心围绕两条原则：

1. agent 需要对工具全局有确定性的认知，即对自身能力边界的稳定认知；
2. agent 需要对当前可用工具的 schema 有确定性的认知，即运行时能力边界的认知。

前提：假设总工具数量固定，且已按组预先分类（动态新增工具的情况单独讨论）。

tools 参数中仍然保留基础工具组（"basic"）；在上述前提下，ResetTools 自身的 schema 也不会变化，直接归入 basic 组，享受 cache 稳定性的好处。

当 ResetTools 被调用时，在正常返回结果之后，紧跟插入一段系统级通知：

<system-reminder>
当前激活工具组：[xxx]。新增激活工具（+N 个 schema）：
[{"type": "function", "function": {...}}, ...]
</system-reminder>

这样做的好处是：不依赖 search_tool 的实现质量；agent 对工具组边界有明确认知，对超出当前能力的任务可以直接拒绝，而不是反复搜索失败；静态部分对 prompt cache 友好。

但这个方案存在两个明显问题：

• 一是 ResetTools 是全量重置而非增量变更，重复切换工具组时已激活的 schema 会被反复输出，造成上下文膨胀；
• 二是一旦发生上下文压缩，agent 就会丢失对当前已装备工具的认知。

针对这两个问题，可以进一步改进：

• 将 ResetTools 拆分为增量原语，例如 ToolGroupActivate 和 ToolGroupDeactivate，每次 ToolGroupActivate 只注入新激活工具的 schema，避免重复输出。
• 同时，在上下文压缩发生时，始终在压缩摘要之后补充当前激活状态及所有激活工具的完整 schema，确保 agent 的工具认知不因压缩而断档。

需要坦诚的是，这套方案对模型的要求不低：

• 模型必须能从一系列激活/停用事件中正确推断出当前的工具装备状态，对"时序状态"的追踪能力是整个方案能否稳定运行的关键变量。
• 此外，不依赖 tools API 也为着失去了 API 提供方的结构化输出验证的保证
• 部分 API provider 可能会自己校验产生的工具调用是否在 tools 字段之内，这个校验会无法通过，因此这里提议里的 execute_tool 的价值凸显了出来；

[iluv7]

This is a valid concern, and I agree that we should run a benchmark first.

arguments could be represented as a nested object rather than a JSON string to avoid double serialization. However, it still adds another level of nesting compared with native tool calls, and the provider cannot validate the actual argument schema based on a dynamic tool_name, so AgentScope would need to validate it again on the backend.

The benchmark could focus on generation and validation success rates for complex strings, lists, dictionaries, and deeply nested arguments across different models and providers.

[iluv7]

@DavdGao
I’ll try to run some evaluations, using an approach similar to the open-source DeepEval project. My current idea is to use hooks to inspect the full tool-call trajectory at four checkpoints:

1. During the reasoning phase, verify that the model generates a valid and correct search_tool call.
2. During the acting phase, evaluate whether the search_tool response is accurate.
3. During the next reasoning phase, verify that the model correctly constructs the nested execute_tool call, including its arguments and escaping.
4. During the acting phase, check whether execute_tool runs successfully and returns the expected response.

This should help us measure both the reliability of generating nested JSON across different model providers and the end-to-end correctness of the proposed design.

[DavdGao]

@iluv7 Sorry for the late reply. That sounds good, the four checkpoints make sense to me. Looking forward to your results — feel free to share here once you have any numbers, even partial ones.

[iluv7]

@DavdGao
I’ve built a lightweight evaluation service for AgentScope, based on DeepEval: https://github.com/iluv7/agentscope-eval.
The tool-loading suite evaluates recorded traces at the four checkpoints we discussed: search-call generation, search results, tool-call generation, and actual execution. It uses deterministic JSON, schema, and expected-value checks, and reports first-attempt success separately from recovery through repairs or retries.

flowchart LR
    A["AgentScope 运行任务"] --> B["采集回答、工具调用和执行结果"]
    B --> C["评测服务"]
    D["测试用例：预期答案、工具、参数"] --> C
    C --> E["逐项结果、失败原因、汇总报告"]

Loading

The evaluation layer is working with synthetic fixtures; real-model experiments are still pending. For the native tool-call vs. execute_tool comparison, do you have any existing test cases or benchmark specifications we could reuse? We would adapt them to the AgentScope runner and our evaluation format.

[iluv7]

If there isn’t an existing benchmark, I’ll put together an initial reproducible suite and share the cases for feedback before running the comparison.

[labmimors]

The two concerns you called out are the right acceptance gates: a stable top-level tool surface is useful only if discovery recall and the extra request path are measured independently.

Disclosure: I maintain MCP Lens, a DeepSeek Harness implementation of the fixed search -> exact schema -> call pattern. We recently froze a 304-prompt / 102-tool, label-hidden convenience holdout before scoring. The current ranker reached Recall@5 0.246656 versus 0.062610 for the released baseline; paired wins/ties/losses were 99/197/8, with a 100,000-replicate 95% CI of [0.144846, 0.224342] for the Recall@5 delta. The rc.9 runtime then reproduced all 304 frozen rankings exactly. The public artifact is here.

That evidence is deliberately narrow: covered-call lexical retrieval, no model calls, not the official MCP-Atlas benchmark, and not an AgentScope result. It does show why cache shape and search quality should be separate benchmark axes.

For AgentScope I would freeze four arms on the same tool catalog and tasks:

1. current ResetTools;
2. provider-native masks where supported;
3. fixed search_tools + execute_tool;
4. incremental activate/deactivate reminders.

Then record: first-request tool-schema JSON bytes; prefix hash continuity; target Recall@1/5 before any model run; model request count; successful target execution; argument-validation failures; and post-compaction recovery. A search result must not grant authority: execute_tool should resolve the target through the existing Toolkit path and repeat the current authorization/policy check at call time.

For global capability awareness, I would keep only bounded inventory metadata stable (groups, counts, short descriptions), not every tool name/schema. Exact schemas should be refreshed near the tail when needed; after compaction, retain discovered identities and make re-search explicit rather than replaying all schemas.

If useful, I can contribute a keyless AgentScope-native fixture for this matrix. It should make no token, latency, cache-cost, or model-quality claim unless those are measured in a separate provider run.

[DavdGao]

@labmimors Yes, I agree — search quality should be treated as a separate engineering problem.

[DavdGao]

Update: I noticed that both Anthropic's and OpenAI's Response APIs support mid-conversation system messages that can add/remove tools on the fly. This means tool group management could potentially be implemented via a special control message, rather than updating the tools schema directly.

However, the current tool context is stored independently from the message context (list[Msg]). If a message within the message context (agent.state) were allowed to control the tool context, the two contexts would become intertwined.

That said, this might not necessarily be a bad thing. We've already run into a related issue on the frontend side: when the frontend subscribes to agent events and needs to be notified of changes like task_context or permission_context, we currently have to emit a corresponding CustomEvent through the event stream anyway. This makes me wonder whether it's worth unifying all context types into a single storage pipeline this time around — similar to how the old Pi coding-agent handled it, where one sequential store captured every state change within a session.

That approach does introduce a new problem, though: after compressing the conversation history, the tools' current schema would be lost. We'd need to either recompute the latest tools state right after compression, or compress the entire tools change history into a single consolidated change. This ties directly into how we implement context compression as a whole.

[iluv7]

I’ve looked into how Claude Code handles this, and I’m going to check OpenAI’s approach next.
I think storing tools and messages in one pipeline is a separate design question, rather than a requirement for dynamic tool loading. Claude Code keeps the current tool definitions in runtime state. ToolSearch adds tool_reference blocks to message history; on subsequent requests, the client uses those references to select schemas from the current tool set. When it compresses history, it saves a snapshot of the discovered tool names at the compression boundary.
If we do choose a unified store, I’d lean toward snapshotting the currently activated tools at compression time, then using a post-compression hook to put that state back into context. That would preserve the current tool state without replaying the full tool-change history.