Replaying an AgentScope run with no provider reachable — and one thing that will never reproduce

[xizhuomengcontin]

I record model traffic at the HTTP boundary — from outside the process — and serve the recording back so a run happens again with no provider called. Ran it against AgentScope 2.0.8 tonight. It captures with no code change, and while checking the result I noticed something about message ids that anyone writing tests here will want to know.

Capture needs no configuration

OpenAIChatModel builds its client as:

self.client: openai.AsyncClient = openai.AsyncClient(
    api_key=self.credential.api_key.get_secret_value(),
    organization=self.credential.organization,
    base_url=self.credential.base_url,     # None unless you set it
    **self.client_kwargs,
)

OpenAICredential.base_url defaults to None, and the OpenAI SDK then falls back to OPENAI_BASE_URL from the environment — which is what a recorder sets for the child process it launches, and only for that process. So this is captured as written:

cred = OpenAICredential(api_key="sk-test")          # base_url left as None
model = OpenAIChatModel(credential=cred, model="gpt-4o-mini", stream=False)
res = await model([Msg(name="user", content=[{"type": "text", "text": "Say hello in three words."}], role="user")])

$ orca record generic-openai -- python app.py
resolved base_url: http://127.0.0.1:54156/v1/
agentscope replied: [TextBlock(type='text', text='the stub answered', ...)]
info recorded run=run_641d009083e2 events=6 exit=0

Then with the origin process killed:

$ orca replay run_641d009083e2 --worktree
info replaying exchanges=1 egress=blocked
resolved base_url: http://127.0.0.1:54176/v1/
info replay.done reused=1/1 exact=1 divergences=0 unmatched=0 exit=0

exact=1 — the outbound request matched the recording byte for byte: formatter output, message blocks, parameters. The model class, the formatter and your own code run for real; only the answer comes from the trace.

Nice consequence of the credential design: because base_url lives on the credential rather than being read from the environment by AgentScope itself, setting it explicitly (OpenAICredential(api_key=..., base_url="http://127.0.0.1:8080/v1")) is a first-class way to point a run at a local endpoint, without touching global environment state that other models in the same process would also see.

The thing that will not reproduce, and it is not the model

Comparing the two runs above — identical request, identical recorded response — the returned block differs:

run 1: TextBlock(type='text', text='the stub answered', id='feb88edfaae64710add64509301af2ea', created_at=...)
run 2: TextBlock(type='text', text='the stub answered', id='84d576d1679f4c42be441a15dcc97674', created_at=...)

AgentScope mints a fresh id (and created_at) per block on the response side, so those fields vary run to run even when the provider returned the exact same bytes. That is clearly deliberate — blocks need identity — but it has a practical consequence: any test that asserts on a whole Msg/TextBlock, or snapshots one, will be non-deterministic regardless of how deterministic the model is. Assert on .text and the block type, or strip id/created_at before comparing.

It does not affect replay fidelity — the comparison is on the outbound request — but it would quietly break a golden-file test, and the failure would look like model non-determinism when it isn't. Worth a line in the testing docs, if there isn't one already.

Limits

• Blocked egress means model-provider egress, not network isolation. Replay re-executes everything else for real — a tool an agent called, it calls again.
• A matching replay is not a determinism result. The model is not re-asked; its recorded answers are served back.
• Embedding calls are not captured by my recorder yet, so a pipeline using OpenAIEmbeddingModel at query time cannot be fully replayed today. Saying so rather than letting someone find out mid-run.
• Tested with agentscope 2.0.8 on Python 3.12 against a deterministic local origin standing in for a provider, non-streaming, no tools. Streaming and tool paths are untested here.

Tool is OrcaReplay (Apache-2.0, npm, Node 20+); I maintain it, so weigh accordingly. The block-id observation is about AgentScope and holds regardless of what produced the response.