Skip to content

feat(eval): add live/audio eval support to the eval UI - #472

Merged
wyf7107 merged 1 commit into
google:mainfrom
allen-stephen:feat/live-eval-v2
Jul 28, 2026
Merged

wyf7107 merged 1 commit into
google:mainfrom
allen-stephen:feat/live-eval-v2

Conversation

@allen-stephen

Copy link
Copy Markdown
Contributor

Summary

Adds live (bidi streaming) evaluation support to the eval UI, including an optional simulated audio user. The run dialog now leads with a Standard | Live mode selector; live runs can drive the agent with synthesized speech or plain text. Live results render as a clean per-turn conversation with playable audio and carry the same per-turn PASS/FAIL annotations that standard runs show.

  • Run dialog: Live runs expose an Input modality dropdown (Audio by default, Text) plus voice and language-code fields for the simulated audio user.
  • Running live evals: runEval posts to the V2 eval-sets/{id}/run endpoint, signaling live mode via the presence of live_model_config and passing user_simulator_config (llm_audio, Gemini TTS) to synthesize the user's turns.
  • Live result rendering: a live run's raw session (audio chunks, partial transcriptions, turn-complete markers) is rebuilt into one bubble per turn, folding the streamed PCM for each response into a single playable WAV clip. PCM→WAV helpers are extracted into core/utils/audio.ts and reused by the existing artifact playback path.
  • Consistency & polish: live result bubbles show per-turn PASS/FAIL plus failing-metric detail just like non-live runs; eval sessions load under the user id the eval actually ran as, the run spinner always clears on empty/error paths, and single-case runs open their result directly.

@wyf7107
wyf7107 merged commit 779a722 into google:main Jul 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants