Qitos cybergym - #23
Open
ravenSanstete wants to merge 37 commits into
Open
Conversation
Engine: - Fix _action_runtime.py early-exit bug: blocked actions no longer kill sibling actions; collect-all-then-merge pattern - Add submit_poc to _CONCURRENCY_SAFE_TOOLS for parallel submission Harness: - Add json_decision_multi_v1 protocol supporting native tool_calls - GLM preset uses json_decision_multi_v1 with parallel_tool_calls=True - Add recommended_request_kwargs to harness types - Auto-merge preset kwargs in harness __init__ Render: - Fix ClaudeStyleHook multi-action rendering with per-index dedup - Add parallel action/observation visual indicators - Extract _render_single_observation for multi-observation support - Content renderer: action_summary/observation_summary support multi-action/multi-observation events
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| def test_delete(self): | ||
| mem = InMemorySharedMemory() | ||
| mem.write("key1", "value1") | ||
| assert mem.delete("key1") is True |
| mem.write("key1", "value1") | ||
| assert mem.delete("key1") is True | ||
| assert mem.read("key1") is None | ||
| assert mem.delete("key1") is False |
| def test_delete(self, tmp_path): | ||
| mem = FileSharedMemory(tmp_path / "shared.json") | ||
| mem.write("key1", "value1") | ||
| assert mem.delete("key1") is True |
| mem = InMemorySharedMemory() | ||
| ns = SharedMemoryNamespace(mem, "agent_a") | ||
| ns.write("key1", "value1") | ||
| assert ns.delete("key1") is True |
| def test_delete_nonexistent(self): | ||
| mem = InMemorySharedMemory() | ||
| ns = SharedMemoryNamespace(mem, "agent_a") | ||
| assert ns.delete("nope") is False |
| from qitos.core.state import StateSchema | ||
|
|
||
|
|
||
| @dataclass |
|
|
||
| def create_snowl_agent(**kwargs: Any) -> Any: | ||
| """Create the {{cookiecutter.agent_name}} agent for Snowl evaluation.""" | ||
| from .src.agent import {{cookiecutter.agent_name | pascalcase}}Agent |
|
|
||
| from __future__ import annotations | ||
|
|
||
| from {{cookiecutter.agent_name}}.src.agent import ( |
| from .backends import CacheBackend | ||
|
|
||
|
|
||
| class CachedModel(Model): |
Previously, reasoning_content (from DeepSeek/QwQ/etc.) was only used as a fallback when content was empty. Now it is: 1. Extracted into ModelResponse.reasoning_content (new field) 2. Included in model_output event payload 3. Displayed by ContentFirstRenderer as the agent's "thought" text 4. Propagated through _hooks_impl to the TUI Also: ToolResult.to_dict() promotes string output as "content" key so ContentFirstRenderer can find it.
Previously _extract_response_text() returned "" when tool_calls were present, discarding any content the model produced alongside its actions. Now content is always extracted regardless of tool_calls. ContentFirstRenderer.thought_text() shows both reasoning_content and raw_output (content text) when present, separated by "---". This means the operator sees the model's complete reasoning: API-level chain-of-thought + any non-tool-call text the model generated.
The agent's task instruction was never shown in the TUI, making it impossible to understand what the agent was asked to do without checking external logs. Now the task text is rendered as a cyan "TASK" line right after the RUN banner.
Previously build_tool_spec() set every parameter description to "". The :param annotations in docstrings were only visible as part of the top-level tool description, forcing the model to parse unstructured text to understand parameters. Now: - _parse_param_descriptions() extracts :param name: desc from docstrings - Each parameter gets its own description in the schema - _strip_param_docs() removes :param lines from the top-level description to avoid duplication This means the model sees structured per-parameter descriptions in the tools API parameter, not a wall of text it has to parse itself.
- _state_stats extracts chain nodes/gates, vulnerability analysis, current hypothesis, and path trace from agent state - _render_chain_summary produces compact multi-line display with ✓/?/✗ badges for confirmed/inferred/refuted gates - _hooks_impl renders chain summary with color-coded lines: cyan for chain overview and nodes, red for refuted gates, yellow for open gates, magenta for hypothesis, blue for analysis
- _model_runtime.py: _state_stats() extracts constraint_board and task_memory from state.metadata (primary) or builds from state fields (fallback). Removes old _render_chain_summary(). - _hooks_impl.py: renders Constraint Board and Task Memory sections with semantic color-coding. Uses exact same text the LLM sees in observation packet, ensuring TUI-LLM consistency.
_state_stats() was trying to extract chain/gate data from the observation object, but the observation doesn't carry the live CyberGymState — it only has serialised data. Now the live state object is passed from run_decide() → _run_llm_decide() → _state_stats(), so constraint_board and task_memory are properly extracted for TUI.
…t loop Sync agent source (commit f63ccad) including: - Vul-only trigger is PARTIAL success — agent keeps refining for precision - stop_criteria: only stop on is_verified(), not vul_crashed() alone - New vul_only_triggered gate + vul_crashed_partial verdict - ASAN trace fallback from raw_output when vul_stderr is empty - Patch-diff-guided refinement feedback - Candidate escape hatches, targeted PoC registration, submit validation - Removed .gitignore exclusion for cybergym/agent directory
…raw_output GLM-5.1 puts identical content in both reasoning_content and raw_output, causing the TUI to display the same thought twice separated by "---". Now skips raw_output when it duplicates reasoning_content.
Add log_file parameter to ClaudeStyleHook that creates a _TeeConsole proxy writing to both terminal and a plain-text log file. Each task in the batch runner now saves tui.log in its trace directory, preserving the STEP/finish/tool_calls/ctx_used format for offline review.
Brings the vendored agent up to date: constraint collection overhaul (3081def) plus the two fixes — submit_poc results keyed by (agent_id, poc) to stop cross-task and parallel-submit verdict leakage, and the missing CyberGymState.pending_reminders field (was causing unrecoverable_error). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… handling runner.py: CYBERGYM_USE_DOCKER_ENV gate (default off) runs each task's agent tools inside an ephemeral container via a same-path bind mount, so the agent cannot wander outside its task workspace while the host-side process (LLM/submit_poc) keeps working. docker_env.py: _inner_path now uses absolute paths verbatim instead of double-nesting them under the workdir. Verified: 4 concurrent containers, per-task grading correct, no leakage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e-feedback fix) Vendored sync of the fix-side leak fix: under CYBERGYM_VUL_ONLY_FEEDBACK (default on) the agent sees vul-side feedback only and stops on the first vul-side crash; the server still grades fix-side into its DB for scoring but the verdict is no longer read back to the agent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Without this file, Python treats this directory as a namespace package when the parent (traj_analyzer/) is on sys.path, shadowing the pip-installed qitos package and causing ImportError for AgentModule.
…e shadowing" This reverts commit 64207ef.
Write exceptions to CYBERGYM_TASK_TRACE_DIR/step_error.log at every exception handler in engine.py (DECIDE, ACT, init_state, setup_toolsets, setup_env) and add sys.excepthook in run_local.py so no exception can be silently lost regardless of stderr redirection.
…oard _model_runtime._state_stats() now extracts sink_candidates and objective from state.metadata. _hooks_impl.py renders them with semantic coloring: - Sink Candidates: bright_magenta for high conf, yellow for nudge - Objective: green inline text
…dering - Extract _tui_phase from metadata in _state_stats for reliable Phase Badge - Add task_context and allowed_tools to _state_stats extraction - Render Task Context section (vulnerability, bug type, strategy, input format) - Render Allowed Tools section with checkpoint-blocked highlighting - Color-code CHECKPOINT BLOCKED text in bold red
- Render Suggested Sinks section in TUI (bright_blue styling) - Add [AUTO] tag styling in Sink Candidates rendering - Extract _tui_suggested_sinks from metadata in _state_stats
Raise body truncation limits from 220/2000 to 50000 for all observation types (tool results, error content, syntax highlighting, terminal output). When truncation does occur (extremely rare at 50K), show total char count. This ensures researchers can see full tool call results in TUI logs.
…ontainer - docker_env.py: container_env dict param, inject -e flags in _ensure_container(), atexit/signal cleanup for managed containers - runner.py: _vul_binary_mounts for staging vulnerable binaries, pass CYBERGYM_STAGE_VUL_BINARY/ENABLE_DYNAMIC_TOOLS into container env
Comment on lines
+54
to
+58
| return WhitzardAgent( | ||
| model=model, | ||
| max_steps=max_steps, | ||
| **kwargs, | ||
| ) |
- Add Current Assessment + Experiments rendering to TUI (_hooks_impl, _model_runtime) - Fix content_renderer to preserve full GDB output in tool result bodies - Fix runtime_context_contract: optimistic defaults when rediscovery_pending - Fix arbitration: _gdb_available returns True when rediscovery_pending - Fix dynamic_execution: GdbDebugTool skips capability check on rediscovery - Fix submit_processor: pending_reproduction after no-trigger - Add staged_binary container-aware discovery - Add tests for content_renderer GDB output handling
- openai.py: _make_openai_client() reads QITOS_INFERENCE_KEY env var and injects x-inspire-inference-key header for load-balancer affinity routing (same task -> same backend node -> better KV cache) - _shared.py: _run_example_job() sets QITOS_INFERENCE_KEY to task's job_key before each run, restores on exit (thread-safe for concurrent)
- cli.py: build_inference_task_id for run-unique routing key - run_local.py: wire inference_key into llm_config
- runner.py: add inference_key to build_agent call so task_id shows in AGENT COMPOSITION header - content_renderer: _action_from_dict handles Action objects (not just dicts), renders as tool_name(key=val) format with zero truncation
- Knowledge packs (PDF, image, packet, sfnt) with scripts and references - Backend registry with 11 optional dependencies - Observation validation and rendering improvements - Recipe IR and candidate builder updates - Fix poc sink management and feedback gate refutation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Describe the change in 2-5 sentences. Focus on user-facing behavior and why this change matters.
What Changed
Why
What problem does this solve? What workflow, bug, or maintenance gap does it address?
Validation
pytest -qCommands run:
# paste commands hereQitOS Design Impact
If relevant, explain how this PR affects:
Docs and Changelog
CHANGELOG.mdupdatedChecklist