jobque: self-describing traces — named categories, colors, unit markers

[borisbat]

Promotes the per-lane jobque event tracer to a self-describing common API — groundwork for the jobque timeline viewer tool (PR2 to follow).

What

• C++ ring (job_que.h/cpp): new TraceTag::Marker kind + JobQue::traceMarker (instant "unit" event on the caller lane — token/frame/layer boundaries), and cold-path registries traceCategory(id, name, rgb) / traceMarkerName(name). traceSave now emits markers as perfetto instant events ("ph":"i") and appends a jobqueProfile sibling key (categories with #RRGGBB colors, marker kinds) after traceEvents — Perfetto ignores unknown keys, so files stay perfetto-compatible and the existing analyzers (trace_window.py, trace_gaps.py, …) work unchanged (verified).
• New builtins: jobque_trace_category, jobque_trace_marker_name, jobque_trace_marker (hot path = one relaxed load no-op when tracing is off), + AOT header decls.
• daslib/jobque_profile.das (new): profile_start/stop/save, profile_category (auto-id or explicit-id, palette fallback by name hash), profile_tag, profile_marker/profile_marker_id, PROFILE_COLOR_* constants.
• dasLLAMA: set_trace_tags(true) now registers the 10 lane-timeline categories with the viewer's exact names/colors, so saved traces carry the legend the browser artifact used to hardcode; unit markers stamped in the traced windows — token (decode_prof), layer (prefill stack), step (batch_decode_perf). Kills the pick_cls.py-style representative-window hunt: viewers navigate unit-to-unit.
• Test: tests/jobque/test_jobque_trace.das — arm → register → team-dispatch storm with tag + markers → save → re-read JSON → assert header/categories/markers/publish-tag packing round-trip. First test coverage of the trace API.
• Docs: das2rst group entries + handmade pages for the new builtins and module; sec_concurrency toctree.

Verified

• tests/jobque: 210/210 pass; new round-trip test passes.
• Real trace: decode_prof --trace on Qwen2.5-0.5B → 106k events, all 10 categories in the header, 8 token markers (args 0..7) on the caller lane, publishes tagged attn-chain/ffn-chain/classifier.
• trace_window.py cuts the new-format file unchanged.
• Sphinx -W html build clean; no Uncategorized, no doc stubs; lint/format clean on all changed .das.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR upgrades the job queue tracer into a self-describing trace format by adding marker events and on-file registries for tag categories (names + colors) and marker kinds, enabling downstream timeline viewers to interpret traces without hardcoded legends.

Changes:

• Added C++ support for trace marker events and persisted registries (jobqueProfile) emitted alongside Perfetto-compatible traceEvents.
• Exposed new jobque tracing builtins and an AOT header surface, plus a new daslib/jobque_profile wrapper module.
• Integrated marker/unit boundaries and category registration into dasLLAMA profiling paths, and added a JSON round-trip test + documentation updates.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/jobque/test_jobque_trace.das	New round-trip test validating categories/markers and marker instant events in saved JSON.
src/misc/job_que.cpp	Implements category/marker registries, marker emission, and writes jobqueProfile into saved trace JSON.
src/builtin/module_builtin_jobque.cpp	Adds builtins for category/marker registration and marker emission.
modules/dasLLAMA/dasllama/dasllama_math.das	Registers trace categories when enabling dasLLAMA trace tags; adds marker helper.
modules/dasLLAMA/dasllama/dasllama_common.das	Adds per-layer unit markers during prefill tracing.
modules/dasLLAMA/benchmarks/decode_prof.das	Adds per-token unit markers in the decode profiler loop.
modules/dasLLAMA/benchmarks/batch_decode_perf.das	Adds per-step unit markers in the batch decode loop.
include/daScript/simulate/aot_builtin_jobque.h	Declares new jobque trace builtins for AOT.
include/daScript/misc/job_que.h	Extends tracer internals with Marker tag and registry storage.
doc/source/stdlib/sec_concurrency.rst	Hooks generated jobque_profile docs into the concurrency section.
doc/source/stdlib/handmade/module-jobque_profile.rst	Adds handmade module overview and usage example.
doc/source/stdlib/handmade/function-jobque-jobque_trace_marker-0x23d3c8a055ea0170.rst	Documents marker builtin behavior and Perfetto instant event mapping.
doc/source/stdlib/handmade/function-jobque-jobque_trace_marker_name-0x11d82ed54aded050.rst	Documents marker kind registration builtin.
doc/source/stdlib/handmade/function-jobque-jobque_trace_category-0x9e6a5d0ed05ba6da.rst	Documents category registration builtin and persistence behavior.
doc/reflections/das2rst.das	Adds doc generation for jobque_profile and includes new builtins in jobque groups.
daslib/jobque_profile.das	New high-level profiling wrapper (sessions, categories, markers, palette constants).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.