This experiment compares DecisionTune 1.0, Laya Multilingual and
Qwen3-0.6B behind the same choose(state, question, options) interface.
It measures classification decisions, separately from the LangGraph workflow's
review gates. The frozen data and
dataset notes define the tasks and original sources.
decision-workbench/public-v1 fixes two English tasks:
| Adaptation | Cases | Classes | Sampling unit |
|---|---|---|---|
| CUAD annotated-span topic classification | 50 | 5 | 10 excerpts per class, from 34 original test contracts |
| BANKING77 card support intent classification | 80 | 8 | 10 utterances per class, from the original test CSV |
The classes, descriptions and questions were chosen before model inference. Selection ranks source IDs and text hashes deterministically, after removing blank text, exact-text duplicates and cross-label exact-text ambiguity. The CUAD ambiguity check includes all source labels, including unselected labels. Neither preparation nor selection examines model output. No length filter, truncation or post-result replacement is allowed. An input that an adapter cannot process remains a failed case.
The manifest binds source revisions, download hashes, class counts, exclusions, prompts, ordered options, original IDs and selected text hashes. The data loader also pins the manifest hash. Changing a task means a new version and new results. The implementation commit and source hashes in each run identify the executed code; later documentation edits do not rewrite that evidence.
The CUAD input is an already extracted, annotated span. This is easier and different from finding clauses in whole contracts. BANKING77 uses eight related intents rather than the original 77. Neither result should be advertised as the original benchmark score. The dataset labels are the publishers' annotations, not labels generated by this project or independently reviewed by its owner.
| Provider | Actual local implementation | Useful property | Cost or limitation |
|---|---|---|---|
| DecisionTune 1.0 | ModernBERT-large decision head, MLX | Native typed choices, 8,192-token total limit | English focus; uncalibrated scores; BANKING77 appears in its declared training sources |
| Laya Multilingual | mmBERT decision head, FP16 MLX port | Smaller weights; multilingual checkpoint with native choice/score/boolean heads | 1,024-token shared context and smaller question/option budgets; this experiment tests English choices only |
| Qwen3-0.6B | Official causal LM, FP16 Torch MPS | A general language model can implement the same closed-choice contract | Extra prompt tokens and letter-position bias; the adapter tests next-token candidate scoring, not generated answers or thinking mode |
All three tested weight repositories declare Apache-2.0. Our adapters use local
verified snapshots and pinned package versions. No model was trained here.
See NOTICE for authors and primary sources; exact asset hashes and
revision IDs live in each run's provenance.model.
The supplied source text, question, option labels and descriptions are identical
across providers. Each adapter applies its own required input format. Qwen maps
options to single-token letters A through T, applies the shipped chat template
with enable_thinking=False, takes one forward pass, and softmaxes only those
candidate logits. It generates no answer tokens. This is one specific adapter,
not the best achievable Qwen configuration. Reversing options also remaps the
letters; positional changes are part of that probe.
Laya retains the upstream decision temperatures, without rounding. It checks the complete sequence against the runtime's prepared sequence and rejects question, option or state truncation. Qwen refuses inputs above the adapter's 4,096-token cap, below the checkpoint's advertised maximum. DecisionTune's runtime rejects over-budget input. No silent shortening is used to make a model complete more cases.
Scores have different meanings across adapters. They are not fitted correctness probabilities, so this study does not rank calibration or apply one shared confidence threshold. That would require a separate development split and a frozen calibration method. Completion coverage here means valid model output, not safe automation coverage or a human-review policy.
- Verify the frozen manifest before loading weights.
- Load one pinned checkpoint. Record model loading separately.
- Make one fixed synthetic warmup call, recorded outside benchmark metrics.
- For every frozen row, pass only its
input.state,input.questionandinput.optionstochoose. Gold, task IDs, filenames, annotation labels and source metadata never enter the provider. - Make one primary call, then repeat with reversed option insertion order. Labels and descriptions stay paired. No retry, prompt revision or best-of selection is performed.
- Validate choices and scores at the adapter boundary. Retain exceptions and malformed outputs as failures, with their original case IDs.
- Flush each paired prediction to disk. A reserved or interrupted run cannot silently overwrite an earlier run; only a complete run gets a final summary.
Each complete model run has 130 primary + 130 reverse-order decisions. The three-model comparison therefore has 780 benchmark attempts, plus three separately recorded warmups. Earlier synthetic results and exploratory smoke calls are separate experiments, not added to those denominators.
No sampling or decoding temperature is used by these forward-pass classifiers. Seed 42 controls the metric bootstrap. Fixed inputs and pinned weights make the experiment repeatable; floating-point kernels and latency can still differ between devices, runtime releases or repeated runs.
Report each task separately. Accuracy divides correct choices by all planned cases, including provider failures. Macro F1 averages every fixed class with zero divisions set to zero. Each class has an explicit support and confusion row; failed predictions contribute false negatives. Conditional accuracy among completed cases is secondary and cannot replace the full denominator.
For uncertainty, 2,000 seeded percentile bootstrap replicates resample whole source groups: contracts for CUAD and individual utterance IDs for BANKING77. The statistic remains case-weighted; it is not a contract-weighted score. Paired order metrics resample the same groups on both sides and report choice changes only among both-completed pairs, with that denominator visible. These small convenience slices and their intervals do not establish model superiority or population coverage. An interval with identical endpoints is marked degenerate.
Call time is sequential host wall time around choose, including serialization,
tokenization and synchronization needed to return scores. Loading and the
single warmup are excluded. Shape-specific compilation, thermal state and runtime
differences remain. Medians and p95s describe this run; they are not a serving
throughput test, memory benchmark or an architecture-only speed comparison.
BANKING77 is explicitly listed in DecisionTune's fine-tuning sources. Its test split is therefore useful for observing the adapter on a familiar-source task, but cannot support a training-unseen claim. CUAD is not named in that model's declared fine-tuning sources; this does not establish that it was never seen. Exposure through pretraining or other training data is unknown for all models.
Excerpts can contain several topics despite having one retained annotation. Removing identical-text ambiguity cannot remove every semantic ambiguity. Public test data may also have annotation errors. No professional legal review, live account evaluation, payment decision or customer outcome is claimed.
Next experiments should use a separately frozen development split for prompt or calibration changes, a larger untouched test slice, other languages for Laya, and additional option permutations. The current artifacts remain as the baseline even when later experiments improve the results.
Use the repository's requirements.lock for offline tests and metric
verification. The complete tested Apple-silicon model environment is
requirements-model-macos.lock; no additional package is needed for the two
new adapters. The result guide contains exact
download and evaluation commands and the measured results.
.venv/bin/python -m workbench.datasets verify --out benchmarks/public-v1
.venv/bin/python -m workbench.benchmark_report --runs reference/public-v1The second command revalidates all frozen IDs, labels, request hashes, choices and option orders, and recomputes the summaries from predictions without loading models or using the network. CI performs these offline checks. Actual neural inference is a separately recorded local run, not a CI result.