Repository navigation
SystemOne Decision Models Integration into lemonade #3751
meghsat
started this conversation in
Request for Comment (RFC)
Replies: 5 comments 4 replies
|
Are we making |
2 replies
|
For score questions does |
1 reply
|
Can phase 1 drop the pin bump as main has already moved past the min required version and the three causal models added to server_models? That can be a follow up PR with tests. |
1 reply
|
Does this cover only llamacpp? what about vllm? |
0 replies
|
A few contract questions that came up while reviewing #3952 and ort-server#4. Since #3952 has no RFC of its own and this RFC takes
|
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Proposal size
Major feature (spans multiple well-scoped PRs)
Updates
main. Adding zero-shot classifier support #3952 (zero_shotclassifier type,labelson/v1/classify) merged on 2026-10-29. Add SystemOne decision models: /v1/systemone, GGUF zero-shot on /v1/classify, and the systemone router classifier #3752 is closed; Add SystemOne decision models: /v1/systemone, GGUF zero-shot on /v1/classify, and the systemone router classifier #3956 replaces it./v1/classify, question checks at registration,state: nulland images. The sections below describe them.User story
As a developer I want to get decisive, straightforward, and quick responses to my questions and LLMs[system-two] models are slow and often produce way too many tokens unless we prompt tune them. To solve this, a new set of models called system-one is released. These models are built to make fast, structured decisions that software can use directly.
llama.cpp added decision models in PR #29818. The PR was merged on 2026-10-02, and release b11361 contains it. A decision model answers typed questions in one forward pass. It does not generate tokens. llama-server serves these models through the /v1/systemone endpoint. The GGUF files are released for the following models:
I also want to use these models in a
collection.routerpolicy. I ask a small local model typed questions about each request, and I route the request on rules that I write in plain words. For example: "Does the message contain personal data?" or "Which team must handle this?".High-level design
Lemonade gives decision models to clients and to the router in three ways.
POST /v1/systemoneThe endpoint is also available under
/api/v0/,/api/v1/,/v1/, and/v0/. Lemonade does these steps:stateor noquestions, or names a model that is not a decision model. A decision model has the labelsclassificationandsystemone, on a backend that answers/v1/systemone(today,llamacpp).statecan benull, for a question about images only. A model registered with itsmmprojfile readsimages(data URLs). llama-server decides which decision types accept anullstate and images.Request:
Response (Laya, llama.cpp b11846, values rounded):
{ "model": "Laya-GGUF", "answers": { "route": {"type": "choice", "choice": "billing", "probabilities": {"billing": 0.9914, "shipping": 0.0059, "technical": 0.0027}, "confidence": 0.9871}, "angry": {"type": "noul", "noul": 0.7881}, "urgency": {"type": "score", "score": 1.897, "legend": {"0": "can wait", "1": "this week", "2": "today", "3": "right now"}, "probabilities": {"0": 0.0394, "1": 0.4668, "2": 0.0508, "3": 0.4430}, "confidence": 0.0238} }, "usage": {"input_tokens": 136, "output_tokens": 0} }The same endpoint with an image and no text (
vllm-d3-edge, registered withmmproj-vllm-d3-edge-Q8_0.gguf; the image is a 64×64 red PNG):{"model": "user.vllm-d3-edge", "state": null, "images": ["data:image/png;base64,iVBORw0KGgo..."], "questions": {"colour": {"type": "choice", "instructions": "What colour is the image?", "criteria": {"red": null, "green": null, "blue": null}}}}Answer:
colourisred, with probability 0.9996.POST /v1/classifyA decision model also answers a
/v1/classifyrequest that has labels. Lemonade asks the model onechoicequestion, with the labels as the options, in request order. The response has the same form as the response of the ONNX zero-shot model. Thus thezero_shotrouter classifier from #3952 operates with a GGUF decision model.The recipe of the model selects the backend:
flowchart LR C1[Client] -->|POST /v1/systemone| H1[SystemOneRoute] P[collection.router policy] -->|systemone| RS[Router::systemone] H1 --> RS P -->|zero_shot| RC[Router::classify] C2[Client] -->|POST /v1/classify + labels| RC RC -->|onnxruntime| O[ort-server] RC -->|llamacpp| A[LlamaCppServer::classify<br/>one choice question] RS --> L[LlamaCppServer::systemone] A --> LS[llama-server /v1/systemone] L --> LS3. A router classifier type:
systemoneA
systemoneclassifier asks one typed question in your words. Its labels are the options of the question:choicecriteria(an array keeps their order)noultrueandfalsescorecriteriaWrite a guard rule (a yes/no property, such as personal data) as a
systemonenoulquestion. Anoulscore does not depend on other labels. Azero_shotscore is a softmax over the declared labels, so it changes when you add or remove a label. Usezero_shotto select one of several labels, such as a topic.{ "classifiers": [ {"id": "pii", "type": "systemone", "model": "Laya-GGUF", "default_label": "true", "question": {"type": "noul", "instructions": "Does the message contain personal data?"}}, {"id": "topic", "type": "zero_shot", "model": "Laya-GGUF", "labels": ["chit-chat", "coding"]} ], "rules": [ {"id": "pii-local", "match": {"classifier": "pii", "min_score": 0.5}, "route_to": "Local-GGUF"}, {"id": "code-to-big", "match": {"classifier": "topic", "label": "coding", "min_score": 0.5}, "route_to": "Big-GGUF"} ] }Models
Lemonade adds these models to the registry:
Julia-1-GGUFlaya(encoder)Q8_0Laya-GGUFlaya(encoder)Q8_0Other decision models operate as user models (labels
classificationandsystemone). These were tested:lev-GGUF,OpenJev-GGUF(with images), andggml-org/vllm-d3-edge,-liteand-nano(with images).The
classificationlabel is the deployment mode. Thus a decision model never loads as a chat model, and it does not unload the chat model of the user. Thesystemonelabel tells Lemonade that the model is a decision model. A llama.cpp model with theclassificationlabel and without thesystemonelabel is still refused at registration.Implementation
/v1/systemoneinllama-serverzero_shotclassifier type,labelson/v1/classify,on_truncation/v1/systemone, the/v1/classifyadapter, thesystemoneclassifier type, load safety, the two registry models, tests and docsBreaking changes
None. The endpoint, the classifier type, and the two models are additions.
Maintenance plan
CI tests the feature. Upstream llama.cpp maintains the prompt layout and the scores of each decision model.
/v1/systemone,/v1/classifyadapter,systemoneandzero_shotclassifierssystemone-llamacpp(cpu, Julia-1 and Laya) and C++ unit testsRisks
No response
All reactions