instructions unclear, haven't slept in 40 hours

[MizuleGPT]

I tried many ways of following the guide in https://github.com/NovaSky-AI/SkyThought/tree/main/skythought/test-time-scaling

I need it to run in kaggle, after even letting grok 3 with "deeper research" have 3 goes at it and trying dozens of variations both hand crafted and generated, I have no idea how to run this.

can we get a more clear guide? especially for the case of it using a local LLM rather than API because not everyone has money.

sorry if I come off as rude or dumb, I'm just in a rush and sleep deprived

[DachengLi1]

Thank you for your interest @MizuleGPT . Here is the instruction to run:

| experiments on final test set are in: final_[]. First run commands under final_oracle to produce all generations without different selection methods, then run commands under final_[]_cached to produce generations with different selection methods.

For local models, for instance if you want to run the oracle setting: https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/test-time-scaling/scripts/final_oracle/qwen1.5b_n_32_debug_public3_select_oracle.sh.

First spin up the server using vllm: vllm serve Qwen/Qwen2.5-Coder-1.5B-Instruct, and run bash scripts/final_oracle/qwen1.5b_n_32_debug_public3_select_oracle.sh.

Please let me know which instruction could be improved. Happy to help more!

[MizuleGPT]

I just tried to run 14 variations of this in a fresh kaggle notebook, all one block, no other code:

!pip install dspy==2.6.2 torch vllm
!git clone https://github.com/NovaSky-AI/SkyThought.git
%cd SkyThought

!uv venv --python 3.10
!uv pip install -q -e . --python .venv/bin/python

!.venv/bin/skythought evaluate \
     --task aime24 \
     --model unsloth/QwQ-32B-unsloth-bnb-4bit \
     --backend vllm \
     --backend-args tensor_parallel_size=2,revision=0dccf55,dtype=bfloat16 \
     --sampling-params max_tokens=4096,temperature=0.1 \
     --system-prompt-name prime_rl \
     --result-dir ./ \
     --batch-size 1

it said "ValueError: No supported config format found in unsloth/QwQ-32B-unsloth-bnb-4bit".

I tried with skythought flash 32B and that tried to work but it tried to fit on only one of the T4s so it OOM'd instantly

[DachengLi1]

@MizuleGPT Thanks you for reporting this.

This is not related to the test-time-scaling branch, @SumanthRH can you take a look at the evaluate functionality with the 4 bit feature? Thanks!

[MizuleGPT]

I feel the need to clarify my end goal:

A kaggle implementation for chatbot-like inference with the method that makes R1 distill 32B approach o1-high performance in coding, from this paper: https://arxiv.org/pdf/2502.14382

I wanted to use the unsloth quant of QwQ 32B because it'd fit in kaggle across the 2 T4 GPUs and the result I assume would be on par with o1-high.

I'm soul-crushingly far from figuring that out, but if one of you can it'd vastly increase the accessibility of this research and be the highlight of my month

[MizuleGPT]

@MizuleGPT Thanks you for reporting this.

This is not related to the test-time-scaling branch, @SumanthRH can you take a look at the evaluate functionality with the 4 bit feature? Thanks!

I tried to clarify above, what I'm ultimately trying to do is related to the test-time-scaling branch but what I was objectively doing may not have been

[SumanthRH]

@MizuleGPT Thanks for reporting the issue.

For the command provided:

!.venv/bin/skythought evaluate
--task aime24
--model unsloth/QwQ-32B-unsloth-bnb-4bit
--backend vllm
--backend-args tensor_parallel_size=2,revision=0dccf55,dtype=bfloat16
--sampling-params max_tokens=4096,temperature=0.1
--system-prompt-name prime_rl
--result-dir ./
--batch-size 1

You have used an invalid revision for the model unsloth/QwQ-32B-unsloth-bnb-4bit .

I believe your command is a modified version of the example "Customized" command in our guide:

 # 'Customized' example for NovaSky-AI/Sky-T1-32B-Flash
skythought evaluate \
    --task aime24 \
    --model  NovaSky-AI/Sky-T1-32B-Flash \
    --backend vllm \
    --backend-args tensor_parallel_size=8,revision=0dccf55,dtype=float32 \
    --sampling-params max_tokens=4096,temperature=0.1 \
    # use a pre-configured system prompt
    --system-prompt-name prime_rl \
    --result-dir ./ \
    --batch-size 128

Note that the above example was meant only for the model NovaSky-AI/Sky-T1-32B-Flash. The revision argument is a specific git commit hash for the model on HuggingFace. If you use the same revision for a different model, it will fail.

For the model you have specified: unsloth/QwQ-32B-unsloth-bnb-4bit, there are also some specific arguments needed to pass to vllm.

In general, if you want to use a specific backend, vllm, openai, etc, and you get an issue, it's always better to confirm that you can load it outside of skythought with the same configuration.

In this case the dtype is actually incorrect for unsloth/QwQ-32B-unsloth-bnb-4bit (it's specified as bfloat16, but it should be left as auto) This is because the model you are using is quantized in the bitsandbytes format.

If you were loading the model from vLLM, you would do:

from vllm import LLM

llm = LLM("unsloth/QwQ-32B-unsloth-bnb-4bit", quantization="bitsandbytes", load_format="bitsandbytes",  tensor_parallel_size=<insert_size>)

Thus, to use with skythought, you should pass:

--backend-args tensor_parallel_size=<insert_size>,quantization=bitsandbytes,load_format=bitsandbytes

Lastly, when I used your original configuration, I realized we had a bug with the prime_rl system prompt. This has been fixed in #104

For the unsloth model, I have now verified that the following works on a 1xA100:

skythought evaluate    \
     --task aime24    \
     --model unsloth/QwQ-32B-unsloth-bnb-4bit   \
     --backend vllm   \
    --backend-args tensor_parallel_size=1,quantization=bitsandbytes,load_format=bitsandbytes  \
    --sampling-params max_tokens=4096,temperature=0.1  \
   --system-prompt-name prime_rl      \
  --result-dir ./   \
  --batch-size 1

With skythought installed from source.

Hopefully that should resolve your issue.

[MizuleGPT]

"ValueError: Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your Tesla T4 GPU has compute capability 7.5. You can use float16 instead by explicitly setting thedtype flag in CLI, for example: --dtype=half."

how might I get around this? I can't use any other GPUs

[MizuleGPT]

also: is there a command to have it just take a prompt of mine, use all its tricks and give me the resulting code?

I was gonna try to modify it for that but the thought alone gives me a headache