Repository navigation
Expand file tree
/
Copy path101_hello_tinker.py
More file actions
217 lines (158 loc) · 6.54 KB
/
Copy path101_hello_tinker.py
File metadata and controls
217 lines (158 loc) · 6.54 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
import marimo
__generated_with = "0.23.8"
app = marimo.App()
@app.cell
def _():
import marimo as mo
return (mo,)
@app.cell(hide_code=True)
def _(mo):
mo.md(r"""
# Tutorial 101: Hello Tinker
Tinker is a remote GPU service for LLM training and inference. You write training loops in Python on your local machine; Tinker executes the heavy GPU operations (forward passes, backpropagation, sampling) on remote workers.
```
Your machine (CPU) Tinker Service (GPU)
+-----------------------+ +------------------------+
| Python training loop | --------> | Forward/backward pass |
| Data preparation | <-------- | Optimizer steps |
| Evaluation logic | | Text generation |
+-----------------------+ +------------------------+
```
You control the logic. Tinker runs the compute.
""")
return
@app.cell
def _():
import warnings
warnings.filterwarnings("ignore", message="IProgress not found")
import tinker
from tinker import types
return tinker, types
@app.cell(hide_code=True)
def _(mo):
mo.md(r"""
## The client hierarchy
The entry point to Tinker is the **ServiceClient**. From it, you create specialized clients:
- **SamplingClient** -- generates text from a model (inference)
- **TrainingClient** -- runs forward/backward passes and optimizer steps (training)
Both talk to the same remote GPU workers. Let's start with the ServiceClient.
""")
return
@app.cell
def _(mo):
api_key = mo.ui.text(kind="password", label="Paste your Tinker API key")
api_key # noqa: B018
return (api_key,)
@app.cell
async def _(api_key, mo, tinker):
import os
mo.stop(
"TINKER_API_KEY" not in os.environ and not api_key.value,
"Paste your API key above",
)
if api_key.value:
os.environ["TINKER_API_KEY"] = api_key.value
# Create a ServiceClient. This reads TINKER_API_KEY from your environment.
service_client = tinker.ServiceClient()
# Check what models are available
capabilities = await service_client.get_server_capabilities_async()
print("Available models:")
for model in capabilities.supported_models:
print(f" - {model.model_name}")
return (service_client,)
@app.cell(hide_code=True)
def _(mo):
mo.md(r"""
## Sampling from a model
Let's create a **SamplingClient** to generate text. We will use `Qwen/Qwen3.5-9B-Base`, a base (non-chat-tuned) model, so we can feed it raw tokens and get a plain text completion -- no chat template required.
The sampling workflow is:
1. Create a `SamplingClient` with a base model name
2. Encode your prompt into tokens using the model's tokenizer
3. Call `sample()` with the prompt and sampling parameters
4. Decode the returned tokens back into text
""")
return
@app.cell
async def _(service_client):
MODEL_NAME = "Qwen/Qwen3.5-9B-Base"
# Create a sampling client -- this connects to a remote GPU worker
sampling_client = await service_client.create_sampling_client_async(base_model=MODEL_NAME)
# Get the tokenizer for encoding/decoding text
tokenizer = sampling_client.get_tokenizer()
return sampling_client, tokenizer
@app.cell
async def _(sampling_client, tokenizer, types):
# Encode a prompt into tokens
prompt_text = "The three largest cities in the world by population are"
print("Prompt tokens:", tokenizer.encode(prompt_text))
prompt = types.ModelInput.from_ints(tokenizer.encode(prompt_text))
# Sample a completion
params = types.SamplingParams(max_tokens=50, temperature=0.5)
result = await sampling_client.sample_async(
prompt=prompt, sampling_params=params, num_samples=1
)
# Decode and print
completion_tokens = result.sequences[0].tokens
print("Completion tokens:", completion_tokens)
print(prompt_text + tokenizer.decode(completion_tokens))
return prompt, prompt_text, result
@app.cell(hide_code=True)
def _(mo):
mo.md(r"""
## Inspecting the response
The `sample()` call returns a `SampleResponse` containing a list of `SampledSequence` objects. Each sequence has:
- `tokens` -- the generated token IDs
- `logprobs` -- log probability of each generated token (if requested)
- `stop_reason` -- why generation stopped (e.g., hit max tokens, hit a stop string)
""")
return
@app.cell
def _(result):
_seq = result.sequences[0]
print(f"Stop reason: {_seq.stop_reason}")
print(f"Tokens generated: {len(_seq.tokens)}")
print(f"Token IDs: {_seq.tokens[:10]} ...")
print(f"Log probs: {_seq.logprobs[:10]} ...") # first 10
return
@app.cell(hide_code=True)
def _(mo):
mo.md(r"""
You can also generate multiple samples at once by setting `num_samples`. Each sample is an independent completion from the same prompt.
""")
return
@app.cell
async def _(prompt, prompt_text, sampling_client, tokenizer, types):
result_1 = await sampling_client.sample_async(
prompt=prompt,
sampling_params=types.SamplingParams(max_tokens=50, temperature=0.7),
num_samples=3,
)
for i, _seq in enumerate(result_1.sequences):
text = tokenizer.decode(_seq.tokens)
print(f"Sample {i}: {prompt_text}{text}")
return
@app.cell(hide_code=True)
def _(mo):
mo.md(r"""
## What about training?
So far we have only done inference. The real power of Tinker is **training** -- running forward/backward passes and optimizer steps on remote GPUs while you control the training loop locally.
The workflow looks like this:
1. Create a **TrainingClient** with `service_client.create_lora_training_client()`
2. Prepare training data as `Datum` objects (input tokens + loss targets)
3. Call `training_client.forward_backward()` to compute gradients
4. Call `training_client.optim_step()` to update weights
5. Save weights and create a **SamplingClient** to evaluate the trained model
We will walk through this in the next tutorial.
""")
return
@app.cell(hide_code=True)
def _(mo):
mo.md(r"""
## Next steps
- **[Tutorial 102: First SFT](./102_first_sft.py)** -- Train a model with supervised fine-tuning
- **[Getting Started Guide](https://tinker-docs.thinkingmachines.ai/tinker/quickstart/)** -- Full walkthrough of training and sampling
- **[Available Models](https://tinker-docs.thinkingmachines.ai/tinker/models/)** -- All supported models and their characteristics
""")
return
if __name__ == "__main__":
app.run()