This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
The Speechless repository is a comprehensive LLM training and inference framework focused on building specialized code generation models. It includes fine-tuning capabilities, evaluation pipelines, and model serving infrastructure.
- Model Training: Full fine-tuning, LoRA, QLoRA, FSDP, and specialized methods like GRPO, PPO
- Model Evaluation: HumanEval, MultiPL-E, lm-evaluation-harness, CodeQwen evaluation
- Model Serving: OpenAI-compatible API servers (vLLM, Llama.cpp, MLX, SGLang)
- Data Processing: Dataset generation, cleaning, and formatting for various domains
- Quantization: GGUF support, AWQ, GPTQ conversion utilities
- Tool Integration: Framework for tool-calling and agent capabilities
speechless/finetune/- Core training pipeline with LoRA/QLoRA supportspeechless/eval/- Evaluation frameworks (HumanEval, MultiPL-E, etc.)speechless/api/- Model serving APIsspeechless/generate/- Model inference utilitiesspeechless/agents/- Tool-calling and agent frameworkspeechless/quant/- Model quantization utilities
# Install dependencies
pip install speechless
# or
poetry install
# Set environment variables
export SPEECHLESS_ROOT=/path/to/speechless# Initialize new training task
python -m speechless.finetune init --task_name my_task
# Run training
python -m speechless.finetune run --task_name my_task
# Merge LoRA adapters
python -m speechless.finetune merge --task_name my_task
# Alternative Makefile commands
make finetune_mistral_7b
make finetune_34b# HumanEval evaluation
export TEST_MODEL_PATH=/path/to/your/model
make humaneval
# MultiPL-E evaluation
make multiple_gen TEST_MODEL_PATH=/path/to/your/model
make multiple TEST_MODEL_PATH=/path/to/your/model
# lm-evaluation-harness
make lmeval TEST_MODEL_PATH=/path/to/your/model# Start API server (vLLM backend)
python -m speechless.api.server start --model /path/to/model --backbone vllm --port 5001
# Alternative Makefile command
make api_server TEST_MODEL_PATH=/path/to/model# Generate synthetic datasets
python scripts/generate_speechless_datasets/generate_speechless_coding_dataset.py
# Convert HF to GGUF
python -m speechless.quant llamacpp --model_path /path/to/hf/model --llamacpp_quant_type q4_k_mKey environment variables to set:
export MODELS_ROOT_DIR=/opt/local/llm_models/huggingface.co
export SPEECHLESS_DATA_DIR=/opt/local/datasets/speechless_data
export SPEECHLESS_ROOT=/path/to/speechlessTraining tasks are configured through task-specific directories containing:
task.env- Environment variables and hyperparametersrun_finetune.sh- Training scriptmerge.sh- Model merging scriptdataset/- Training data location
- Code Generation: HumanEval, MultiPL-E pass rates
- General benchmarks: lm-evaluation-harness results
- Tool-calling: ToolBench datasets performance
- SQL Generation: SQLEval benchmark
Supported model backends:
- Hugging Face Transformers
- GGUF (GGML quantized)
- AWQ quantized
- MLX (for Apple Silicon)
- vLLM (high-performance serving)
- Training Pipeline:
init → run → merge → eval → deploy - Evaluation Pipeline:
generate → evaluate → analyze results - Model Serving:
train → quantize → serve via API
- Training: PyTorch Lightning, Transformers, PEFT, DeepSpeed
- Optimization: LoRA, QLoRA, FSDP, Gradient Checkpointing
- Evaluation: Custom eval frameworks, benchmarks
- Serving: FastAPI, vLLM, Llama.cpp
- Data: JSONL format, conversation-style datasets
This framework emphasizes reproducibility and extensibility for LLM research and applied AI systems.