Skip to content

README.md

Latest commit

 

History

History
340 lines (261 loc) · 12.6 KB

README.md

File metadata and controls

340 lines (261 loc) · 12.6 KB
Raw

Hyperloom Quickstart

This README is the main entry point for setting up Hyperloom and launching the model demo skills. The recommended customer path is to prepare a dedicated workspace, open that directory in Cursor, Claude Code, or Codex, and install the wheel into the current directory with pip install --target .. The source-clone path is kept at the end for developers and manual debugging.

Recommended Path: Install the Wheel

Use this path for customer demos and clean validation. The current directory is both the install target and the agent workspace. Prepare a dedicated clean directory first, then open that directory in Cursor, Claude Code, or Codex before running the install command.

Recommended run mode: Docker. Running the demos in the provided ROCm container ships a validated ROCm + framework stack, gives reproducible results, and keeps your host untouched. Bare-metal mode is for advanced users: it depends on your host's existing ROCm/torch and installs framework components into your environment, which can cause environment-specific issues or conflicts. Prefer Docker for a validated, reproducible stack.

Prerequisites

  • Python 3.10+ and pip.
  • Access to the Anthropic LLM provider.
  • Access to the published GitHub release wheel, or a locally downloaded Hyperloom wheel.
  • A dedicated workspace directory opened in the user's agent.

From the agent terminal in that workspace, install the published release wheel into the current directory:

python3 -m pip install \
  https://github.com/AMD-AGI/Hyperloom/releases/download/v1.0.0a1/hyperloom_inference_optimizer-1.0.0a1-py3-none-any.whl \
  --target .

It is normal for the current directory to contain many Python package directories after install; users do not need to inspect them. Do not use an existing project directory unless it is acceptable for Hyperloom to create or update .env there.

Run /hyperloom-setup

With the agent still opened in the same workspace, run:

/hyperloom-setup

In Cursor and Claude Code, use /hyperloom-setup; in Codex, use $hyperloom-setup.

That command runs the setup skill installed from src/hyperloom/skills/hyperloom-setup/SKILL.md.

The setup skill is interactive. It creates or updates .env in the current workspace, records the selected run scenario, and stops before launching an optimization. Run /hyperloom-setup once per workspace; demo skills reuse the values already written to .env.

It asks for these values with a fixed option order:

  1. Anthropic URL:
    • Use default (https://api.anthropic.com)
    • Use AMD gateway (https://llm-api.amd.com/anthropic)
    • Custom
  2. Model:
    • Use default (claude-opus-4-8)
    • Custom
  3. Secrets:
    • Setup writes placeholders in .env.
    • Edit secrets directly in .env; never paste API keys into chat.
    • If .env already exists, setup preserves unrelated keys but updates the Hyperloom setup keys selected in this run.
  4. USER_DATA_PATH:
    • Default: <workspace>/session
    • Custom path
  5. Run mode, recorded in .env as HYPERLOOM_RUN_MODE:
    • docker
    • baremetal

Setup Scenarios

Hyperloom supports two local setup scenarios. Pick the one matching where the serving framework will run.

Scenario A: Bare Metal

Use this when the current host is the AMD GPU host where Hyperloom will run directly.

Requirements:

  • ROCm runtime and ROCm torch are already installed.
  • git is available for dependency checkouts.
  • A serving framework is either already installed, or setup may install one.

In this scenario, /hyperloom-setup runs the packaged setup backend on the host:

export REPO_ROOT="$(pwd -P)"
PYTHONPATH="$REPO_ROOT" python3 -m hyperloom.inference_optimizer.setup

The backend runs install_baremetal.sh in five phases:

  1. Base preflight: checks ROCm, GPU arch, ROCm torch, torch/triton alignment, and serving framework imports.
  2. Framework install: optionally installs the SGLang or vLLM framework layer.
  3. ROCm hotfix: applies the profiler hotfix when the ROCm stack is eligible.
  4. Credentials: resolves LLM gateway credentials into .env.
  5. Runtime env: persists bare-metal runtime vars (framework, ROCm/venv roots, etc.) into .env.

Scenario B: Bare Metal + Docker

Use this when the workload will run inside a ROCm container. This is the recommended path when the host does not have ROCm torch or a serving framework installed, or when the serving framework should come from a known container image.

Requirements:

  • Docker with AMD GPU access (/dev/kfd, /dev/dri) on the selected target host.
  • A ROCm container image that already ships the serving framework, such as SGLang or vLLM.

In this scenario, /hyperloom-setup writes .env only and does not start a container. The selected demo skill owns the container lifecycle.

If Slurm is available, setup also checks the current user's allocation so Docker runs on the intended single GPU host instead of a login host. The user chooses whether Docker should run on:

  • the current host;
  • one allocated Slurm host;
  • a custom host.

The chosen host is written to .env:

HYPERLOOM_DOCKER_TARGET_HOST=<hostname>

The demo skill reads this value to target the chosen host.

Environment Written by Setup

LLM defaults:

Mode Required secret Default base URL Default model
Anthropic ANTHROPIC_API_KEY https://api.anthropic.com CLAUDE_MODEL=claude-opus-4-8

Setup creates or updates .env in the current workspace and writes the resolved values there.

Common keys:

  • USER_DATA_PATH
  • HYPERLOOM_RUN_MODE
  • HYPERLOOM_DOCKER_TARGET_HOST (only when HYPERLOOM_RUN_MODE=docker)

Bare-metal setup may also write runtime vars such as FRAMEWORK, ROCM_PATH, VIRTUAL_ENV, and VLLM_VENV_ROOT. Kernel-agent paths (MAGPIE_PATH, INFERENCEX_PATH, TRACELENS_ROOT, GEAK_ROOT) are added later by the workload skill's install.sh.

.env in the current workspace is the single source of truth; no extra script needs sourcing. Setup only needs to run again when changing the LLM provider, base URL, model, USER_DATA_PATH, run mode, Docker target host, or bare-metal framework setup choice.

Run a Demo

When setup finishes in baremetal mode (and FRAMEWORK is set), or when .env is written in docker mode, the setup skill offers a model demo run and hands off to the matching demo skill. Pick a preset or the advanced custom run:

  • 3h — Qwen3-8B, short no-kernel run; best for a first end-to-end check.
  • 12h — Qwen3-14B-FP8, medium-length FP8 run.
  • custom advanced — user-selected model, framework, TP/EP, concurrency, ISL/OSL, precision, budget, phase toggles, and advanced CLI flags.

The preset demos reuse the values already in .env, so nothing is re-entered. The custom advanced run also reuses setup values, then asks for workload and phase choices before launch.

Use a Custom Model

For a custom model with a preset workload, start from one of the fixed demo skills above. Pick the demo whose runtime shape is closest to the model and experiment you want to run, then provide your model path when the skill asks for it. You can also set it before launching the demo:

export MODEL_PATH=/path/to/your/model

MODEL_PATH should point to a model directory that the selected serving framework can load; a local Hugging Face-style directory should contain config.json. When MODEL_PATH is set and valid, the demo skill uses that directory instead of downloading its default model.

The fixed demo skill is still a preset workload. Replacing the model path does not automatically retune tensor parallelism, concurrency, input/output lengths, precision, or the run budget. If the custom model is much larger, smaller, or uses a different architecture than the preset model, use custom advanced to choose the model, framework, TP/EP, CONC, ISL/OSL, precision, budget, skip flags, and other optimizer CLI flags explicitly.

What to Expect During a Demo

Demo optimizations are long-running background jobs. The agent should not stream every debug log line, but it should make progress visible before and after launch.

Before launch, expect a short plan that includes the resolved model path, run mode, framework, TP, concurrency, ISL/OSL, precision, run budget, and USER_DATA_PATH. After launch, expect the optimizer PID, run log path, launch-info JSON path, session directory, state.json path, and the initial health check result.

During the run, the agent should report a concise status summary about every 300 seconds. Useful fields include whether the process is still alive, the current phase, stop_reason, baseline throughput, current best throughput, cumulative gain, latest benchmark result or candidate decision, and the most relevant recent log lines. Secrets such as API keys, tokens, and custom headers must never be printed.

Troubleshooting

  • If the current workspace contains many package folders after pip install --target ., that is expected.
  • If /hyperloom-setup is not visible, confirm the setup skill exists under the current workspace. It is installed to .claude/skills/hyperloom-setup/ (Claude Code), .cursor/skills/hyperloom-setup/ (Cursor) and .agents/skills/hyperloom-setup/ (Cursor/Codex); restart the agent if needed.
  • ImportError: libamdhip64.so.7 or libhipblas.so.3 means the installed framework torch wheel expects different ROCm user-space libraries; align ROCM_PATH and LD_LIBRARY_PATH.
  • hipDeviceAttributePciChipId missing during AITER build means hipcc is using older ROCm headers; put the matching ROCm bin first on PATH.

Source Checkout / Manual Path

Use this path only when developing Hyperloom, testing local source changes, or debugging setup internals. Customers should prefer the wheel install above.

Clone the repository:

git clone https://github.com/AMD-AGI/Hyperloom.git
cd Hyperloom

In source mode, the agent workspace is the repository root. Create .env yourself with placeholders and fill in the real values before launching. Never paste API keys into chat.

cat > .env <<'EOF'
ANTHROPIC_API_KEY=<PLEASE_FILL_IN>
ANTHROPIC_BASE_URL=https://api.anthropic.com
CLAUDE_MODEL=claude-opus-4-8
# Writable artifact root for runtime files, dependency checkouts, logs,
# optimizer runs, and generated env files. Set an absolute path you own.
USER_DATA_PATH=<PLEASE_FILL_IN>
HYPERLOOM_RUN_MODE=baremetal
EOF

Bare metal (source)

Make sure the host already provides the required base environment:

  • ROCm runtime and a ROCm-built torch.
  • A serving framework (SGLang or vLLM) importable in the active Python.
  • git for the dependency checkouts the optimization skill performs.

With that in place, open the repository root in the agent and paste a launch prompt, filling in your workload:

@src/hyperloom/inference_optimizer/SKILL.md

Optimize inference for this workload:
- Model: /path/to/your/model
- Framework: sglang
- GPU: MI300X
- TP: 1
- CONC: 64
- ISL: 1024
- OSL: 1024
- Goal: improve throughput by at least 10%
- Budget: 24 hours

Requirements:
1. Report the session ID, log path, PID, and initial health check result.
2. Monitor the process every 300s until the optimization is complete or failed.

Docker (source)

Use a ROCm image that already ships the serving framework, so nothing is installed inside the container beyond Hyperloom's runtime deps:

  • vllm: docker.io/primussafe/vllm-openai-rocm:v0.21.0-rocm720-profilerfix
  • sglang MI300X: docker.io/primussafe/sglang:v0.5.12-rocm720-mi30x-profilerfix
  • sglang MI355X: docker.io/primussafe/sglang:v0.5.12-rocm720-mi35x-profilerfix

Start a long-running container from the repo root, mounting it at the same path so .env, logs, and session artifacts stay valid:

export HYPERLOOM_IMAGE=docker.io/primussafe/vllm-openai-rocm:v0.21.0-rocm720-profilerfix
export REPO_ROOT="$(pwd -P)"
docker run -d \
  --name "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" \
  --shm-size "${HYPERLOOM_SHM_SIZE:-64g}" \
  --entrypoint tail \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add video \
  -v "$REPO_ROOT:$REPO_ROOT" \
  "$HYPERLOOM_IMAGE" \
  -f /dev/null

Then run all Hyperloom commands inside that container with docker exec -w "$REPO_ROOT" "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" ..., using PYTHONPATH="$REPO_ROOT/src" so the source checkout is importable. Use the same launch prompt as bare metal above.