Rust + CUDA inference engine tuned separately for RTX PRO 6000 Blackwell and RTX 5090, with OpenAI-compatible serving and model-specific correctness gates.
Install · Models · Serving · Performance · Hosted API
Prebuilt binaries require Linux x86_64, an NVIDIA driver with CUDA 13 runtime support, and the
CUDA runtime libraries. The installer selects the GPU build, verifies its checksum, and does not
require nvcc.
curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
kernel-check # expect: ALL GREENStart a server with a supported public checkpoint. The first run downloads and caches the model:
MEMRA_MODELS="q38=hf:Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF:Q5K-mtp" \
memra-serverThe server listens on 127.0.0.1:8080. Send a streaming chat completion from another terminal:
curl -sS -N http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "q38",
"messages": [{"role": "user", "content": "Explain KV caching in one sentence."}],
"max_tokens": 128,
"stream": true
}'Use the cookbook for qualified model-and-card configurations. See installation for source builds and platform requirements.
Memra is narrow on purpose. The best execution path depends on the model, checkpoint, hardware, and workload, so Memra measures those combinations separately instead of promising that one loader or one default supports everything. A fast path is promoted only when its own correctness and performance evidence is current.
The project favors explicit support, reproducible receipts, and useful failures over silent fallbacks. Issues, model requests, hardware reports, and evidence-backed pull requests are welcome. Look elsewhere if you need broad hardware coverage, a large architecture catalog, or a Python library embedded in your application.
By model: Qwen3.8 27B · Qwen3.5 9B · Qwen3.6 27B · Qwen3.6 35B-A3B · Qwen-AgentWorld 35B-A3B · Ornith 1.5 35B-A3B · Ornith 1.0 9B · Ornith 1.0 35B · Gemma 4 12B · Gemma 4 26B-A4B · Gemma 4 31B · Gemma 4 E4B · Step 3.7 Flash · DeepSeek V4 Flash
By hardware: RTX PRO 6000 Blackwell · RTX 5090 / 50-series · H100 · Ada · B200
By workload: interactive agents · long-context and prefix reuse · batch throughput · multimodal · large models on multiple GPUs
| Document | Use it for |
|---|---|
| Installation | Prebuilt requirements, source builds, architecture selection |
| Model cards | Best starting path for each supported model |
| Hardware cards | Recommended use by GPU target |
| Workload cards | Recommended use by request shape |
| Cookbook | Copy-paste model and card configurations |
| Models | Supported checkpoints, formats, drafters, and hardware |
| Serving | HTTP contract, caching, auth, admission, multi-GPU, operations |
| API surfaces | Anthropic Messages and OpenAI Responses compatibility |
| Performance | Measurements, methodology, rigs, and receipts |
| Flags | Audited environment-variable reference |
| Testing | Correctness gates and evidence requirements |
| Architecture | Runtime structure and Blackwell implementation ledger |
| Decisions | Adopted and rejected design choices with evidence |
| Releases | Changelog and release artifacts |
Issues and requests are welcome:
- Report a bug
- Request a model or feature
- Submit a hardware validation
- Report a vulnerability privately
For a model request, include the exact checkpoint, quantization, target GPU, and why the model is useful. For a performance report, include the command, model artifact, hardware, and raw output.
Issues are welcome even without a proposed fix. Pull requests are welcome when they include the proof required for the affected path; GPU changes cannot be validated by GitHub's compile-only runners. Start with CONTRIBUTING.md.
MIT — see LICENSE. Built by Avi Fenesh at tiyuvta.