Download
Install kLLM
kLLM targets Linux with an NVIDIA GPU and Python 3.10 or later. It installs from source today; pip install kllm becomes available with the first tagged release.
From source
Install from source
Choose the engine extra for the GPU host. The client machine in client-keyed mode needs no engine.
# Clone the repository and install with sealing and the SGLang engine
$ git clone https://github.com/ANetResearch/kLLM && cd kLLM
$ pip install -e '.[seal,serve]'
# Check the host
$ kllm doctor
pip install kllm with the first tagged release; installers for macOS and Windows; published container images.
Official models
Run an official sealed model
Official sealed models are rolling out under ANetResearch on Hugging Face, each under its own model key. Apply for the keys of the models to evaluate; each key arrives as a passphrase-protected kLLM key file.
The published weights are locked. The first start unlocks them into ~/.cache/kllm/unlocked (or KLLM_UNLOCKED_DIR), so the host needs free disk space for a second copy of the weights. Unlocking ran at about 0.7 GB/s on a GPU and 0.1 GB/s on CPUs, once per key and file set.
# Import the key file issued for Qwen3-8B-kllm, then pull and run the model
$ kllm key import qwen3-8b-kllm Qwen3-8B-kllm.key
$ kllm pull qwen3:8b-sealed
$ kllm run qwen3:8b-sealed --key qwen3-8b-kllm
Extras
Optional dependencies
Install only the extras a host needs. Extras combine, for example '.[seal,serve]'.
| Extra | Installs | Purpose |
|---|---|---|
| seal | Key handling, torch, safetensors, numpy | Generate keys and seal checkpoints |
| serve | sglang 0.5 or later | Serve on SGLang |
| vllm | vllm 0.11 or later | Serve on vLLM |
| client | seal, transformers, fastapi, uvicorn | Client-keyed mode for language models: local tokenization and the local proxy |
| image | seal, diffusers 0.36 or later, transformers, fastapi, uvicorn, pillow | Z-Image sealing, serving and local decoding |
| int8 | triton 3 or later, comfy-kitchen | Optional INT8 DiT kernels for MiniMax H3, separate from sealing |
| keyed | cryptography 42 or later | Key files only |
| dev | seal, pytest, ruff | Development |
Container
Build the Docker image
The Dockerfile installs kLLM on top of an engine image: SGLang by default, or vLLM through a build argument. The base images are not pinned to a version; pin them for production use.
# Build on the SGLang engine image (default)
$ docker build -t kllm:sglang .
# Or build on the vLLM engine image
$ docker build -t kllm:vllm --build-arg BASE=vllm/vllm-openai:latest .
# Serve a sealed model; KLLM_PASSPHRASE is read from the environment
$ docker run --gpus all --shm-size 32g -v ~/models:/models -v ~/.kllm:/root/.kllm -e KLLM_PASSPHRASE -p 8000:8000 kllm:sglang serve /models/Qwen3-8B-sealed --key
No container image is published yet; build the image from the repository. Docker guide
Host check
Check the host with kllm doctor
Before pulling or serving a model, kllm doctor checks the host and reports each check as ok, warn or fail. Add --json for machine-readable output.
- Python version
- Engine and sealing packages
- NVIDIA driver and CUDA
- GPUs and their memory
- Free disk in the kLLM home
- Shared memory in /dev/shm
- Running servers and free ports
- kLLM home directory
- Key-file permissions
# Search the library and download a model
$ kllm search qwen
$ kllm pull qwen3:8b
# Chat with it; the engine server listens on 127.0.0.1
$ kllm run qwen3:8b
# Shell completion for bash, zsh or fish
$ kllm completion bash > ~/.local/share/bash-completion/completions/kllm
Platforms
Platform matrix
Serving requires an NVIDIA GPU supported by the chosen engine. Sealing and the client-side tools do not.
| Platform | Serve | Seal and client tools | Status | Notes |
|---|---|---|---|---|
| Linux, NVIDIA GPU | SGLang, vLLM | Yes | Supported | The primary target. Measured on NVIDIA A40 48 GB GPUs. |
| Linux, CPU only | No | Yes | Partial | Sealing runs on the CPU; Qwen3-30B-A3B-FP8 sealed in 12 min, measured before weight locking was added. |
| Confidential VM (TDX, SEV-SNP, Hygon CSV) | SGLang, vLLM | Yes | Partial | Attested key release is part of kLLM Enterprise; not yet validated on TEE hardware. |
| macOS | No | Client-side Z-Image decoding measured | Planned | Not a serving target. The Z-Image client-side VAE decoded in 3.1 s on an Apple M3 (MPS, fp16). Installer planned. |
| Windows | No | Not tested | Planned | Installer planned. |