Skip to content
Agent NetworkkLLM

Open sourceApache-2.0Agent Network

Run any model. Protect what matters.

kLLM runs open models from a curated library on SGLang and vLLM. A sealed model produces no useful output without its key. Pull a model, unlock it with a key, and run it.

Apply for model keysBrowse modelsDocumentation
# Pull a model from the library and chat with it on a local engine
$ kllm pull qwen3:8b
$ kllm run qwen3:8b
The server listens on 127.0.0.1 with an OpenAI-compatible API.

Install from source today; pip install kllm follows the first tagged release. Installation

How it works

Three steps from download to API

  1. 01

    Pull

    Download a model from the kLLM library or Hugging Face to the local store. A manifest records the hash of every file.

  2. 02

    Unlock with a key

    A sealed model is locked on disk and produces useful output only with its key. Apply for the model keys of official sealed models, or generate a key and seal models locally with the open-source tool.

  3. 03

    Run

    kLLM starts SGLang or vLLM on 127.0.0.1 and opens a chat interface. The same server exposes an OpenAI-compatible API.

How sealing works

Model library

Transformer models across modalities

Sealing requires a validated implementation for each architecture. The library expands as the team adds architectures; unsupported models can be requested.

Language

Dense and mixture-of-experts models: Qwen3, Qwen3-MoE, Qwen2.5 and Llama 3.x, including FP8 checkpoints.

Image diffusion

Diffusion transformer for image generation: Z-Image, with a client-keyed mode.

Video generation

Video and audio generation: MiniMax H3 FL2VA. Sealed under perm, it measured the same end-to-end time as plain serving (8 steps, one A40).

All modelsRequest a modelRolling outOfficial sealed models on Hugging Face

Official sealed models

Popular open models, sealed ahead of time

The kLLM team seals selected public models at a pinned revision, locks the weights and publishes each one under its own model key. With the key, anyone can check a published model against its public base with kllm verify.

  • Qwen3-8B-kllmRolling out

    Qwen/Qwen3-8B @b968826d9c

    Text · Apache-2.0 · client-keyed

  • Qwen3-4B-Instruct-2507-kllmRolling out

    Qwen/Qwen3-4B-Instruct-2507 @cdbee75f17

    Text · Apache-2.0 · client-keyed

  • Qwen3-32B-kllmRolling out

    Qwen/Qwen3-32B @9216db5781

    Text · Apache-2.0 · client-keyed

  • Qwen2.5-7B-Instruct-kllmRolling out

    Qwen/Qwen2.5-7B-Instruct @a09a35458c

    Text · Apache-2.0 · client-keyed

  • Llama-3.1-8B-Instruct-kllmRolling out

    meta-llama/Llama-3.1-8B-Instruct @0e9e39f249

    Text · Llama 3.1 Community License · client-keyed

  • Qwen3-30B-A3B-Instruct-2507-kllmRolling out

    Qwen/Qwen3-30B-A3B-Instruct-2507 @0d7cf23991

    Text (mixture of experts) · Apache-2.0 · client-keyed

  • Qwen3-30B-A3B-Instruct-2507-FP8-kllmRolling out

    Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 @5a5a776300

    Text (mixture of experts) · Apache-2.0 · client-keyed

  • Z-Image-kllmRolling out

    Tongyi-MAI/Z-Image @04cc4abb7c

    Text to image · Apache-2.0 · client-keyed

  • Z-Image-Turbo-kllmRolling out

    Tongyi-MAI/Z-Image-Turbo @f332072aa7

    Text to image · Apache-2.0 · client-keyed

  • MiniMax-H3-FL2VA-kllmHeld

    MiniMaxAI/MiniMax-H3 @42ed227ee7

    Text and images to video with audio · MiniMax H3 Community License

Apply for model keysOfficial models in the libraryArtifacts are rolling out to huggingface.co/ANetResearch; MiniMax-H3-FL2VA-kllm is held for licence reasons.

Engines

Stock engines, unmodified

kLLM is a layer on top of the inference engine, in the way LMCache is for KV caches. It does not replace or fork SGLang or vLLM. A sealed checkpoint keeps its file names, tensor names and shapes, so the engine loads it with its native model classes and kernels.

SGLang
0.5 or later; measured on 0.5.21
vLLM
0.11 or later; measured on 0.31.0
A plain checkpoint is sealed under a 256-bit model key. The sealed checkpoint keeps file names, tensor names and shapes, so stock SGLang or vLLM loads it unchanged on the GPU. The key and the decrypted, permuted tokenizer stay in the trusted domain: a confidential VM, or the user's own machine in client-keyed mode.
Seal on a trusted machine and serve on GPU infrastructure; the key and the decrypted tokenizer stay within the trusted boundary.

Measured results

Same hardware, close to the same speed

On the same GPUs as stock vLLM, SGLang or Ollama, kLLM adds a degree of privacy protection. In every measured configuration served with a key on one A40, sealed output throughput was never more than 0.3% below plain serving on the same engine.

Largest measured drop in sealed output throughput, keyed serving at concurrency 16
0.3%
Largest measured drop in sealed output throughput, keyed serving at concurrency 16
Sealing and locking Qwen3-1.7B (4.1 GB) on one A40 server, catalog build
36 s
Sealing and locking Qwen3-1.7B (4.1 GB) on one A40 server, catalog build
Throughput cost of client-keyed mode under vLLM at concurrency 16
7%
Throughput cost of client-keyed mode under vLLM at concurrency 16

Sealed throughput matched plain serving

Output throughput at concurrency 16, tok/s · plain, perm, orth · one NVIDIA A40 48 GB, bf16, 1024 input / 128 output tokens

Output throughput at concurrency 16 for plain, perm and orth serving of Qwen3-1.7B on vLLM 0.31.0 (1097, 1094, 1095 tok/s) and Qwen3-8B on SGLang 0.5.21 (333, 333, 332 tok/s).QWEN3-1.7B · VLLM 0.31.0plainplain: 1,0971,097permperm: 1,0941,094orthorth: 1,0951,095QWEN3-8B · SGLANG 0.5.21plainplain: 333333permperm: 333333orthorth: 332332
Capsule bars · output throughput · benchmarks/llm.sh · single runs on a shared GPU, no variance reported

First-token latency increased by up to 7.2 ms

Change in median time to first token versus plain serving, concurrency 1, ms · perm, orth · same hardware and conditions

Change in median time to first token at concurrency 1 versus plain serving. Qwen3-1.7B on SGLang 0.5.21: plus 7.2 ms (perm) and 6.3 ms (orth) from 98.0 ms; Qwen3-1.7B on vLLM 0.31.0: plus 1.0 and 3.1 ms from 65.3 ms; Qwen3-8B on SGLang: plus 0.6 and 0.9 ms from 199.5 ms.permorthplain1.7B · SGLang1.7B · SGLang, perm: +7.2+7.21.7B · SGLang, orth: +6.3+6.31.7B · vLLM1.7B · vLLM, perm: +1.0+1.01.7B · vLLM, orth: +3.1+3.18B · SGLang8B · SGLang, perm: +0.6+0.68B · SGLang, orth: +0.9+0.92 ms4 ms6 ms8 ms
Dot strip · median TTFT, difference from plain · benchmarks/llm.sh · single runs, no variance reported

Conditions: NVIDIA A40 48 GB; SGLang 0.5.21, vLLM 0.31.0; bf16; 1024 input / 128 output tokens; GPU shared with another tenant's light job; single runs, no variance reported. Throughput in some rows was higher when sealed, which is attributed to noise on the shared host. All published results

Roadmap

Combined with confidential GPUs, transformed tensors can cross nodes without an extra encryption step, a design target for disaggregated serving.

Security

Security as stated design targets

Without the key, recovering prompts or weights is intended to require an academic, high-cost attack beyond the reach of commercial adversaries, with recovery cost far above generation cost. These are design targets. The measured limits of the current release are published in full.

Security modelMeasured limits

Deployment profiles · trust boundaries, not scores

  • Official catalogModel keys

    Without the model key, the published files compute nothing useful and cannot be matched to the public base. A key holder can reproduce them from the public base with kllm verify.

    Does not claim: Confidentiality between key holders, who share each model's key, or protection of the weights once unlocked to run: the base models are public.

  • Private SealOpen source

    Without the key, the locked checkpoint computes nothing useful and cannot be matched to the public model it derives from. Cloud logs hold permuted token ids.

    Does not claim: Protection of the copy the cloud runs, which is unlocked, from a cloud that reads GPU memory or analyses the weights.

  • Attested CPUEnterprise

    Key and decrypted tokenizer stay in confidential-VM memory. Users can check that TLS terminates in the attested VM.

    Does not claim: GPU memory protection. Not yet validated on TEE hardware.

  • Confidential GPURoadmap

    Not implemented.

Locking protects the files, not a running model: whoever runs the model holds the unlocked weights. Outside a confidential VM, kllm serve --key is a development mode with no protection from the host. To an operator who reads GPU memory, user text reduces to a substitution cipher over tokens.

Research

Published research

The kLLM team publishes research on confidential transformer serving. The perm transform extends STIP's permutation approach to grouped-query attention, RoPE, MoE and FP8; the research systems themselves are separate from what kllm seal ships today.

  • ACM SIGCOMM 2025SCX: Stateless KV-Cache Encoding for Cloud-Scale Confidential Transformer ServingMu Yuan, Lan Zhang, Liekang Zeng, Siyang Jiang, Bufang Yang, Di Duan, Guoliang XingDOI 10.1145/3718958.3750509Code
  • NDSS 2026STIP: Three-Party Privacy-Preserving and Lossless Inference for Large Transformers in ProductionMu Yuan, Lan Zhang, Yihang Cheng, Miao-Hui Song, Guoliang Xing, Xiang-Yang LiDOI 10.14722/ndss.2026.230035

Research

Get started

Model keys

Apply for model keys

Each official sealed model on Hugging Face has its own model key. One application can cover several models. The team reviews applications and sends the keys of the approved models with instructions. Every approved holder of a model's key holds the same key, so it is not a confidentiality boundary.

Apply for model keys

Enterprise

kLLM Enterprise

Day-zero support for newly released models, models sealed under a key issued for the customer alone, enterprise deployment and maintenance, and attested key release with a gateway.

Contact the team