Skip to content
Agent NetworkkLLM
Blog

Introducing kLLM

kLLM brings a pull-and-run workflow to stock SGLang and vLLM and adds keys: models sealed under a key keep their tensor layout, stay locked on disk and run on unmodified engines.

kLLM team

Run any model. Protect what matters.

Today Agent Network (ANet) publishes kLLM, an open-source project under the Apache-2.0 license. kLLM brings a pull-and-run workflow to production inference engines and adds keys to it. A model sealed under a key keeps its tensor layout, so stock SGLang and vLLM serve it with their own kernels, batching and parallelism. On disk, its weights are locked: encrypted in place under the key. Loaded in a stock engine without the key, the sealed model produces no useful output. With kLLM and the key, it runs normally.

kLLM aims to be a better Ollama for production: the same pull-and-run workflow, on production inference engines rather than a desktop runtime, with keys. It is a layer on top of the inference engine, in the way LMCache is for KV caches. It does not replace SGLang or vLLM, and it does not fork them. Its scope is transformer models across modalities: dense and mixture-of-experts language models, image diffusion and video generation.

Why kLLM

Production inference engines such as SGLang and vLLM provide the batching, CUDA graphs and tensor parallelism that serving at scale requires. A model served on a stock engine is held by the serving host as readable weights, and the host tokenizes and detokenizes every request. Approaches that keep a model or its traffic away from the serving host typically modify the engine or replace it.

kLLM takes a third path. It changes the model, not the engine. The model is rewritten under a key before it reaches the GPU host, and the key and the tokenizer stay in a trusted domain. The engine runs unchanged.

How sealing works

kllm seal rewrites a Hugging Face checkpoint with static transforms derived from a key. Two transforms are available:

  • orth applies a random orthogonal basis to the residual stream.
  • perm applies a signed permutation. The sealed weights are the original values, reordered and sign-flipped.

Tensor names, shapes and dtype are unchanged. The stock engine therefore runs its own model classes, kernels, CUDA graphs, batching and tensor parallelism. The permuted tokenizer is encrypted under the key. It is decrypted only in a trusted domain, which is a confidential VM or, in client-keyed mode, the user's machine. It is never sent to the GPU.

The transforms are exact in exact arithmetic; float64 tests on tiny random models agree to 1e-9. Serving in bf16 changes accumulation order only.

By default, kllm seal then locks the weights. Each value is shifted, within the finite values of its format up to the tensor's largest power of two, by an AES-256-CTR key stream derived from the model key and the tensor's name. Locked files keep their names, shapes and dtypes and load in any engine, but compute nothing useful, and without the key nothing in them can be matched to the public base. kllm serve --key unlocks them once into a cache readable only by its owner (~/.cache/kllm/unlocked, or KLLM_UNLOCKED_DIR) before the engine starts; kllm seal --no-lock leaves them readable. Locking protects the files, not a running model: whoever runs the model holds the unlocked weights.

The transform and locking algorithms are open source, so anyone with the key can verify that a model is sealed as stated.

Sealing requires a validated implementation for each architecture. kLLM does not seal unknown architectures on the fly. Unsupported models are collected on a public wishlist, and the team adds architectures over time.

The workflow

kLLM targets Linux with an NVIDIA GPU and Python 3.10 or later. The commands below are on the main branch; no release has been tagged yet. Install from source today; pip install kllm becomes available with the first tagged release.

git clone https://github.com/ANetResearch/kLLM && cd kLLM
pip install -e '.[seal,serve]'      # SGLang; use '.[seal,vllm]' for vLLM
kllm doctor                         # driver, CUDA, disk, shared memory, engine packages

Pull a model and run it:

kllm pull qwen3:8b
kllm run qwen3:8b                                      # interactive chat; /bye to exit
kllm run qwen3:8b "Explain RoPE in two sentences."     # one-shot

kllm run starts a background engine server bound to 127.0.0.1 and opens a streaming chat. kllm pull accepts library aliases (name:tag), Hugging Face repositories (hf://org/repo), ModelScope repositories when ModelScope is installed, and local paths. kllm list, kllm ps, kllm stop, kllm rm, kllm logs, kllm show, kllm search and kllm check manage stored models and running servers; kllm check re-hashes a stored model against its store record and, for a catalog model, its published manifest.

For direct control, kllm serve wraps the stock engine and exposes its OpenAI-compatible API on port 8000. kllm client proxy and kllm client generate provide client-keyed serving, in which the key and the tokenizer stay on the user's machine and the GPU host receives only the cloud part of the sealed model. kllm cloud writes that part from a sealed model and unlocks its weights on the user's machine, because the GPU host computes with them.

Official sealed models and model keys

The team seals popular open models ahead of time and publishes them under huggingface.co/ANetResearch, with names such as Qwen3-8B-kllm. Each is a public base model at a pinned revision, sealed under perm with locked weights. The language and image models are also prepared for client-keyed serving. The artifacts are rolling out.

Official model Base model Modality Status
Qwen3-8B-kllm Qwen/Qwen3-8B Language, dense Rolling out
Qwen3-4B-Instruct-2507-kllm Qwen/Qwen3-4B-Instruct-2507 Language, dense Rolling out
Qwen3-32B-kllm Qwen/Qwen3-32B Language, dense Rolling out
Qwen2.5-7B-Instruct-kllm Qwen/Qwen2.5-7B-Instruct Language, dense Rolling out
Llama-3.1-8B-Instruct-kllm meta-llama/Llama-3.1-8B-Instruct Language, dense Rolling out
Qwen3-30B-A3B-Instruct-2507-kllm Qwen/Qwen3-30B-A3B-Instruct-2507 Language, mixture of experts Rolling out
Qwen3-30B-A3B-Instruct-2507-FP8-kllm Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 Language, mixture of experts, FP8 Rolling out
Z-Image-kllm Tongyi-MAI/Z-Image Image Rolling out
Z-Image-Turbo-kllm Tongyi-MAI/Z-Image-Turbo Image Rolling out
MiniMax-H3-FL2VA-kllm MiniMaxAI/MiniMax-H3 (FL2VA) Video with audio Held

Each official model is sealed under its own model key, derived from an offline master key held by the team. A key issued for one model reveals nothing about the master key or about any other model's key. Anyone may apply for model keys on the application page; one application can cover several models. The team reviews each application and sends the keys of the approved models, each as an ordinary, passphrase-protected kLLM key file, together with an explanation of what a model key is and what it is not.

kllm key import qwen3-8b-kllm Qwen3-8B-kllm.key        # the key file issued for this model
kllm pull qwen3:8b-sealed                              # ANetResearch/Qwen3-8B-kllm
kllm run qwen3:8b-sealed --key qwen3-8b-kllm

Every approved holder of one model's key holds the same key, so a model key is not a confidentiality boundary between its holders. The base models are public, so sealing them does not protect their weights. What an official model provides is locked files, permuted token ids instead of text in engine logs, request captures and stored payloads, and reproducibility.

Every official model carries a kllm-manifest.json with its base repository and revision, the key fingerprint and key version, the kLLM version and the SHA-256 digest of every file. With the key, kllm verify downloads the public base at the pinned revision, seals it again under the same key and transform, and compares the result with the published model tensor by tensor:

kllm verify Qwen3-8B-kllm --base Qwen/Qwen3-8B --key Qwen3-8B-kllm.key   # revision from the manifest

Under perm, every tensor must be equal to the bit, except tensors that sealing rewrites in floating point, such as the latent interfaces of a client-keyed Z-Image model, which may differ by one unit in the last place. The sealing code is open source, so a match shows that the published files are what that code makes of the public base under that key, and nothing else.

Llama-3.1-8B-Instruct-kllm is Built with Llama and is distributed under the Llama 3.1 Community License. MiniMax-H3-FL2VA-kllm is held: the MiniMax H3 Community License excludes the EU, the United Kingdom, the Republic of Korea and the United States, which a worldwide public repository cannot honor. MiniMax H3 can still be sealed locally where its licence permits.

Private sealing

To seal a model under a private key, generate the key and seal locally:

kllm key new private                                   # prompts for a passphrase
kllm pull qwen3:8b
kllm create qwen3-private --from qwen3:8b --key private --transform perm
kllm run qwen3-private --key private

The key and the sealing run stay on the local machine, and the sealed weights are locked by default. Serving with a key on an ordinary host is a development mode. To keep the key and the tokenizer off a GPU host, use client-keyed serving.

Model library

Plain serving covers every model the chosen engine supports. Sealing covers the architectures below.

Model Modality Library tags Transforms Engines
Qwen3 Language, dense 1.7b 4b 8b 14b 32b, each also as -fp8 orth, perm (FP8: perm) SGLang, vLLM
Qwen3-MoE Language, mixture of experts 30b-a3b 30b-a3b-fp8 orth, perm (FP8: perm) SGLang, vLLM
Qwen2.5 Language, dense 0.5b 1.5b 3b 7b orth, perm SGLang, vLLM
Llama 3.x Language, dense llama3.2: 1b 3b; llama3.1: 8b orth, perm SGLang, vLLM
MiniMax H3 Video with audio (FL2VA) fl2va perm (recommended), orth SGLang
Z-Image Image 6b Own scheme, client-keyed mode kLLM latent server

Before weight locking was added, sealing took 14 to 20 s for 1B to 2B models, 12 min for Qwen3-30B-A3B-FP8 on CPU, and about 15 min for MiniMax H3 FL2VA (135 GB) on one A40. In the catalog build on one A40 server, sealing and locking together took 36 s for Qwen3-1.7B (4.1 GB) and 102 s for Z-Image in client-keyed mode (20.5 GB). To request a model, open a model request or run kllm request MODEL_ID.

Measured results

Output throughput at concurrency 16, in tokens per second. Conditions: one NVIDIA A40 48 GB, SGLang 0.5.21 and vLLM 0.31.0, bf16, 1024 input and 128 output tokens, GPU shared with another tenant's light job, single runs with no variance reported.

Model Engine Plain perm orth
Qwen3-1.7B vLLM 1097 1094 1095
Qwen3-8B SGLang 333 333 332
  • In every measured configuration served with --key, sealed throughput matched plain serving and was never more than 0.3% lower.
  • Latency deltas are visible. Median time to first token at concurrency 1 for Qwen3-1.7B on SGLang was 98.0 ms plain and 105.2 ms perm. Time per output token at concurrency 16 for Qwen3-8B was 35.3 ms plain and 37.3 ms orth.
  • Client-keyed mode under vLLM cost 7% throughput at concurrency 16 (1036 to 963 tok/s).
  • Quality: Qwen3-1.7B under vLLM scored MMLU 0.5560 / 0.5562 / 0.5548 and GSM8K (strict) 0.6945 / 0.6952 / 0.6960 (plain / perm / orth), all within one standard error.
  • MiniMax H3 sealed under perm ran as fast as plain end to end on one A40 (8 steps: 220.4 s for both).

Full tables and methodology are in the benchmarking guide.

Security: design targets and measured limits

kLLM states its security as design targets. Without the key, recovering prompts or weights is meant to be an academic, high-cost attack outside the reach of commercial adversaries, with recovery cost far above generation cost. These targets are not guarantees of the current code. Its measured limits are published in full in the security documentation.

Deployment profiles are trust boundaries, not scores:

  • Official catalog. Public base models sealed under perm and locked, each under its own model key. Without the model key, the published files compute nothing useful and cannot be matched to the public base, and a key holder can reproduce them with kllm verify. It claims no confidentiality between key holders, who share each model's key, and no protection of the weights once unlocked to run.
  • Private Seal. kllm keygen and kllm seal, served with --client. The key and the tokenizer stay on the user's machine, and without the key the locked checkpoint cannot be matched to the public model it derives from. It does not protect the copy the cloud runs, which is unlocked, against a cloud that reads GPU memory or analyses the weights.
  • Attested CPU. kllm serve --key-source cdh --production in a confidential VM (TDX, SEV-SNP, Hygon CSV); the key source and the gateway are kLLM Enterprise. The key and the decrypted tokenizer stay in confidential-VM memory. It does not protect GPU memory on an ordinary passthrough GPU, and it has not yet been validated on TEE hardware.
  • Confidential GPU. Roadmap; not implemented.

Measured limits of the current code:

  • kllm serve --key outside a confidential VM decrypts the tokenizer into host /dev/shm, unlocks the weights into a cache on the host, and offers no protection from the host. It is a development mode.
  • Against an operator who reads GPU memory, user text reduces to a substitution cipher over tokens. On Qwen3-8B, unigram frequency alone decoded 23.8% of tokens.
  • Locking protects the files, not a running model. kllm serve --key unlocks the weights on the serving machine, and the keyless copy for client-keyed serving is unlocked before it reaches the cloud. Each tensor's largest binary exponent is stored in the clear.
  • From the unlocked weights, a holder of the public model that a sealed checkpoint derives from recovers the transforms and 98.5% of the vocabulary permutation. This includes fine-tunes of public base models.
  • MiniMax H3 decoded video and audio are in GPU memory in the clear. In client-keyed Z-Image, latents in GPU memory show the image.
  • kllm serve --client binds 0.0.0.0 by default and has no authentication. Bind it to 127.0.0.1 or place it behind an authenticating gateway.

See the attacks page for methods and data. Report vulnerabilities privately through a GitHub security advisory.

Same hardware, close to the same speed

On the same GPUs as stock vLLM, SGLang or Ollama, kLLM adds a degree of privacy protection. Under the measured conditions above, sealed throughput was close to plain serving on the same engine. Sealing and client-keyed serving require no special hardware.

On the roadmap, combined with confidential GPUs, transformed tensors can cross nodes without an extra encryption step. This is a design target for disaggregated serving. Confidential-GPU support is not implemented today.

kLLM Enterprise

kLLM Enterprise is licensed separately and adds services and components for production deployments:

  • Day-zero support for newly released models, which reach Enterprise customers before the public catalog of official sealed models.
  • Models sealed under a key issued for the customer alone.
  • Enterprise deployment and maintenance.
  • Attested key release from confidential-VM key sources and an attesting TLS gateway.

The open-source edition includes local sealing with locked weights, the curated library, the official sealed models with their model keys, kllm verify, the public wishlist, client-keyed serving and the user-side binding check, python -m kllm.keyed.attest. See the Enterprise page for contact details.

Research

The kLLM team publishes research on confidential transformer serving. The perm transform extends STIP's permutation approach to grouped-query attention, RoPE, MoE and FP8; the research systems themselves are separate from what kllm seal ships today.

Further work is under submission and will be released after publication.

Available and planned

Available on the main branch:

  • kllm pull, run, list, ps, stop, rm, logs, show, search, check, create, key, doctor, completion and request.
  • kllm keygen, seal (with locked weights), serve, unlock, verify and cloud on SGLang and vLLM; client-keyed serving with kllm client proxy and kllm client generate.
  • Sealing for Qwen3, Qwen3-MoE, Qwen2.5, Llama 3.x, MiniMax H3 and Z-Image.
  • Model key applications through the website and its review back office; official sealed models are rolling out.

Planned, not available today:

  • pip install kllm with the first tagged release; published container images; installers for macOS and Windows.
  • A resident daemon on 127.0.0.1:11400 that routes OpenAI requests by model, with keep-alive and admission control.
  • An Ollama-compatible API (/api/chat, /api/generate), a local web console and Modelfile-style recipes.
  • Confidential-GPU support and validation on TEE hardware.

Get started

kLLM is an open-source project of Agent Network.