Question 1

What is kLLM?

Accepted Answer

kLLM runs open models from a curated library on SGLang and vLLM. It seals models under a key so that a sealed copy produces no useful output without that key. It is a layer on top of the inference engine and does not replace or fork SGLang or vLLM.

Question 2

How does kLLM relate to Ollama?

Accepted Answer

Ollama offers a one-command pull and run of plain models for local use. kLLM brings the same workflow, kllm pull and kllm run with a curated library, to production engines and adds model keys. These commands are on the main branch as part of the upcoming 0.2 line and are not yet in a tagged release. A resident local service with an Ollama-compatible API is planned.

Question 3

Which commands are available today?

Accepted Answer

Sealing and serving are available in 0.1.0.dev0: kllm keygen, kllm seal, kllm serve, kllm unlock and the kllm client commands. Weight locking, kllm verify and kllm cloud were added on the main branch. Model management commands, including kllm pull, kllm run, kllm list, kllm check, kllm create, kllm key and kllm doctor, are on the main branch as part of the upcoming 0.2 line.

Question 4

How is kLLM installed?

Accepted Answer

kLLM is not yet on PyPI. Clone the repository and install from source with pip install -e '.[seal,serve]', or with '.[seal,vllm]' for vLLM. pip install kllm becomes available with the first tagged release. A Docker image can be built from the repository Dockerfile; its base images are not pinned.

Question 5

Under which license is kLLM available?

Accepted Answer

kLLM is open source under Apache-2.0. The confidential-VM key sources and the attesting TLS gateway are kLLM Enterprise components and are licensed separately. The user-side binding check python -m kllm.keyed.attest is open source.

Question 6

Which models can be sealed?

Accepted Answer

The supported architectures are Qwen3, Qwen3-MoE, Qwen2 and Qwen2.5, and Llama 3.x, including FP8 checkpoints of these families with block-wise or per-tensor scales (perm only); the MiniMax H3 FL2VA video pipeline; and Z-Image in client-keyed mode. kllm seal rejects other architectures with an error that lists the supported ones. Plain serving without a key covers every model the chosen engine supports.

Question 7

Can an unsupported model be added?

Accepted Answer

Sealing requires a validated implementation for each architecture, so unsupported models are not sealed on the fly. Request them on the public wishlist with the Model request issue form on GitHub. Requests are prioritized by demand and by the work each architecture requires.

Question 8

Which inference engine should be used?

Accepted Answer

SGLang is the default. vLLM is selected with --engine vllm. Both run sealed language models under orth and perm, FP8 checkpoints and client-keyed mode. Only SGLang serves MiniMax H3, runs the INT8 DiT kernels and uses the built-in recipes.

Question 9

Does kLLM fork or patch SGLang or vLLM?

Accepted Answer

No. A sealed checkpoint keeps its file layout, tensor names, shapes and dtypes. The engine loads it with its own model classes and runs its own kernels, batching and parallelism. The one exception is the optional INT8 DiT kernels for MiniMax H3, which replace SGLang's attention and linear implementations inside the H3 DiT blocks when requested with --int8.

Question 10

Does sealing slow down inference?

Accepted Answer

Served with a key (--key), sealed output throughput matched plain serving in every measured configuration on one NVIDIA A40 48 GB, never more than 0.3% lower at concurrency 16. Some latency differences are visible: the median time to first token of Qwen3-1.7B on SGLang at concurrency 1 was 98.0 ms plain and 105.2 ms under perm. Client-keyed serving under vLLM cost 7% throughput at concurrency 16. All figures are single runs on a GPU shared with another tenant's light job.

Question 11

Does sealing change the model's outputs?

Accepted Answer

The transforms are exact in exact arithmetic; tests check this in float64 to 1e-9 on small random models. In bf16 serving, only the order of floating-point accumulation changes. On Qwen3-1.7B under vLLM, MMLU was 0.5560, 0.5562 and 0.5548, and GSM8K (strict) was 0.6945, 0.6952 and 0.6960 for plain, perm and orth, all within one standard error.

Question 12

What are the official sealed models?

Accepted Answer

Popular open models that the kLLM team seals ahead of time at a pinned revision, under perm and with locked weights, and publishes under ANetResearch on Hugging Face with the suffix -kllm, for example Qwen3-8B-kllm. The artifacts are rolling out. Each published model carries kllm-manifest.json with its base and revision, the key fingerprint and key version, the kLLM version and the SHA-256 digest of every file. MiniMax-H3-FL2VA-kllm is held for licence reasons and is not published.

Question 13

What are model keys?

Accepted Answer

Each official sealed model is sealed under its own model key, derived from an offline master key held by the kLLM team. A key issued for one model reveals nothing about the master key or about any other model's key. Every approved holder of one model's key holds the same key, so a model key provides no confidentiality between its holders.

Question 14

How are model keys obtained?

Accepted Answer

Apply on the application page, select the official models to evaluate and confirm the email address. One application can cover several models. The team reviews the application and delivers the key files of the approved models through one-time, expiring download links, with an explanation of what a model key is and is not. Key files are never sent as email attachments. Each key is an ordinary, passphrase-protected kLLM key file, so kllm key import, kllm run --key and kllm serve --key accept it.

Question 15

How is an official model run with its key?

Accepted Answer

Import the key file once, then pull and run the model: kllm key import qwen3-8b-kllm Qwen3-8B-kllm.key, kllm pull qwen3:8b-sealed and kllm run qwen3:8b-sealed --key qwen3-8b-kllm. kllm pull checks every file against the published manifest, and kllm run unlocks the weights on the local machine. For client-keyed serving, kllm cloud Qwen3-8B-kllm -o Qwen3-8B-kllm-cloud --key Qwen3-8B-kllm.key writes the keyless copy for the cloud; the key and the tokenizer stay with the user.

Question 16

How can a published model be checked?

Accepted Answer

With the model key, kllm verify MODEL --base REPO --revision REV --key KEYFILE downloads the public base at the pinned revision, seals it again under the same key and transform in a temporary directory, and compares the result with the published model tensor by tensor, unlocked, together with the tokenizer, every other file and the manifest. Under perm, every tensor must be equal to the bit, except tensors that sealing rewrites in floating point, such as the latent interfaces of a client-keyed Z-Image model, which may differ by one unit in the last place. kllm check re-hashes a model in the local store against its manifest; it does not need the key.

Question 17

Can a model key protect private prompts or models?

Accepted Answer

No. Every holder of an official model's key holds the same key, and the official models are sealed copies of public models. Treat prompts and outputs processed under an official model's key as not confidential. For data or weights that must stay confidential, generate a private key, seal the model locally and choose a deployment profile that matches the adversary.

Question 18

What happens if a key or its passphrase is lost?

Accepted Answer

A model sealed under a private key can no longer be served. kLLM keeps no copy of a private key and has no recovery mechanism. Back up the key file with its passphrase, or seal the original checkpoint again under a new key.

Question 19

Is sealing a form of encryption?

Accepted Answer

Sealing itself is not. It rewrites the checkpoint with static transforms derived from a 256-bit model key, so the network computes the same function in a secret basis. Two parts of a sealed model are encrypted at rest: the permuted tokenizer, under AES-GCM, and, unless the model was sealed with --no-lock, the weights, which are locked in place with AES-256-CTR. The weights are unlocked before the engine loads them, so the computation runs on unlocked weights. Against an operator who reads GPU memory, user text reduces to a substitution cipher over tokens.

Question 20

What does the cloud see?

Accepted Answer

Visibility depends on the deployment profile, which is a trust boundary rather than a score. Under Private Seal, the key and the tokenizer stay on the user's machine, and cloud logs hold token ids under the vocabulary permutation. The cloud serves a copy whose weights are unlocked on the user's side, and a cloud that reads GPU memory or analyses the weights is not defended against. Under Attested CPU, the key and the decrypted tokenizer stay in confidential-VM memory; GPU memory is not protected, and the profile has not yet been validated on TEE hardware.

Question 21

Is kllm serve --key safe on an ordinary cloud host?

Accepted Answer

No. Outside a confidential VM, it unlocks the weights into its cache and decrypts the tokenizer into host /dev/shm, and offers no protection from that host. It is a development mode. To keep the key off the cloud host, use client-keyed serving.

Question 22

Does sealing protect the weights of a fine-tuned model?

Accepted Answer

Locked files cannot be matched to the public base without the key. Once unlocked to run, they can: whoever runs the model holds the unlocked weights. A fine-tune of a public base is recoverable from the unlocked sealed weights under both transforms: under perm at any fine-tune size, under orth for weight changes up to 3% of the weight standard deviation. The same analysis recovers the transforms and 98.5% of the vocabulary permutation from a sealed copy of an unmodified public model. For weights with no public relative, there is no known attack under orth, and no proof of security either.

Question 23

Are confidential GPUs supported?

Accepted Answer

No. The serving GPU is an ordinary device whose memory the cloud can read. Confidential-GPU support is on the roadmap.

Question 24

How are security issues reported?

Accepted Answer

Report them privately through GitHub Security Advisories on the kLLM repository, as described in SECURITY.md. Do not open public issues for vulnerabilities.

Question 25

What is kLLM Enterprise?

Accepted Answer

It is the commercial offering of Agent Network around kLLM: day-zero support for newly released models, which reach Enterprise customers before the public catalog; models sealed under a key issued for the customer alone; enterprise deployment and maintenance; and attested key release with a gateway for confidential VMs.

Question 26

How does kLLM relate to SCX and STIP?

Accepted Answer

The kLLM team publishes research on confidential transformer serving. The perm transform extends STIP's permutation approach to grouped-query attention, RoPE, MoE and FP8. The research systems themselves are separate from what kllm seal ships today.

Frequently asked questions

General

Models and engines

Official models and model keys

Security

Enterprise and research

Further questions