Skip to content
Agent NetworkkLLM

How it works

Sealed models on stock engines

kLLM seals a checkpoint under a key and serves it on stock SGLang and vLLM. It is a layer on top of the inference engine, in the way LMCache is for KV caches. It neither replaces nor forks the engine.

ConceptsSealing in detail

Architecture

A plain checkpoint is sealed under a 256-bit model key. The sealed checkpoint keeps file names, tensor names and shapes, so stock SGLang or vLLM loads it unchanged on the GPU. The key and the decrypted, permuted tokenizer stay in the trusted domain: a confidential VM, or the user's own machine in client-keyed mode.
Seal on a trusted machine and serve on GPU infrastructure; the key and the decrypted tokenizer stay within the trusted boundary.
  1. 01

    Seal the checkpoint

    kllm seal rewrites a Hugging Face checkpoint with static transforms derived from the key. orth applies a random orthogonal basis to the residual stream. perm applies a signed permutation, so the sealed weights are the original values reordered and sign-flipped. Tensor names, shapes and dtype do not change.

  2. 02

    Lock the weights

    By default kllm seal then encrypts every weight in place with AES-256-CTR under a key derived from the model key, within the finite values of its format. The files still load in any engine, but compute nothing useful and cannot be matched to the public base without the key. --no-lock skips this step.

  3. 03

    Serve on a stock engine

    kllm serve --key unlocks the weights once into a cache readable only by its owner and starts the engine on them. Because the checkpoint keeps its layout, the stock engine runs its own model classes, kernels, CUDA graphs, batching and tensor parallelism.

  4. 04

    Keep the key in a trusted domain

    The permuted tokenizer is encrypted under the key. It is decrypted only in a trusted domain, a confidential VM or the user's own machine in client-keyed mode, and it is never sent to the GPU.

Transforms

Two transforms: orth and perm

Both transforms are exact in exact arithmetic. In bf16 serving they change accumulation order only: the mean absolute change in log-probability against plain serving was 0.03 to 0.10, comparable to switching between fp16 and bf16 (0.053).

Comparison of the orth and perm transforms
orthperm
OperationRandom orthogonal basis for the residual streamSigned permutation: original values reordered and sign-flipped
ExactnessExact in exact arithmetic; float64 tests agree to 1e-9 on tiny random modelsSealed weights are bit-exact rearrangements of the originals
FP8 checkpointsNot supportedBlock-wise and per-tensor FP8
Weights derived from a public modelRecoverable from the unlocked sealed weights for weight changes up to 3% of the weight standard deviation; weights with no public relative have no known attack, and no proof eitherRecoverable from the unlocked sealed weights at any fine-tune size

Choosing a transform

Locked weights

Locked at rest, unlocked to run

Locking protects the files, not a running model. Unless a model is sealed with --no-lock, its weights stay encrypted until kLLM unlocks them on the machine that runs it.

  • On disk

    Every value is shifted, within the finite values of its format up to the tensor's largest power of two, by an AES-256-CTR key stream derived from the model key and the tensor's name. The files keep their names, shapes and dtypes and load in any engine, but compute nothing useful. Only each tensor's largest binary exponent is stored in the clear.

  • Served with the key

    kllm serve --key unlocks the weights once into ~/.cache/kllm/unlocked (KLLM_UNLOCKED_DIR, mode 0700) and serves the result, which is the sealed model. Unlocking ran at about 0.7 GB/s on a GPU and 0.1 GB/s on CPUs, once per key and file set.

  • Client-keyed

    kllm cloud --key unlocks the weights on the user's side into the keyless copy that the cloud serves with kllm serve --client; kllm seal --client writes the same copy when sealing. The key and the tokenizer stay with the user.

Whoever runs the model holds the unlocked weights, and with the public base at hand they can be matched to it. The measured attacks on sealed weights apply to the unlocked weights.

Official models

Official sealed models and model keys

The kLLM team seals selected public models at a pinned revision under perm, locks the weights and publishes them under ANetResearch on Hugging Face with names such as Qwen3-8B-kllm. Each model has its own model key, derived with HKDF-SHA256 from an offline master key. A key issued for one model reveals nothing about the master key or about any other model's key, and raising a model's key version rotates its key.

Approved applicants receive the keys of the models they apply for as ordinary, passphrase-protected kLLM key files. Every holder of one model's key holds the same key. Each published model carries kllm-manifest.json with its base, revision, key fingerprint and version, kLLM version and the SHA-256 digest of every file, and kllm verify rebuilds it from the public base for comparison.

Official modelsThe catalogModel keys

Where the key lives

Three ways to serve a sealed model

The protection a deployment offers depends on where the key and the decrypted tokenizer live, not on the engine.

  • Client-keyed

    Private Seal

    Key and tokenizer on the user's machine

    The GPU host receives the sealed weights and configuration only, with no key and no tokenizer, and serves token ids in and out (kllm serve --client). The weights are unlocked on the user's side first (kllm cloud --key), because the GPU host computes with them. A local proxy on the user's machine (kllm client proxy) tokenizes requests and exposes an OpenAI-compatible API on 127.0.0.1.

    Documentation

  • Confidential VM

    Attested CPU · Enterprise

    Key and tokenizer in confidential-VM memory

    kllm serve --key-source cdh --production releases the key into a confidential VM after attestation, and users can check that TLS terminates in the attested VM. The GPU is an ordinary passthrough device, so GPU memory is not protected. Not yet validated on TEE hardware.

    Documentation

  • Development mode

    No protection from the host

    Tokenizer decrypted into host /dev/shm

    kllm serve --key on an ordinary host unlocks the weights into its cache and decrypts the tokenizer into host /dev/shm. It is convenient for development and offers no protection from the host. kllm run on a sealed model uses this mode on the local machine.

    Documentation

Profiles, claims and measured limits

Workflow

Pull, unlock with a key, run

The library, the local store and named keys wrap the same sealing and serving commands. kllm run starts SGLang or vLLM in the background on 127.0.0.1, unlocking a sealed model with its key, and opens a chat.

# Pull a plain model from the library and chat with it
$ kllm pull qwen3:8b
$ kllm run qwen3:8b
Each library tag pins an upstream Hugging Face revision; a manifest records the hash of every file.

Architectures

Transformer models across modalities

Sealing requires a validated implementation for each architecture; there is no on-the-fly sealing of unknown architectures. Plain serving runs any model the chosen engine supports.

  • Language models

    Qwen3, Qwen3-MoE, Qwen2 and Qwen2.5, and Llama 3.x, under orth or perm; FP8 block-wise or per-tensor checkpoints of these under perm.

  • Image diffusion

    Z-Image, sealed by its own scheme with a client-keyed mode: the text encoder is sealed and the DiT's latent interface uses a key-derived patch basis. In client-keyed mode the VAE decodes on the user's machine.

  • Video generation

    MiniMax H3 FL2VA, served with SGLang. perm is recommended; orth re-rounds the text encoder's rotated weights, to which the audio stream is most sensitive.

Model libraryRequest a modelMeasured results