Qwen · Text · MoE
Qwen3-MoE FP8
FP8 mixture-of-experts Qwen3; expert weights and router rows are permuted under the key.
Commands
Pull, seal and run
kllm pull downloads a library tag into the local store and records the hash of every file. kllm run starts SGLang or vLLM in the background on 127.0.0.1 and opens a chat. kllm create seals a pulled model under a named key.
# Download the library tag into the local store; a manifest records each file's hash
$ kllm pull qwen3-moe:30b-a3b-fp8
# Start SGLang or vLLM in the background on 127.0.0.1 and open a chat
$ kllm run qwen3-moe:30b-a3b-fp8
Requests to the running server
# The local proxy listens on 127.0.0.1:8001 and tokenizes on your machine
curl http://127.0.0.1:8001/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model": "Qwen3-30B-A3B-FP8-sealed", "messages": [{"role": "user", "content": "Hello"}]}'
Official sealed models
Pre-sealed under model keys
The kLLM team seals selected public models at a pinned revision under the perm transform, locks the weights and publishes them under ANetResearch on Hugging Face. Each model has its own model key, issued to approved applicants. The artifacts are rolling out; a repository is available once it appears in the organization.
| Repository | Base and revision | Modality | Licence | Transform | Client-keyed | Status |
|---|---|---|---|---|---|---|
| Qwen3-30B-A3B-Instruct-2507-FP8-kllm | Qwen/Qwen3-30B-A3B-Instruct-2507-FP8@5a5a776300 | Text (mixture of experts) | Apache-2.0 | perm | Yes | Rolling out |
- Each repository carries kllm-manifest.json: the base and its revision, the key fingerprint and key version, the kLLM version and the SHA-256 digest of every file.
Check a published model against its public base
With the model key, kllm verify seals the public base again under the same key and transform in a temporary directory and compares the result tensor by tensor, unlocked, together with the tokenizer, every other file and the manifest.
$ hf download ANetResearch/Qwen3-30B-A3B-Instruct-2507-FP8-kllm --local-dir Qwen3-30B-A3B-Instruct-2507-FP8-kllm
$ kllm verify Qwen3-30B-A3B-Instruct-2507-FP8-kllm --base Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --revision 5a5a776300a41aaa681dd7ff0106608ef2bc90db --key Qwen3-30B-A3B-Instruct-2507-FP8-kllm.key
Library tags
Tags
Each tag pins an upstream Hugging Face revision, so the same alias downloads the same bytes on every host. Sizes and minimum GPU memory are taken from the kLLM library.
| Alias | Upstream | Size | Min. GPU memory | Transforms | Copy |
|---|---|---|---|---|---|
| qwen3-moe:30b-a3b-fp8 | Qwen/Qwen3-30B-A3B-FP8 | 32 GB | 39 GB | perm |
Validated variants
Measured variants
Variants that the team has sealed and served, with their measured engines and hardware.
| Variant | Transforms | Engines | GPU | Sealing time | Input and output |
|---|---|---|---|---|---|
| Qwen3-30B-A3B-FP8 | perm only | vLLM 0.31.0, TP=2 (measured); SGLang not measured | A40 48 GB, TP=2 | 12 min on CPU | Text in, text out |
Measured results
Plain against sealed
Published measurements for this model, each with its conditions. Single runs; they do not generalize beyond the configurations shown.
Qwen3-30B-A3B-FP8, tensor parallel 2, plain against sealed
Two NVIDIA A40 48 GB GPUs, tensor parallelism 2; bf16; random token ids, 1024 input / 128 output tokens; concurrency 1 runs 10 requests, concurrency 16 runs 128; GPU shared with another tenant's light job; single runs, no variance reported. Perplexity over 14690 tokens of the GPL-3, GFDL-1.3 and Apache-2.0 licence texts (tools/ppl.py). The A40 has no FP8 tensor cores; vLLM runs FP8 through weight-only Marlin kernels. The README attributes the higher sealed throughput to noise on the shared host.
| Engine | Plain: tok/s · perplexity | perm: tok/s · perplexity · mean |Δ log-prob| |
|---|---|---|
| vLLM 0.31.0 | 359 · 1.3424 | 378 · 1.3428 · 0.033 |
Security
What sealing does not hide for this model
- To an operator who reads GPU memory, user text reduces to a substitution cipher over tokens: 23.8% of tokens were decoded by unigram frequency alone on Qwen3-8B.
- The sealed weights derive from a public checkpoint. Locked files cannot be matched to it without the key; once unlocked to run, a holder of that checkpoint recovers the transforms and 98.5% of the vocabulary permutation (measured on Qwen3-8B), for unmodified public models and fine-tunes alike.
- Client-keyed mode keeps the key and the tokenizer on the user's machine. The copy that the cloud serves is unlocked on the user's side (kllm cloud --key), because the cloud computes with the weights. The cloud server (kllm serve --client) binds 0.0.0.0 by default and has no authentication; bind it to 127.0.0.1 or place it behind a gateway.
- FP8 checkpoints are sealed with the perm transform only.
- Unless sealed with --no-lock, the weights on disk are locked: encrypted in place with AES-256-CTR under a key derived from the model key. kllm serve --key unlocks them into ~/.cache/kllm/unlocked (KLLM_UNLOCKED_DIR, mode 0700) before the engine starts, so whoever runs the model holds the unlocked weights. Each tensor's largest binary exponent is stored in the clear.
- Every approved holder of an official model's key holds the same model key, and the team derives it from an offline master key. A model key is not a confidentiality boundary between its holders; for a key known to no one else, seal the model under a private key.
- Outside a confidential VM, kllm serve --key decrypts the tokenizer into host /dev/shm and offers no protection from the host. It is a development mode.