Qwen · Text · Dense
Qwen3 FP8
FP8 block-quantized Qwen3, sealed under perm: weights are bit-exact rearrangements with sign flips.
Commands
Pull, seal and run
kllm pull downloads a library tag into the local store and records the hash of every file. kllm run starts SGLang or vLLM in the background on 127.0.0.1 and opens a chat. kllm create seals a pulled model under a named key.
# Download the library tag into the local store; a manifest records each file's hash
$ kllm pull qwen3:8b-fp8
# Start SGLang or vLLM in the background on 127.0.0.1 and open a chat
$ kllm run qwen3:8b-fp8
Requests to the running server
# The local proxy listens on 127.0.0.1:8001 and tokenizes on your machine
curl http://127.0.0.1:8001/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model": "Qwen3-1.7B-FP8-sealed", "messages": [{"role": "user", "content": "Hello"}]}'
Library tags
Tags
Each tag pins an upstream Hugging Face revision, so the same alias downloads the same bytes on every host. Sizes and minimum GPU memory are taken from the kLLM library.
| Alias | Upstream | Size | Min. GPU memory | Transforms | Copy |
|---|---|---|---|---|---|
| qwen3:1.7b-fp8 | Qwen/Qwen3-1.7B-FP8 | 2.7 GB | 5 GB | perm | |
| qwen3:4b-fp8 | Qwen/Qwen3-4B-FP8 | 5.2 GB | 8 GB | perm | |
| qwen3:8b-fp8 | Qwen/Qwen3-8B-FP8 | 9.4 GB | 13 GB | perm | |
| qwen3:14b-fp8 | Qwen/Qwen3-14B-FP8 | 16 GB | 21 GB | perm | |
| qwen3:32b-fp8 | Qwen/Qwen3-32B-FP8 | 34 GB | 41 GB | perm |
Validated variants
Measured variants
Variants that the team has sealed and served, with their measured engines and hardware.
| Variant | Transforms | Engines | GPU | Sealing time | Input and output |
|---|---|---|---|---|---|
| Qwen3-1.7B-FP8 | perm only | SGLang 0.5.21, vLLM 0.31.0 (measured) | One A40 48 GB | 14–20 s (1–2B models) | Text in, text out |
Measured results
Plain against sealed
Published measurements for this model, each with its conditions. Single runs; they do not generalize beyond the configurations shown.
Qwen3-1.7B-FP8, plain against sealed
One NVIDIA A40 48 GB; bf16; random token ids, 1024 input / 128 output tokens; concurrency 1 runs 10 requests, concurrency 16 runs 128; GPU shared with another tenant's light job; single runs, no variance reported. Perplexity over 14690 tokens of the GPL-3, GFDL-1.3 and Apache-2.0 licence texts (tools/ppl.py). The A40 has no FP8 tensor cores, so both engines run FP8 through weight-only Marlin kernels.
| Engine | Plain: tok/s · perplexity | perm: tok/s · perplexity · mean |Δ log-prob| |
|---|---|---|
| SGLang 0.5.21 | 1055 · 5.7789 | 1055 · 5.7752 · 0.064 |
| vLLM 0.31.0 | 605 · 5.7770 | 628 · 5.7815 · 0.058 |
Security
What sealing does not hide for this model
- To an operator who reads GPU memory, user text reduces to a substitution cipher over tokens: 23.8% of tokens were decoded by unigram frequency alone on Qwen3-8B.
- The sealed weights derive from a public checkpoint. Locked files cannot be matched to it without the key; once unlocked to run, a holder of that checkpoint recovers the transforms and 98.5% of the vocabulary permutation (measured on Qwen3-8B), for unmodified public models and fine-tunes alike.
- Client-keyed mode keeps the key and the tokenizer on the user's machine. The copy that the cloud serves is unlocked on the user's side (kllm cloud --key), because the cloud computes with the weights. The cloud server (kllm serve --client) binds 0.0.0.0 by default and has no authentication; bind it to 127.0.0.1 or place it behind a gateway.
- FP8 checkpoints are sealed with the perm transform only.
- Unless sealed with --no-lock, the weights on disk are locked: encrypted in place with AES-256-CTR under a key derived from the model key. kllm serve --key unlocks them into ~/.cache/kllm/unlocked (KLLM_UNLOCKED_DIR, mode 0700) before the engine starts, so whoever runs the model holds the unlocked weights. Each tensor's largest binary exponent is stored in the clear.
- Outside a confidential VM, kllm serve --key decrypts the tokenizer into host /dev/shm and offers no protection from the host. It is a development mode.