Measured results

Benchmarks

Published measurements of sealed against plain serving, each with its hardware, engine version and conditions. They are single runs on shared A40 GPUs and do not generalize beyond the configurations shown.

Conditions for every result on this page unless stated otherwise

GPU: One NVIDIA A40 48 GB, shared with another tenant's light job
Engines: SGLang 0.5.21, vLLM 0.31.0
Precision: bf16
Request shape: 1024 input tokens, 128 output tokens
Repetitions: Single runs; no variance reported

Largest measured drop in sealed output throughput at concurrency 16, keyed serving: 0.3%; Largest measured drop in sealed output throughput at concurrency 16, keyed serving
Throughput cost of client-keyed mode under vLLM at concurrency 16: 7%; Throughput cost of client-keyed mode under vLLM at concurrency 16
MiniMax H3, 8 steps end to end, plain and sealed (perm) alike: 220.4 s; MiniMax H3, 8 steps end to end, plain and sealed (perm) alike

Throughput

Sealed throughput close to plain serving

In every measured keyed configuration, sealed output throughput at concurrency 16 was at most 0.3% below plain serving. Some rows were higher when sealed, which is attributed to noise on the shared host.

Served with --key, sealed throughput was never more than 0.3% below plain

Output throughput at concurrency 16, tok/s · plain, perm, orth · one NVIDIA A40 48 GB, bf16, 1024 input / 128 output tokens

Capsule bars · output throughput · benchmarks/llm.sh · single runs on a shared GPU, no variance reported

Output throughput at concurrency 16, tok/s
Model · engine	Plain	perm	orth
Qwen3-1.7BvLLM 0.31.0	1097	1094	1095
Qwen3-8BSGLang 0.5.21	333	333	332

Output throughput at concurrency 16, tokens per second.

Visible latency deltas

Two latency changes visible in the published tables.

Visible latency deltas, plain against sealed
Metric	Configuration	Plain	Sealed
Median TTFT, concurrency 1	Qwen3-1.7B, SGLang 0.5.21, perm	98.0 ms	105.2 ms
Median TPOT, concurrency 16	Qwen3-8B, SGLang 0.5.21, orth	35.3 ms	37.3 ms

Rows higher when sealed

Attributed to noise on the shared host, not to sealing. Output throughput at concurrency 16, tok/s.

Rows in which sealed throughput was higher than plain
Model	Configuration	Plain	Sealed
Llama-3.2-1B-Instruct	SGLang 0.5.21, perm; One A40	1630	1835
Qwen3-30B-A3B-FP8	vLLM 0.31.0, perm; Two A40, tensor parallelism 2	359	378

Client-keyed mode

Keeping the key on the user's machine cost 7% under vLLM

In client-keyed mode the GPU host serves token ids without the key or the tokenizer, and requests pass through kllm client proxy on the user's machine. Measured on Qwen3-1.7B sealed with perm.

Client-keyed serving reduced vLLM throughput by 7%

Output throughput at concurrency 16, tok/s · --key, --client · Qwen3-1.7B (perm), vLLM 0.31.0, A40, 1024 input / 128 output tokens

Capsule bars · output throughput · benchmarks/client.sh · single runs, no variance reported

Quality and exactness

Accuracy within one standard error

Qwen3-1.7B under vLLM 0.31.0, full test sets with lm-evaluation-harness. All differences are within one standard error.

Accuracy of Qwen3-1.7B, plain, perm and orth
Benchmark	Plain	perm	orth
MMLU	0.5560	0.5562	0.5548
GSM8K, strict	0.6945	0.6952	0.6960

Exact in exact arithmetic. Float64 tests on tiny random models agree to 1e-9.
Bit-exact under perm. Sealed weights are bit-exact rearrangements of the original values.
Accumulation order only in bf16. The mean absolute change in log-probability against plain serving was 0.03 to 0.10, comparable to switching between fp16 and bf16 (0.053).

Video generation

MiniMax H3: sealed as fast as plain end to end

MiniMax H3 end-to-end time, plain against sealed (perm)
Denoising steps	Plain	Sealed, perm
8	220.4 s	220.4 s

One A40 48 GB per run with layer-wise offload, SGLang 0.5.21.

Optional INT8 DiT kernels

Separate from sealing

kllm serve --int8 reduced seconds per denoising step on MiniMax H3 from 24.56 to 18.38 (-25%) on one A40. This is a kernel optimization separate from sealing, with measured numerical drift. Only the attention part is verified to commute with perm; the linear part is not, and weights are re-quantized on every call. It is not a sealing speed-up.

Sealing

Sealing time

Sealing is a one-time step per model and key. The first table predates weight locking; the catalog builds below include it.

Sealing time, measured before weight locking was added
Model	Hardware	Time
1–2B language models	Device not stated	14–20 s
Qwen3-30B-A3B-FP8	CPU	12 min
MiniMax H3 FL2VA (135 GB)	One A40	About 15 min

Catalog builds: seal, lock and verify

Building official sealed models on one A40 server
Model	Size	Seal and lock	kllm verify
Qwen3-1.7B	4.1 GB	36 s	41 s, 311 tensors equal to the bit
Z-Image (client-keyed)	20.5 GB	102 s	136 s, 1163 tensors equal to the bit

One A40 server, with the base downloaded through a Hugging Face mirror; download time is excluded. Sealing and locking ran at about 0.1 to 0.2 GB/s. Unlocking ran at about 0.7 GB/s on a GPU and 0.1 GB/s on CPUs, once per key and file set. Source: tools/catalog/README.md.

Method

How to read these results

Each figure is a single run on a GPU shared with another tenant's light job. No variance is reported, so small differences in either direction are not evidence of a speed-up or a slowdown. The results apply to the models, engines and hardware listed and do not generalize to other configurations.

The measurement scripts are in the repository under benchmarks/. Per-model tables, including perplexity, are on each model page.

Per-model results Same-hardware comparison Measurement scripts