Skip to content
Agent NetworkkLLM

Compare

Same hardware, close to the same speed

On the same GPUs as stock vLLM, SGLang or Ollama, kLLM adds a degree of privacy protection, and measured sealed throughput is close to plain serving on the same engine. kLLM runs SGLang and vLLM unchanged, so the comparison is a question of what the sealing layer adds and what it costs.

Install kLLMAll measured results

What it costs

Measured against plain serving on the same engine and GPU

Plain serving here is the unsealed checkpoint on the same stock engine and the same GPU. kLLM has not published a measurement against Ollama.

Served with --key, sealed throughput was never more than 0.3% below plain

Output throughput at concurrency 16, tok/s · plain, perm, orth · one NVIDIA A40 48 GB, bf16, 1024 input / 128 output tokens

Output throughput at concurrency 16. Qwen3-1.7B on vLLM 0.31.0: plain 1097, perm 1094, orth 1095 tok/s. Qwen3-8B on SGLang 0.5.21: plain 333, perm 333, orth 332 tok/s.QWEN3-1.7B · VLLM 0.31.0plainplain: 1,0971,097permperm: 1,0941,094orthorth: 1,0951,095QWEN3-8B · SGLANG 0.5.21plainplain: 333333permperm: 333333orthorth: 332332
Capsule bars · output throughput · benchmarks/llm.sh · single runs on a shared GPU, no variance reported

  • 0.3%

    Largest measured drop in sealed output throughput at concurrency 16, keyed serving.

  • 98.0 ms to 105.2 ms

    Median time to first token at concurrency 1, Qwen3-1.7B on SGLang 0.5.21 (perm).

  • 7%

    Throughput cost of client-keyed mode under vLLM at concurrency 16 (1036 to 963 tok/s).

Conditions: One NVIDIA A40 48 GB; SGLang 0.5.21 or vLLM 0.31.0; bf16; 1024 input and 128 output tokens; GPU shared with another tenant's light job; single runs, no variance reported.

What it adds

kLLM, stock engines and Ollama

The comparison is limited to the aspects listed. Descriptions of other projects summarize their public documentation and may change.

Comparison of kLLM, stock vLLM and SGLang, and Ollama
kLLMStock vLLM or SGLangOllama
Inference engineSGLang or vLLM, unmodified and not forkedThe engine itselfIts own runtime
Model library with name:tag aliasesYes, for example qwen3:8b; each tag pins an upstream revisionNo; serves Hugging Face repositories and local pathsYes
Pull and runkllm pull, kllm runOne serve command per modelollama pull, ollama run
OpenAI-compatible APIYes, through the engineYesYes
Batching, tensor parallelism, CUDA graphsThose of the engine, unchanged on sealed checkpointsYesNot compared
Weights sealed under a keyYes, for architectures with a validated implementationNoNo
Weights at restLocked by default: encrypted in place under the key, and unlocked on the machine that runs the modelAs distributedAs distributed
Pre-sealed modelsOfficial sealed models rolling out on Hugging Face, each under its own model keyNoNo
Check a sealed model against its public basekllm verify, with the model key: seals the base again and compares tensor by tensorNot applicableNot applicable
Prompts as received by the serving hostToken ids under the vocabulary permutation in client-keyed mode; not ciphertextPlain textPlain text
GPU memory of the serving hostNot protected; user text reduces to a substitution cipher over tokensNot protectedNot protected
PlatformsLinux with an NVIDIA GPU; macOS and Windows installers plannedPrimarily Linux with GPUsmacOS, Windows and Linux

Profiles, claims and measured limits

What it does not claim

Not a confidential GPU

kLLM does not claim the security of a confidential GPU. A confidential VM with an ordinary GPU does not protect GPU memory, and no part of kLLM is presented as equivalent to a confidential-GPU service.

Outside a confidential VM, kllm serve --key is a development mode with no protection from the host.

Roadmap

Disaggregated serving with confidential GPUs

Combined with confidential GPUs, transformed tensors can cross nodes without an extra encryption step. This is a design target for disaggregated serving, not a feature of the current release.

Planned

Planned for the local workflow

These items are design work and are not available in the current release.

Planned items
ItemScope
Resident daemon on 127.0.0.1:11400One endpoint that routes OpenAI requests by model, with keep-alive and admission control
Ollama-compatible API/api/chat and /api/generate
Local web console and Modelfile-style recipesBrowser management and declarative model recipes
Installers and packagespip install kllm with the first tagged release; installers for macOS and Windows; published container images
RoadmapkLLM Enterprise