vLLM vs SGLang vs TensorRT-LLM: How to Choose and Deploy an Inference Engine

The vLLM vs SGLang question, with TensorRT-LLM beside it, comes up at one point of inference work: you have an open model (one whose files you can download and run) and no API vendor, so you must serve it to real users yourself.

This guide is the serving step of inference optimization. Before it comes how a model writes an answer (our explainer). After it comes tuning how the server groups requests and, later, compressing the model.

Picture one team with a chat feature on an open model, run on one GPU server. What happens when N users type at the same time?

What you need, and which tool does what

• A Linux server with a GPU. TensorRT-LLM needs NVIDIA; vLLM and SGLang also list AMD, Intel and others.

• An open model, and one engine that loads it and starts the server.

• Your existing code. vLLM’s docs call its OpenAI-compatible server a drop-in replacement for applications that use the OpenAI API.

The queue you cannot see

A server answers only so many requests at once. The rest wait in a queue. One tester never fills it, so the demo looks fast.

N users do fill it. The last user waits for everyone ahead before seeing the first word. The model is not broken. The limit on requests at once is a setting, and nobody chose it for this load.

Step 1: Write down your workload

Five facts decide it:

• Peak concurrency: how many users wait for an answer at once (the N of our team).

• Wait budget: how long a user may wait for the first word (time to first token, TTFT) and between words (time per output token).

• Prompt and answer length, in tokens (short pieces of text, often a word or part of one).

• Shared starts: whether many requests begin with the same text, such as a system prompt or the chat so far.

• Hardware: which GPU and how much memory (NVIDIA or not).

A live chat needs short waits. An overnight batch needs throughput (tokens written per second, summed over all users) and ignores waits.

Step 2: Match the engine to your hardware and traffic

Read the table from your Step 1 answers. The last column is our rule of thumb; no benchmark backs it.

vLLM and TensorRT-LLM reuse a shared start too, so it is a reason to test SGLang and no reason to skip the others. On NVIDIA GPUs all three run; our rule is to start with the one your team already runs.

Which is fastest? We found no current three-way comparison from a neutral source. SGLang’s and vLLM’s own teams published some in 2024, on older versions; NVIDIA’s table covers TensorRT-LLM only. Run your own in Step 4.

If you only wanted to choose an engine, you can stop here. The rest is for the person who types the commands.

Step 3: Start the server with deliberate first settings

Install the latest release. On 10 November 2026 that means vLLM 0.32 or newer, SGLang 0.5.23, and TensorRT-LLM 1.2.1. Defaults change between releases, so run the engine’s --help on your version.

# vLLM
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct

# SGLang
uv pip install --prerelease=allow sglang
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000

# TensorRT-LLM (installed as NVIDIA's quick-start guide describes)
trtllm-serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

Commands from each project’s quickstart. A name like Qwen/Qwen2.5-1.5B-Instruct points to a model on Hugging Face, a public site that hosts open models; use your own. SGLang’s 0.0.0.0 listens on every network interface: use 127.0.0.1 unless the server sits on a private network.

vLLM and TensorRT-LLM listen on port 8000. Older TensorRT-LLM tutorials start with trtllm-build, a step that converted the model first. Since release 1.2 it is gone: trtllm-serve loads the model directly.

Set these deliberately (flags as each project’s docs name them):

The memory flags decide how much room is left for the KV cache, where the engine keeps earlier tokens (our KV cache explainer), after the weights (the model’s stored numbers) take their share.

We found no formula for the cache size in the docs, so read your model’s card and watch for out-of-memory errors in Step 4. SGLang counts weights and cache together and says to lower its value after such an error.

Start the requests-at-once flag at your Step 1 peak and let Step 4 correct it. The vLLM docs call the default for --max-num-seqs “mainly for convenience when testing”. Settings for continuous batching, running many requests together, go deeper in our continuous batching guide.

vLLM’s --performance-mode takes balanced (the default), interactivity (low latency) or throughput (more tokens per second at high concurrency). Keep the one your Step 4 test favours.

Quantization stores the model’s numbers at lower precision, so they take less memory.

FP8 is one such format: each number takes 1 byte instead of 2.

In vLLM, FP8 on weights and calculations is supported on NVIDIA Ada, Hopper and Blackwell GPUs and on GPUs such as the AMD MI300x; Turing and Ampere cards can use it for the weights only. vLLM’s docs report 2 times less model memory. Check accuracy on your own prompts.

Keep the server off the open internet. For vLLM, --api-key guards only some paths, and vLLM’s security docs say not to rely on it alone.

Update first. vLLM 0.8.3 to 0.14.0 has a critical flaw in video handling that lets an attacker run code on your server (CVE-2026-22778), and newer advisories exist, so pin a recent release. SGLang 0.5.9 and older have two like it (CVE-2026-3059, CVE-2026-3060), fixed in 0.5.10. We did not check TensorRT-LLM.

Step 4: Load-test it before your users do

Each project ships a client: vllm bench serve, python -m sglang.benchmark.serving and trtllm-bench. To compare engines, use one client for all, because metric names differ between tools.

vllm bench serve --backend vllm --model <model> --endpoint /v1/completions \
  --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 200 --max-concurrency <N> --metric-percentiles 95

<model> is the model name you gave vllm serve. --max-concurrency is the number of requests in flight at once; one request stands in for one waiting user. Without it, every prompt is sent at time 0. ShareGPT is a public set of chat conversations; the file name in the command is the one the vLLM benchmarking docs use.

The command prints throughput, TTFT and time per output token, each with a percentile. --metric-percentiles 95 asks for the 95th (p95): the value that 95 of 100 requests stay below.

Follow one procedure (it is ours; the cache warning in step 1 comes from the vLLM benchmarking docs):

1. Use different prompts that look like your traffic, and restart the server before every run. A repeated prompt or a warm cache inflates throughput (vLLM warns of it). In our test, a repeated prompt’s first-word wait fell to one twentieth.

2. Run three levels: 1 user, N users (your Step 1 peak) and 2N.

3. Repeat all three levels 3 times and compare numbers from the same round.

4. At each level read p95 TTFT, total tokens per second and errors.

5. Pass means p95 TTFT at N within your wait budget, with no errors. Otherwise change one Step 3 setting and rerun; keep it only if the numbers improve.

What our test showed

On 10 November 2026 we ran the queue in miniature. We had no NVIDIA GPU and none of the three engines installed, so we used llama.cpp’s llama-server on an Apple M3 laptop with 24 GB of memory. The model was Qwen2.5-1.5B-Instruct.

A slot is one place in that server where one request runs; --parallel N sets N slots. We used 1, 4 and 8 slots, and 1, 4, 8 and 16 simulated users who each send 3 requests in a row.

We ran 3 rounds: 783 counted requests, 0 errors. Every request had a unique start and the prompt cache was off. The laptop was busy with other jobs, so we show ratios and counts only.

The numbers come from a small model on a laptop (one prompt of about 500 tokens, 100-token answers), so they do not transfer to a GPU server. Look for the pattern in your own test: a queue above the cap, a server-reported reading time that stays flat while users wait, a cache that flatters a repeated prompt.

Slots are llama.cpp’s own setting; we did not test whether vLLM’s --max-num-seqs behaves alike.

Mistakes to avoid

Write down every setting you change

Every flag you leave at its default is a decision the engine made for you. For each flag you change, write four lines: what changed, which number should move, how you check it, how you undo it.

A prompt for your coding agent

Fill the brackets and paste this into your AI coding agent. It turns Steps 1 to 4 into a spec.

I need a deployment spec for an open-model chat server.

Model: [name and size]
Engine: [vLLM | SGLang | TensorRT-LLM]
Hardware: [GPU type, count, memory]
Workload: [live chat | batch | mixed]
Peak concurrent users: [number]
Wait budget: [p95 time to first token]
Prompt and answer length: [tokens]

Generate:
1. The start command with every setting from Step 3 set explicitly (GPU split, memory share, longest prompt plus answer, requests at once). Use only flags that appear in the output of the engine's --help for my installed version; if unsure, tell me to check.
2. A load-test plan with the engine's own benchmark client: 1 user, my peak and twice my peak, 3 repeats, the server restarted before every run, varied prompts, reporting p95 time to first token, tokens per second and errors.
3. A change log: for each setting, what changed, which number should move, how to check, how to undo.
4. Network rules: private network only, version pinned to a current release.