Engine internals guide · October 14, 2026 · 32 min read

Inside SGLang

Ask an agent five questions about the same codebase and it sends the same ten thousand tokens of context five times. A plain serving engine computes those tokens five times. SGLang keeps them in a tree and computes them once. This guide follows one request through SGLang v0.5.22, the release tagged on October 13, 2026, and stops at each part that decides speed or memory.

The Scheduler box does the work that made SGLang known. On an NVIDIA L4, a shared 4,096-token prompt cut the first token from about 14.7 s to under 0.5 s. The radix tree already held that prefix. The steps below follow the line one box at a time.

1. 1Why SGLang exists
2. 2Follow one prompt through CPU and GPU
3. 3Look inside the forward pass
4. 4Grow a prefix tree, then run out of room
5. 5Find where your prompt breaks the match
6. 6Decide how much memory becomes KV cache
7. 7Extend the cache to host memory and storage
8. 8Admit requests on a forecast, retract when it fails
9. 9Keep the GPU busy between steps
10. 10Split long prompts into chunks
11. 11Constrain every token with a grammar
12. 12Guess several tokens, check them in one pass
13. 13Spread one model across many GPUs
14. 14Tune a server in a fixed order
15. 15When inference becomes part of training
16. ·When to choose SGLang

Eight figures let you drive the real rules, and small diagrams show the rest. Numbers are measured on a Mac, an NVIDIA L4, or a DGX Spark, or cited from the source. The full measurements and their setups are in the SGLang 101 manual.

Step 1: Why SGLang exists

A language model answers one token at a time. At every layer, each new token reads the stored key and value vectors of all earlier tokens. An engine keeps those keys and values in GPU memory so it does not compute them again. That store is the KV cache. It grows with every token of every live request, and after the model weights it takes the most memory.

In 2023, most engines threw the KV cache away when a request finished. That was a waste for the programs people were starting to write. A few-shot prompt repeats the same examples on every call. A chat repeats the whole history on every turn. An agent repeats its system prompt, its tool list, and its notes on every step. Tree-of-thought search branches from one shared stem.

The SGLang paper by Zheng and colleagues (arXiv 2312.07104) made one design choice the center of the system. Keep the KV cache of finished requests in a radix tree keyed by token sequence. A new request walks the tree, takes the longest prefix that is already there, and computes only the rest. The authors named this RadixAttention. They also wrote a small frontend language, with gen, select, and fork, so that programs with branches would produce shared prefixes on purpose.

That frontend is still in python/sglang/lang/ at v0.5.22, but most users now call the OpenAI-style HTTP API or the in-process Engine. The frontend is a client library. It talks to the runtime through the same /generate route as everyone else. The idea that lasted is in the runtime: cached prefixes and running requests share one memory pool, and the scheduler knows about the tree.

The project grew from that idea into a standard piece of AI infrastructure. It started in 2023 at LMSYS, the research group behind Vicuna and Chatbot Arena, and TechCrunch traces it to Ion Stoica's lab at UC Berkeley. In March 2025 it joined the PyTorch ecosystem. On October 16, 2026, the GitHub API reported 37,209 stars, 9,615 forks, and 2,074 contributor entries, including anonymous commits.

Two habits explain much of that growth. The first is day-0 support. When a large open model launches, the SGLang team often publishes a serving recipe on the same day, and DeepSeek releases are the long-running example. The second is published results at scale. In May 2025 the team reported serving DeepSeek on 96 H100 GPUs at 52.3k input tokens and 22.3k output tokens each second on each node. That is the team's own claim, with the setup published for anyone to repeat.

In May 2026 the core maintainers started a company, RadixArk, with a $100 million seed round led by Accel. The launch release says SGLang runs on hundreds of thousands of GPUs and produces trillions of tokens a day. It names Google, Microsoft, NVIDIA, Oracle, AMD, LinkedIn, xAI, and others as users. Those are company claims. LinkedIn has written about its own production use in detail.

Popularity does not explain speed, though. The rest of this guide does. It follows one prompt from the HTTP socket to the last streamed token. On the way it stops at each mechanism that makes SGLang fast. The order is the CPU and GPU split, the attention math, prefix reuse, the memory tiers, the scheduler, decoding, and scale-out.

Keep one question in mind for the rest of this guide. Every few milliseconds the scheduler asks which tokens the GPU computes next. Each step below is one part of the answer.

Step 2: Follow one prompt through CPU and GPU

Start the server and send it one prompt. On a Mac, SGLang runs through MLX. This is the exact command behind every Mac number in this guide, with SGLang 0.5.22, MLX 0.32.4, and the 4-bit Qwen3-0.6B model:

SGLANG_USE_MLX=1 sglang serve --model-path mlx-community/Qwen3-0.6B-4bit \
  --disable-cuda-graph --host 127.0.0.1 --port 30000 \
  --enable-cache-report --enable-metrics --mem-fraction-static 0.4

curl -s localhost:30000/generate -H 'content-type: application/json' \
  -d '{"text":"The capital of France is","sampling_params":{"max_new_tokens":12,"temperature":0}}'

The process tree that answers has three parts, and the reason is Python. Tokenizing and detokenizing text is CPU work. The scheduler must start a GPU step every few milliseconds. If all three shared one Python interpreter, they would wait on one global lock. So the Engine starts them as separate processes and connects them with ZMQ sockets.

The main process runs the HTTP server and the TokenizerManager. It turns the request into token ids, records a ReqState with an asyncio event, and pushes a TokenizedGenerateReqInput message to the scheduler. The HTTP handler then waits on that event.

The scheduler process owns the GPU. Under tensor parallelism each GPU has a scheduler process, and only rank 0 reads the socket. Rank 0 copies every request to its peers, so all ranks build an identical batch. The scheduler builds a Req, places it in the waiting queue, and later runs it in a batch through the ModelRunner.

Each step, the scheduler sends new token ids to the detokenizer process. The detokenizer decodes them with two offsets per request. It holds back text that ends in half of a multi-byte character, and it sends complete text back to the main process. There, the event fires and the HTTP response streams the new text.

Stop conditions live in the scheduler, not in the detokenizer. The scheduler checks stop strings before end-of-sequence tokens, because speculative decoding can accept several tokens in one step. A streamed response also delays text at the end of the output if it matches the first characters of a stop string. If you see a few characters arrive late, that is the reason.

What runs on the CPU, and what runs on the GPU

Split the work of one request into two kinds. The CPU parses the HTTP body, applies the chat template, and turns text into token ids. In the scheduler process, the CPU also walks the radix tree, builds the batch, allocates KV slots, and fills the grammar mask. The detokenizer process turns ids back into text on the CPU.

The GPU does the arithmetic. It looks up the embedding row of each token and runs every layer's matrix multiplies and attention. Then it projects the last hidden state to logits over the vocabulary and samples the next token. For Qwen3-0.6B that is 28 layers and a vocabulary of about 152,000 entries.

ZMQ moves each message between processes over a local IPC socket. By default the message is a pickled Python object, because SGLANG_USE_PICKLE_IPC is true at v0.5.22. The hop is cheap, but it is not free, and it happens for every request and every step of output.

The figure below replays five real requests to the Mac server. Each one carries the timestamps that --enable-metrics adds to meta_info. Pick a request, then step through the stages and watch which process holds it.

Three readings stand out. Tokenizing and the handoff to the scheduler took well under one millisecond each, even for a 200-token prompt. The first real prefill took about 800 ms, because MLX compiled its kernels on that request. The same prompt a second time took 20 ms, with 9 of its 10 tokens served from the tree.

On a big GPU, the arithmetic per step is much faster, so the CPU share of each step grows. LinkedIn's write-up found that its first bottlenecks were CPU tokenization, ZMQ message splitting, Python garbage collection, and the global interpreter lock. That is the reason SGLang keeps adding Rust pieces at the edges, such as the gRPC server and the radix tree core.

The native response from the command above carries more than text. Its meta_info reported prompt_tokens: 5, cached_tokens: 2, num_retractions: 0, a queue time of 0.24 ms, and a first-token latency of 889 ms. That first request paid for warm-up.

The fields matter more than the values. The cached_tokens field tells you how much of the prompt came from the tree. The num_retractions field tells you whether the scheduler removed this request in the middle of its answer. Steps 4 and 8 explain both.

Step 3: Look inside the forward pass

The GPU spends its time in the forward stage of the last figure. Most of that time goes to two operations: matrix multiplies against the weights, and attention. Attention is also where the KV cache enters the math. So it helps to see the matrices with real numbers.

Take Qwen3-0.6B. Its hidden size is 1,024. Each layer projects every token into queries for 16 heads, and into keys and values for 8 heads. Each head is 128 numbers wide. For a prompt of n tokens, one layer makes a query matrix of n by 2,048. The key and value matrices are n by 1,024 each.

Two query heads share each key and value head. This is grouped-query attention. It halves the KV cache compared with one key and value head per query head. That is why the model stores 114,688 bytes per token and not twice that.

For one head, attention takes four steps. Multiply the query matrix by the transposed key matrix to get a score for every pair of tokens. Scale the scores by 1 over the square root of 128, which is 0.0884. Set every score above the diagonal to minus infinity, so no token can look at a later token. Then take a softmax along each row and multiply the result by the value matrix.

The figure runs those steps on real numbers. I loaded the 4-bit Qwen3-0.6B weights with MLX and recorded the queries, keys, and values of every layer for one short sentence. The sentence is "The cat sat on the mat because it was tired".

Look at layer 6, head 1. The row for " it" puts 0.939 of its weight on " cat". That head resolves the pronoun. Layer 27, head 0, puts 0.759 of the same row on the first token, "The". Many heads put spare weight on the first token when no other token matters. People call this pattern an attention sink.

Prefill and decode have different shapes

During prefill, every prompt token is a query at the same time. The score matrix for one head is n by n, and the GPU fills it with large matrix multiplies. That work is compute-bound, because each weight and each key is reused across many queries.

During decode, there is one new query per request. The score matrix is 1 by n. The new token must still read the key and value of every earlier token, in every layer. For this model at 32,768 tokens of context, that is about 3.76 GB of KV per step per request. Decode is therefore memory-bound. Its speed depends on how fast the GPU reads memory, and batching many requests shares the weight reads across them.

Paged KV and the kernels that read it

The keys and values do not sit in one block per request. SGLang stores them in a pool of slots, one slot per token by default. A table maps each request position to a slot. SGLang calls this table req_to_token. The attention kernel reads the slot indices for a request and gathers its keys and values from wherever they are.

That indirection is what makes prefix sharing possible. When two requests share a prefix, the leading entries of both rows hold identical slot numbers. No copy happens.

The kernels come from attention libraries that SGLang calls through one backend interface. FlashInfer handles both shapes well. For prefill it has a ragged kernel for the new tokens, stored back to back, and a paged kernel for the cached prefix. For decode it has a paged kernel that reads one query against many scattered slots. At v0.5.22 the default on Hopper GPUs is FlashAttention 3, and on Blackwell GPUs it is the TensorRT-LLM attention kernels through FlashInfer.

Decode has one more cost that is not math. Each step launches hundreds of small kernels for a few tokens, and launching them from Python takes time. SGLang records those launches once per batch size as a CUDA graph and then replays the graph with one call. Decode graphs are on by default, for batch sizes up to a limit set by GPU memory. That limit matters again in Step 14.

The saving depends on how long one step takes. On a DGX Spark running the mixture-of-experts Qwen3-30B-A3B, time per token was about 33 ms with or without graphs. Each step there is dominated by reading the chosen experts from memory, so launch time is small next to it.

A dense model showed the same thing. On an NVIDIA L4 with Qwen3-8B, graphs saved under 1 percent per token, and the Triton attention backend tied FlashInfer. Each step reads about 16 GB of weights, so the weight reads set the pace.

Graphs still cost something at startup, about 40 s and 4 GB to capture on that machine. Keep them on, and measure with and without them before you blame the launch path.

Step 4: Grow a prefix tree, then run out of room

The tree is the RadixCache class in mem_cache/radix_cache.py, and at v0.5.22 the default controller is UnifiedRadixCache with a Rust core. The rules are the same in both. Each node holds a run of tokens and, as its value, the KV pool slots where those tokens live. Children are keyed by their first token, so a lookup walks one branch.

A lookup is match_prefix. It walks down while the tokens agree. If the match ends in the middle of a node, the node splits in two, so the tree records the exact boundary. The walk also updates the access time of each node it passes. The result is the matched length and the slot indices along the path.

One rule surprises people. The scheduler caps a match at the prompt length minus one. Even a fully cached prompt computes its last token, because the engine needs that token's logits to sample the first output. Asking for prompt log-probabilities from position k also caps reuse at k.

When a request finishes, its prompt and its output join the tree as new nodes. If another request already added the same prefix, the duplicate slots go back to the free list. The finished request's tokens are now cache, and the next turn of the same chat can start from them.

The tree and the running requests share one pool. There is no separate cache size to set. Cache is any KV that no running request holds right now.

A running request locks the last node it matched, and the lock covers its whole path to the root. When the pool is short, evict frees unlocked leaves in least-recently-used order. A parent that becomes a leaf is next in line. So eviction removes a branch from its tip toward the root, and a shared stem lasts longest.

Drive it below. The pool holds a support bot's 2,048-token system prompt, a translator's prompt, and the questions and answers of a few users. Send requests in any order. Then make the pool smaller, or pin user A's path as if their request were still running, and watch what eviction is allowed to take.

Two things in that figure carry over to real traffic. First, the system prompt is the node that survives, because every user's path runs through it and it is touched the most. Second, a pinned path cannot be evicted. When pins fill the pool, a new request waits in the queue even though the tree holds plenty of old answers.

A Mac shows how the tree works. A GPU shows the size of the effect. On an NVIDIA L4 with Qwen3-8B, the test used a shared 4,096-token system prompt. The median first token fell from about 14.7 s with the tree off to under 0.5 s with it on. Output rose more than four times.

With nothing shared, the tree cost almost nothing, so there is no reason to turn it off. Figure 8 at the end of this guide replays that sweep.

The size of the gain depends on the model. On a DGX Spark with the mixture-of-experts Qwen3-30B-A3B, the same test cut the first token about 9 times, not 30. That model activates about 3B parameters per token, so its prefill is cheap, and the tree has less repeated work to save.

The scheduler can also use the tree to order work. The default --schedule-policy is fcfs. With lpm, longest prefix match, the waiting queue is sorted by how much of each prompt the tree already holds. The paper proves that a depth-first visit of the tree gives the best hit rate for a fixed batch, and lpm follows that order. The code turns lpm back into first-come-first-served when more than 128 requests wait, because the sort costs CPU time on every pass.

There is also a small trick for siblings that arrive together. Two waiting requests can share a prefix of 32 tokens or more that is not yet cached. Then the scheduler holds the second one back for one round. The first one writes the shared part, and the second one then reads it from the tree.

Step 5: Find where your prompt breaks the match

The tree only helps if your prompts start the same way. One different token near the start makes every token after it a miss, because a radix match stops at the first difference. Timestamps, request ids, and shuffled tool lists are the usual causes. This is a client problem, and no server flag repairs it.

I measured the effect on the Mac. One script sent five rounds of a chat request with a long system prompt and a 16-token answer. In the first case the prompt stayed the same. In the second case a fresh request id sat at the very start of the system message. In the third case the same id moved to the end, just before the user question.

The stable prompt had 2,999 tokens. From round 2 on, the tree served 2,998 of them, and the median time to first token was 45 ms. With the id at the start, the tree served about 5 tokens of chat template header. The median time to first token was 1,255 ms. With the id at the end, the tree served about 2,985 tokens and the median was 88 ms.

The fix is an order, and it costs nothing. Place fixed content at the top: system prompt, tool definitions in a stable order, and long reference text. Put content that changes per request last: ids, dates, and the user turn. Some agent clients add a status block after the last system instruction. Each update to that block moves the break point into the middle of a long prompt.

Sharing also shows up at scale. A second benchmark sent 64 requests in 4 groups of 16, eight at a time. Each group had its own 2,048-token system prompt and a 128-token question. The server reported a cache hit rate of 0.94 and finished in 48.8 seconds. With --disable-radix-cache, the same run took 129.7 seconds, and the mean first-token latency rose from 1.3 seconds to 7.1 seconds.

The first three rows show the other half of the story. Output throughput stopped growing after 8 concurrent requests, and the time to first token grew five times between 8 and 32. Past that point, more concurrency only adds waiting. The last row is the subject of the next step.

Step 6: Decide how much memory becomes KV cache

Every number in the last two steps depends on the pool size, so look at where it comes from. Start with the cost of one token. Take a model with grouped-query attention. One token costs the KV heads per GPU, times the key plus value width, times the layers, times the bytes per value.

Qwen3-0.6B has 28 layers, 8 KV heads, and a head width of 128. In 16-bit precision that is 8 times 256 times 28 times 2, or 114,688 bytes per token. The MLX log printed the same number as bytes_per_slot=114688. Llama-3.1-8B costs 128 KiB per token by the same formula, so a 100,000-token conversation holds 12.5 GiB. Models with multi-head latent attention, such as DeepSeek-V3, store one compressed latent per token. That is about 68.6 KiB, but every tensor-parallel rank keeps a full copy.

The flag that sets the pool is --mem-fraction-static. Its docstring defines it as model weights plus KV pool, divided by device memory. It is not the share of memory the server uses.

Activations, CUDA graph buffers, and the grammar and sampling buffers live outside it. On CUDA, SGLang picks a default from a reserve estimate. The estimate is 512 MB, plus 1.5 MB per token of chunk size, plus room for parallelism and CUDA graphs. GPUs above 60 GB keep at least 10 GB in reserve.

The MLX path on a Mac sizes the pool with its own formula in _compute_pool_size. The budget is the smaller of two values. One is the Metal working-set limit times the fraction, minus memory MLX already holds. The other is the system memory available at startup times the fraction. The pool size in tokens is the budget over the per-token cost.

That second value is where a Mac run failed. The default fraction resolved to 0.872, and with other apps open the pool took 33.4 GB. At 32 concurrent requests Metal ran out of memory, and every SGLang process exited, not only the request that hit the limit. With --mem-fraction-static 0.4, every request finished.

The same rule applies on GPUs. The default assumes the device belongs to SGLang. On a shared machine, or next to another process on the same GPU, set the fraction yourself. Then read the startup log line, which prints max_total_num_tokens, chunked_prefill_size, max_running_requests, context_len, and available_gpu_mem. The tuning doc suggests 5 to 8 GB of free memory for activations on a GPU. Raise the fraction when you see more, and lower it when you see less.

On a machine with unified memory, the pool size also changes between starts of the same command. So read the startup line each time, and do not credit a flag for a pool difference from one start.

The KV dtype is the other lever. With --kv-cache-dtype fp8_e4m3, each token costs half as many bytes, so the same memory holds twice the tokens. It needs an attention backend that reads FP8 without a separate dequantize pass. It also needs scales from the checkpoint, because the help text warns that a missing scale defaults to 1.0 and can cost accuracy.

On a small GPU the dtype decides how many requests run at all. On the L4, Qwen3-8B in BF16 left room for 24,094 tokens of KV. With FP8 weights and an FP8 cache, the pool held 142,369 tokens, 5.9 times as many.

At 64 concurrent requests the BF16 pool was full, 26 requests waited, and output stopped at 251 tokens per second. FP8 ran all 64 and reached 736. FP8 was also faster per token at one request. Decode on an L4 reads every weight for each token, and FP8 weights are half the bytes.

FP8 matters most when memory is tight. We gave both builds the same smaller budget. The BF16 pool held 6,559 tokens and could not keep four 2,048-token prefixes. The FP8 pool held 107,299 tokens, kept every prefix, and gave 6 times the output.

On a DGX Spark with Qwen3-30B-A3B, FP8 freed 28 GB of weight memory for the cache. With half the bytes per token as well, the pool held about 4.8 times as many tokens.

A bigger pool helps only when the pool is the limit. Giving the DGX Spark pool 63 percent more room changed nothing at 64 users, because they used about 11 percent of it. In an overload, with hundreds of requests in flight, the same extra room let BF16 finish about a third more requests.

So size the KV pool from token usage at peak load, bursts included, not from free memory.

Step 7: Extend the cache to host memory and storage

GPU memory fills fast with long sessions. A coding agent that has run for an hour can hold a context of 100,000 tokens. For Qwen3-0.6B that is 11 GB of KV for one session, and for larger models it is many times more. Once the GPU pool has no free slots, LRU eviction drops old prefixes, and the next turn of that session must compute them again.

HiCache adds two more tiers under the GPU pool. Turn it on with --enable-hierarchical-cache. The second tier is pinned host memory, sized by --hicache-ratio, which defaults to 2.0 times the GPU pool. The third tier is optional storage, set by --hicache-storage-backend, with choices that include file, mooncake, hf3fs, and nixl.

The radix tree records where each node's KV lives. A node can be on the GPU, in host memory, or in both. The controller class is HybridCacheController, and the host pool for standard attention is MHATokenToKVPoolHost.

Writes go down the tiers under a write policy, set with --hicache-write-policy. Under write_through, the default, a node is copied to host memory when its hit count reaches 1. Under write_through_selective, the threshold is 2, so only reused nodes are copied. Under write_back, a node is copied only when it is evicted from the GPU.

Reads come back up. When a request matches a prefix that is only in host memory, the controller copies it back to fresh GPU slots. It does this layer by layer while the forward pass runs, so the copy for layer 2 overlaps the compute of layer 1. The default IO backend, kernel, uses GPU copy kernels for this.

Storage reads are prefetches, and they follow a policy set with --hicache-storage-prefetch-policy. Under best_effort, the prefetch stops as soon as prefill can run. Under wait_complete, prefill waits for all of it. Under timeout, the default, the prefetch stops after 1 second plus 0.25 seconds per 1,024 tokens. A storage hit shorter than 256 tokens is not prefetched at all, because recomputing it is cheaper.

The figure plays an agent workload through the three tiers. Six agents share one 2,048-token system prompt, and each turn adds 3,000 tokens of tool output and reply. Each node is real KV at 114,688 bytes per token. Change the tier sizes, the write policy, and the order in which agents speak.

Play the default, then make the GPU pool smaller. With host memory at ratio 0, every eviction means recompute. With ratio 2.0, most evicted prefixes come back from host memory, and the cost turns into bytes over the PCIe link. Switch to one agent at a time and the GPU alone holds almost everything, because each session finishes before the next one starts.

The write policy changes where the bytes go. write_through copies every new node down at once. Evictions then cost nothing, but the link carries every node. write_back copies only at eviction. It saves link traffic for nodes that stay on the GPU. write_through_selective sits between them.

The SGLang team's HiCache post claims up to 6 times higher throughput and up to 80% lower time to first token on multi-turn workloads. Those are team numbers. User reports also describe deadlocks and wrong answers from host-tier hits on hybrid models. Test HiCache on your own model before you rely on it.

We tested it on an NVIDIA L4 with Qwen3-8B. Groups of 8 users shared a 2,048-token system prompt, and the GPU pool held about 11 of those prompts. While every prompt fit on the GPU, HiCache changed nothing.

At 32 groups the prompts no longer fit. The GPU-only server evicted them and computed them again, while HiCache brought them back from host memory. The first token came 6.7 times sooner, and throughput was 2.7 times higher. At 64 groups the prompts outgrew GPU and host memory together, and HiCache only halved the wait.

On a DGX Spark, the same workload fit in the GPU pool at every size. Nothing was evicted, and the first token stayed near 450 ms. That machine also shares one memory between CPU and GPU, so a host tier would take memory from the same place as the pool.

The two machines measure one rule from both sides. Add the host tier when the distinct prefixes your users return to are larger than the GPU pool. When they fit on the GPU, the host tier adds a copy and no gain.

Step 8: Admit requests on a forecast, retract when it fails

When the scheduler admits a request, it does not know how long the answer will be. It only knows max_new_tokens. If it reserved the full maximum for every running request, the pool would sit mostly empty, because most answers stop early. If it reserved nothing, a long answer could fill the pool in the middle of a decode step.

SGLang takes a middle path and keeps a number called new_token_ratio. The PrefillAdder charges each running request its remaining max_new_tokens times that ratio. A request that asks to join must fit its whole input plus its full remaining max_new_tokens in what is left. The ratio starts at 0.7 and falls by a fixed amount on each decode step. After 600 steps it reaches 0.098, so the scheduler grows bolder while nothing goes wrong.

One flag turns the guess off. A request with ignore_eos always runs to its limit, so the scheduler charges it at a ratio of 1.0 and the reservation is exact. Benchmarks set ignore_eos by default. That is why the small-pool run in the table never retracted. Each request needed 256 input tokens plus 512 reserved tokens, so only 5 fit in a 4,096-token pool. The log showed #running-req: 5 for most of the run, the queue reached 15, and the median time to first token was almost 28 seconds.

When the guess is wrong, decode runs out of slots. Before each decode step, check_decode_mem checks that the pool has a free slot for the next token of each running request. If not, retract_decode sorts the running requests by output length, then by longer input, and removes requests from the end of that order. The requests with the least progress go first. Their KV goes back to the pool, and they return to the waiting queue to prefill again later.

After a retraction the ratio jumps. The new value is the decoded tokens plus 20 per remaining request, divided by the sum of their max_new_tokens, capped at 1.0. The log line reads KV cache pool is full. Retract requests. with the old and new ratio. A retracted prompt often still sits in the tree, so its second prefill is cheaper than the first.

Play both modes at the same pool size. With ignore_eos the queue is long and nothing is wasted. With natural stops and many long answers, more requests start early, then some of them lose their work. The forecast trades queue time for the risk of recompute.

The project's tuning doc says one retraction a minute is acceptable. Frequent retraction is a configuration problem. Raise --schedule-conservativeness, which scales the starting ratio, or lower --max-running-requests. One user report on the project tracker describes a retraction after which decode dropped from roughly 429 tokens per second to less than 40. It never recovered. Treat the retraction log line as a performance alarm.

Priority adds a second way to remove a running request. With --enable-priority-scheduling, a new request can push out running ones if its priority beats theirs by 10 or more, the default threshold. With --retraction-policy priority, retraction also removes low-priority requests first.

Step 9: Keep the GPU busy between steps

Each scheduler loop does CPU work: read new requests, sort the queue, match prefixes, allocate slots, and process the last step's tokens. If the GPU waits for all of that, it sits idle for part of every step. With a small model, the idle part can be a large share.

The overlap scheduler, event_loop_overlap, runs one batch ahead. The GPU starts on batch N+1 first. The CPU then handles the output of batch N during that forward pass. Batch N+1 is built before the tokens of batch N exist on the CPU. So its input ids are placeholders, and a FutureMap fills them on the GPU from batch N's output. Since late 2025 this uses two CUDA streams instead of a helper thread.

The SGLang team reported a 1.1 times throughput gain from this change in its v0.4 release post of December 2024. Its profiles showed no idle GPU gaps across five decode batches. Overlap is on by default. The code turns it off where it cannot work, such as the Torch MPS path without MLX and speculative decoding on CPU.

Turning it off is for debugging. On the MLX path the default loop is a separate overlap version built on mx.async_eval, and --disable-overlap-schedule did not work there at v0.5.22.

One side effect of running ahead is in flight tokens. A request that finishes on its length limit in batch N is still part of batch N+1 when that batch is planned. The v0.5.22 scheduler tracks those requests and leaves them out of the next decode batch. The same code avoids a retraction when finishing requests are about to free KV. That case is behind crash reports on earlier releases.

Step 10: Split long prompts into chunks

A prompt of 30,000 tokens is one large forward pass. While it runs, every other user waits for their next token. Chunked prefill splits the prompt and runs one chunk per step, so no single forward pass holds the GPU for the whole prompt.

The PrefillAdder handles the split. If a request's remaining input is larger than the chunk budget, only the first chunk runs now. The request keeps its place as the chunked request, and the next prefill step continues it before anything else. Its finished chunks go into the tree as they complete, so another request with the same prefix can reuse them.

The default --chunked-prefill-size depends on GPU memory. It is 2,048 under 20 GB, 4,096 up to 60 GB, 8,192 up to 160 GB, and 16,384 above. The Mac run resolved it to 4,096. A separate budget, --max-prefill-tokens, caps one prefill batch at 16,384 tokens by default.

By default a step is either prefill or decode, so decode still pauses during each chunk. The flag --enable-mixed-chunk puts the decode rows into the same forward pass as the chunk. In principle that smooths inter-token latency for other users, at the cost of a slower chunk. It is off by default, and it turns itself off when a request asks for log-probabilities.

In theory a smaller chunk protects running users, and a larger chunk finishes the long prompt sooner. In our runs a smaller chunk did not smooth the streams. When requests queue up, prefill goes first, so a stream waits whatever the chunk size. On an overloaded L4, --enable-mixed-chunk trimmed the worst token gap by only 5 to 8 percent. When a card cannot keep up with long prompts, no scheduler flag smooths the stream. You need more prefill capacity or fewer long prompts. Keep the chunk at or below the default for your GPU. On an L4, a chunk above the default ran out of memory at startup.

Below overload, the flag mattered more. With prompts of mixed lengths, it cut the worst token gap about in half and matched vLLM. With prompts that all had one length, it made the stream less smooth. On a DGX Spark that keeps up with prefill, it trades a smoother stream for a slower first token. Turn it on when your traffic mixes long and short prompts, then measure.

Step 11: Constrain every token with a grammar

Structured output in SGLang means the model cannot sample a token that breaks the format. A request asks for json_schema, regex, ebnf, or structural_tag. The default backend is xgrammar. Outlines and llguidance are the other choices.

Grammar compilation takes time, so it happens off the main path. The GrammarManager keys each grammar by its type and text and checks a cache. On a miss, the request waits in a grammar queue while a thread pool compiles it. Across tensor-parallel ranks, the scheduler admits a request only after all ranks finish compiling its grammar.

Each step, the grammar fills a bitmask over the whole vocabulary. Before sampling, ModelRunner applies that mask to the logits, so forbidden tokens have zero probability. After sampling, the grammar matcher accepts the token and moves its state. Under the overlap loop, the CPU builds the following mask during the forward pass, and sampling waits for it.

The compile cost shows once. On a DGX Spark, the first new schema after startup paid about 330 ms, and later new schemas paid no visible cost. Send each common schema once after a deploy, so a user does not pay it.

The idea behind fast structured output came early. In February 2024 the team described a compressed finite state machine. Some parts of a schema allow only one path, such as the text {"name":. There the engine can add the whole path in one step, not one token per step. The post claimed up to 2 times lower latency and 2.5 times higher throughput on JSON decoding.

The paper described a jump-forward mode that writes forced runs of text without a forward pass. The methods still exist on the grammar objects at v0.5.22, but nothing in the runtime calls them. Do not plan around jump-forward on this release.

Step 12: Guess several tokens, check them in one pass

Decode is slow because each step produces one token per request and reads all of the weights. Speculative decoding lets a small draft model guess a few tokens. The large target model then checks all of them in one forward pass. Every accepted guess is a step saved.

SGLang's main family is EAGLE, and DeepSeek's multi-token prediction runs as EAGLE with the draft layer inside the target checkpoint. One step in the v2 worker has three parts. The draft runs speculative_num_steps times and keeps topk candidates at each level, which grows a small tree. The target scores the best num_draft_tokens nodes of that tree in one pass, in a mode called TARGET_VERIFY. Last, the draft model runs over the accepted tokens to update its own cache and hidden state.

The gain is the acceptance length, the tokens committed per verify step, divided by the cost of drafting plus verifying. The defaults show the trade. Llama-family models get 5 steps, top-4, and 8 draft tokens. DeepSeek, GLM, and most others get 3 steps, top-1, and 4 draft tokens. When speculative decoding is on and you did not set --max-running-requests, SGLang sets it to 48, because the method wins at small batches.

Speculative decoding also uses KV slots for the draft tokens. One user report measured an agent replay with EAGLE on. The prefix hit rate fell from 97% to between 40% and 53%, and input throughput fell by more than three times. The draft reserve had pushed cached prefixes out of the pool. If your traffic depends on prefix reuse, measure speculative decoding and the radix cache together.

We measured both sides of the trade with EAGLE-3. On a DGX Spark with the mixture-of-experts Qwen3-30B-A3B, time per token was 1.6 times shorter at one request and 1.9 times shorter at 32. A mixture-of-experts step reads only the experts its tokens chose. The verify pass reuses those weights for the draft positions, so the gain grew with the batch.

The dense Qwen3-8B on a 24 GB L4 gave a different answer. Each token still came about twice as fast. But the draft model took memory, and the KV pool fell from about 24,000 to about 10,000 tokens. The throughput gain shrank from 2.4 times at one request to none at 32.

Speculative decoding spends memory that the batch needs. On a small card, measure it at the concurrency you serve.

Step 13: Spread one model across many GPUs

Everything so far ran in one world of GPUs. In SGLang, one model copy is a torch.distributed world of tensor parallel size times pipeline parallel size. Attention data parallelism, context parallelism, and expert parallelism are different ways to split that same set of GPUs for different layers. They do not add GPUs on top of it.

Tensor parallelism splits each layer's matrices across GPUs. Every rank runs the same scheduler loop on the same batch, which is why rank 0 broadcasts requests. For grouped-query models, more tensor-parallel ranks also split the KV heads, so each rank stores less per token.

Multi-head latent attention breaks that rule. DeepSeek-V3 has one latent per token, so tensor parallelism of 8 stores the same KV 8 times. Attention data parallelism fixes this. Each attention rank holds a separate set of requests with their KV, while the expert layers stay split. At v0.5.22 the flag is --attn-dp-size. The older --enable-dp-attention still works, is rewritten to the new flag, and is set to be removed after 2026.

Mixture-of-experts models add expert parallelism, where each GPU holds some experts and tokens travel to them by all-to-all. --moe-a2a-backend picks the transport, such as DeepEP. In its automatic mode, DeepEP uses a high-throughput mode for prefill and a low-latency mode, which works with CUDA graphs, for decode. EPLB moves hot experts between GPUs every 1,000 forward passes by default.

Prefill and decode want different hardware. Prefill is compute-bound on long inputs, and decode is memory-bound on many short steps. Prefill-decode disaggregation runs them on separate servers with --disaggregation-mode prefill and --disaggregation-mode decode.

The router chooses one server of each role and sends the request to both of them, with the same random 63-bit bootstrap_room id. First the decode server allocates KV slots and tells the prefill server where they are. The prefill server runs the forward pass and writes the KV straight into those slots with a one-sided transfer, by default through Mooncake. The decode server then starts from a prebuilt batch that skips prefill.

The transfer is also where a split can fail quietly. In one test over plain TCP, the KV transfer between prefill and decode broke. The router still returned HTTP 200 with empty text, and the benchmark counted every request as a success. A second run with the prefill and decode roles swapped failed the same way.

So check the generated text, not only the status codes, before you trust a disaggregated benchmark. The benchmark line Total generated tokens (retokenized) shows it.

Those pieces are what the 96-GPU DeepSeek deployment combined. Prefill and decode ran on separate nodes. Attention ran data-parallel to stop KV duplication for MLA. Experts were spread with DeepEP, with GEMMs from DeepGEMM and two-batch overlap to hide the all-to-all time. The team reported up to 5 times the output throughput of plain tensor parallelism on the same hardware. That is the team's claim from its own published setup.

Requests for all of these servers arrive at sgl-model-gateway, a separate Rust binary. Its default policy is cache_aware. It keeps an approximate radix tree over the raw request text, without a tokenizer. Each node records the workers that served that prefix.

When loads are balanced and more than 30% of a request's text matches a worker's prefixes, the request goes to that worker. Loads count as uneven when the longest queue is 64 requests longer than the shortest and 1.5 times as long. In that case the least loaded worker gets the request. The tree is a guess built from routing history, and it can name a worker that already evicted the prefix.

The tuning doc prefers several replicas behind the gateway to --dp-size inside one server. The in-process data-parallel controller uses round robin by default, and it does not know which replica holds which prefix.

Round robin also does not know how big each replica is. On two L4 cards, where one card also ran other work, the copy on that card had an eighth of the pool. Round robin still sent it half the requests, and the two copies served less than one card alone.

Tensor parallelism refused the same pair at startup, because every rank must hold the same share of memory. The flag --load-balance-method total_tokens sends each request to the copy that holds the fewest tokens. On that pair it lifted output from 42 to 115 tokens per second at 64 users, still below 251 on one card. The GPU also ran at a higher clock in that run, so part of the gain is the card. Keep replicas the same size.

Step 14: Tune a server in a fixed order

With the parts in place, the order of tuning follows from them. Start with a launch line you can read. This one is for one node with two GPUs and a shared-prefix workload, and every flag on it exists in the v0.5.22 help:

sglang serve --model-path Qwen/Qwen3-8B --tp-size 2 \
  --mem-fraction-static 0.85 --chunked-prefill-size 4096 \
  --max-running-requests 256 --cuda-graph-max-bs 256 \
  --schedule-policy lpm --enable-cache-report --enable-metrics

Then work through the gauges in this order. Each one depends on the one before it.

1. Read the startup line. Check max_total_num_tokens and available_gpu_mem, and move the fraction until 5 to 8 GB stay free.
2. Match the concurrency limits. The default max_running_requests comes from KV capacity with a floor of 2,048, while the largest decode CUDA graph comes from a GPU memory table. A user on an L40 measured 4,096 against 32, and decode fell to eager mode past 32 requests. Set both to your target.
3. Load the server and read the decode lines. The tuning doc aims for token usage above 0.9 with #queue-req between 100 and 2,000 for offline work. Low usage with a queue means the forecast is too careful. Frequent retraction means it is too bold.
4. Balance first-token latency against inter-token latency with the chunk size and --enable-mixed-chunk.
5. Fix prompt order for prefix reuse, then try lpm if many requests share prefixes and the queue stays under 128.
6. Scale out behind the gateway with cache_aware routing.

Measure every change with more than one run. A single benchmark run can reverse a result, so run several and read the spread.

The length mix of a benchmark matters as much. When every request has the same length, requests arrive in waves. Each first token waits for the prefill of the whole wave, and then the streams are smooth.

When lengths are mixed, requests arrive one at a time. The first token comes fast, and each new prefill makes the running streams stall for a moment. On an L4, fixed lengths gave a first token of about 2 s, and mixed lengths about 0.3 s with longer stalls.

Keep fixed lengths to compare builds. Test your real length mix before you set latency targets.

A few failures repeat often enough to name. A KV pool that fills during decode used to retract in a loop and hang. At v0.5.22, the last request that still cannot fit is aborted with HTTP 500 instead. Hybrid models with Mamba or sliding-window layers can cap concurrency below your setting and only say so in a log line. The /metrics endpoint, turned on with --enable-metrics, exposes sglang:cache_hit_rate, sglang:token_usage, and sglang:num_retracted_reqs, and those three catch most of these problems early.

On a Mac, use the MLX path with SGLANG_USE_MLX=1, set the fraction yourself, and keep the overlap loop on. The Torch MPS path without MLX turns overlap off and rejects quantized checkpoints. MLX runs the same Python scheduler, radix tree rules, and log lines as a GPU server. That makes a Mac a good place to learn the scheduler. It is not a place to measure GPU kernels, CUDA graphs, or attention backends.

Step 15: When inference becomes part of training

Reinforcement learning on language models needs rollouts, and rollouts are inference. Miles, an RL framework from RadixArk, runs real SGLang servers as its rollout engines and talks to them over HTTP. After each training step it pauses generation, sends new weights, and resumes. With trainer and server on the same GPUs, weights move as CUDA IPC handles through /update_weights_from_tensor. Otherwise they move by NCCL broadcast through /update_weights_from_distributed.

Read the version note before you copy a setup. Miles does not run on stock v0.5.22. It needs a patched SGLang from a sglang-miles branch on a v0.5.21 base, and some routes it calls do not exist upstream.

Two SGLang details matter more in RL than in serving. The log-probabilities SGLang returns for a sampled batch come from the temperature-scaled distribution, unless SGLANG_RETURN_ORIGINAL_LOGPROB=1 is set. Those values enter the policy ratio of the loss. The flag --enable-deterministic-inference swaps several matrix and reduction kernels for batch-invariant versions. Then a rollout and its replay agree.

Determinism needs that flag even in serving.

On a Mac, one greedy prompt sent twice gave two different answers. The first run computed all 10 prompt tokens. The second took 9 from the tree and computed 1, and its output split after three tokens. Different kernel shapes give slightly different float sums, and a near tie can flip.

GPUs show the same split. On a DGX Spark and on an L4, repeated greedy requests for one prompt wrote two to four different texts. The text depended on whether the prompt came from the tree and how the batch was formed. With --enable-deterministic-inference, every output matched.

The flag has a price. With the flashinfer backend it turns the radix cache off, so every repeated prefix pays its full prefill. In one test it also cost about 18 percent of throughput. Turn it on for RL rollouts and evaluations you need to repeat, and leave it off for production traffic.

When to choose SGLang

Why it is fast, in one place

Now the speed story can be told in order, one prompt at a time. Each mechanism removes a specific kind of waste, and each one appeared in a step above.

The process split removes waiting on Python's global lock. Tokenizing and detokenizing run in their own processes, so the scheduler can start a GPU step every few milliseconds. Rust pieces now take over the busiest CPU edges.

The radix tree removes repeated prefill. A prompt that shares a prefix with an earlier request computes only its new tokens. On the Mac, a repeated prompt needed 20 ms from scheduler entry to its first token. The first run needed 800 ms, partly for kernel warm-up.

The shared pool removes reserved but unused memory. Cached prefixes and running requests use the same slots, so there is no fixed cache size to tune. The admission forecast then fills the pool on a guess about answer length, and retraction corrects the bad guesses.

HiCache removes recompute for long sessions that do not fit in GPU memory. It turns that recompute into copies from host memory or storage, which cost bytes on a link and not GPU time.

The overlap loop removes idle GPU time between steps. The CPU prepares batch N+1 during the forward pass of batch N.

Chunked prefill removes the stall that one long prompt causes for everyone else. Decode for other users keeps going between chunks.

CUDA graphs remove kernel launch time in decode. The paged attention kernels read scattered KV slots without copying them into one block first.

The grammar backend removes invalid output without a retry. The CPU builds each mask during the forward pass, in parallel with the GPU.

Speculative decoding removes some of the memory reads of decode. One verify pass checks several draft tokens and reads the weights once for all of them.

At scale, attention data parallelism removes duplicate MLA caches, expert parallelism spreads the experts, and disaggregation gives prefill and decode separate hardware. The cache-aware gateway keeps each conversation on the worker that already holds its prefix.

The GPU runs add one rule to that list. Every large gain we measured came from a mechanism that removes repeated work, such as the prefill of a shared prompt. Every loss came from a feature that spends memory the batch needs, or from parts of a deployment that differ in size. None of the losses printed an error.

None of these is unique to SGLang alone today. The combination, the defaults, and the speed of day-0 releases are what users name when they explain their choice.

Turn on prefix reuse when your traffic shares prefixes. Agents, multi-turn chat, few-shot templates, and retrieval with a common instruction block all fit. The tree turns repeated context into cache hits. On the Mac, that meant a 45 ms first token instead of more than a second. A shared-prefix run took 38% of the time.

Prefix reuse alone does not pick SGLang, though. SGLang is often described as the engine for shared prefixes, but other engines cache prefixes too. Our own head-to-head on the L4, described below, found a tie when the prefixes fit the pool. Check that the prefix cache is on, then measure your own traffic.

Choose it for DeepSeek-class mixture-of-experts models across many GPUs. Attention data parallelism, expert parallelism with DeepEP, EPLB, and prefill-decode disaggregation are first-class features with flags, metrics, and docs.

Be more careful with unique prompts and a single dense model. There, field reports find small differences between engines, and the deciding factors are model support, startup time, and your hardware. Spheron reported 1,920 output tokens per second for SGLang 0.5.9 and 1,850 for vLLM 0.18.0 at 50 users. The test used Llama 3.3 70B FP8 on one H100 with unique prompts. The often repeated 29 percent gap between the two engines has a weak base. The test behind it paired SGLang v0.2.3 with a release of the other engine about two years newer, as DeepInfra pointed out. For the other popular engine, the vLLM guide on this site walks through its block pool and scheduler in the same depth.

We ran that comparison ourselves on an NVIDIA L4: SGLang 0.5.22 against vLLM 0.31.0, both serving Qwen3-8B with KV pools of the same size. With unique prompts, the two engines tied on throughput up to 32 users. With shared prompts and both prefix caches on, they also tied, within 2 percent at every prefix length. At 4,096 shared tokens, turning the caches off slowed the first token more than 30 times in both. The cache decides that latency, not the engine.

They differed when the pool was full. At 64 users vLLM kept admitting new prompts, so its first token came about 1.7 times sooner. Those new prefills shared steps with running decodes, and its worst gap between tokens was about 12 times longer than on SGLang.

At a fixed arrival rate the pair tied until the GPU fell behind. Then vLLM admitted more requests and gave about 11 percent more output, while SGLang kept each stream smoother.

So the shape of your traffic picks the winner. When requests arrive together, at a full pool or in waves of equal length, SGLang protected the stream and vLLM the first token. When prompts of mixed lengths arrive one by one, vLLM led on both throughput and smoothness. Turning on --enable-mixed-chunk closed that gap on SGLang. Measure your own length mix on your own model and GPU.

Whatever you choose, keep the habits from this guide. Put stable content first in every prompt. Read the startup line. Watch the retraction count. Set the memory fraction yourself on any machine you share.