LMSYS has added an experimental Unified Memory allocator to SGLang for hybrid model architectures including GPT-OSS, Qwen3.5 and Kimi K3. These models mix full-attention layers, sliding-window attention, and recurrent or linear-attention layers, each of which consumes GPU high-bandwidth memory (HBM) in a different way. Previously, SGLang required operators to divide HBM among those state types before launch; the new allocator lets them share a single arena and reshapes the layout as traffic shifts, reducing idle capacity and avoidable evictions.
Why fixed pools waste HBM
Autoregressive decoding stores key and value (KV) tensors from earlier tokens so the model can compute the next one. Full-attention layers keep KV for the entire context, so their memory footprint grows with sequence length. Sliding-window layers retain only a bounded suffix, while recurrent and linear-attention layers hold a fixed-size state for each active request, which makes their consumption depend mainly on concurrency.
SGLang's earlier allocator carved HBM into separate pools configured through --swa-full-tokens-ratio and --mamba-full-memory-ratio. Long-context traffic could exhaust the full-attention pool while window KV and recurrent-state capacity sat idle, and short, highly concurrent traffic could fill the bounded-state pools while full-attention space went unused. Free memory could not cross pool boundaries.
One arena, mixed layouts
With --enable-unified-memory, full-attention KV is placed at the high end of a single arena and grows downward. Sliding-window KV and recurrent state begin at the low end and grow upward, while lazy compaction maintains a free region between the two. The ratio flags still exist, but only as initial placement hints; they no longer create fixed capacity partitions after startup.
The allocator represents one token, or one recurrent-state slot, as an entry holding rows for every participating layer. A page is a contiguous run of entries, so allocation, eviction and page movement act on all layers together. Rows inside an entry may keep different shapes and data types. That arrangement lets speculative draft-model KV share storage and lifecycle rules with target-model KV even when their data types differ, and it keeps Kimi K3.1's Dynamic Sparse Attention index keys beside the Multi-head Latent Attention KV they point to.
Admission becomes a packing problem
An incoming request can need full-attention pages, sliding-window pages and a recurrent-state slot at once. SGLang evaluates the available layout in three steps of rising cost:
- Reuse in place: fill existing holes and extend either end band without moving data.
- Compact the ends: pack allocations within the high and low bands to remove internal gaps.
- Relayout: move eligible allocations into the space between the compacted end bands.
Virtual page IDs keep logical references stable while SGLang copies a page and remaps its physical location. A page-use ledger tracks model forward passes, HiCache copies and network transfers that still reference each page. When a request releases a page, SGLang marks it logically free and withholds its physical address from new allocations until the ledger shows no outstanding users.
Where the H200 gained headroom
LMSYS compared unified memory against SGLang's default separate-pool configuration on a single NVIDIA H200. The workloads tested showed near-parity under moderate pressure and higher request capacity whenever a fixed pool became the bottleneck.
Single-H200 benchmark results reported by LMSYS:
| Model and workload | Separate pools | Unified memory |
|---|---|---|
| Qwen3.5-4B, 1,024 input tokens, 256 output tokens, concurrency 64 | 5,880 output tokens/s | 5,830 output tokens/s with identical GSM8K accuracy |
| GPT-OSS-20B under maximum concurrency | 618 simultaneous requests; 43 s time to first token | About 1,000 simultaneous requests; 6 s time to first token; 14% higher throughput |
| Qwen3.5-4B, 16K input tokens, 4,096 output tokens, concurrency 64 | 22 active requests | 32 active requests and 26% higher throughput |
| Qwen3.5-4B, 512 input tokens, 16,384 output tokens, concurrency 51 | About 294 retractions per 1,000 completed requests | About 233 retractions per 1,000 completed requests |
A separate GPT-OSS-20B sensitivity test illustrated the cost of a bad static split: setting --swa-full-tokens-ratio to 0.1 cut throughput by 12% relative to the default ratio.
A retraction happens when memory pressure forces the scheduler to pull an in-flight request out of the active batch. Fewer retractions mean less scheduling churn and less repeated work during long generations.
These numbers cover one H200 and the listed workloads only; multi-GPU deployments and production traffic still require workload-specific validation.
Support across the serving stack
SGLang integrations available with unified memory:
| Area | Supported paths |
|---|---|
| Scheduling and caching | Overlap scheduling, Radix cache, and HiCache |
| Distributed execution | Prefill/decode disaggregation, pipeline parallelism, decode context parallelism, and data-parallel attention |
| Speculative decoding | EAGLE, EAGLE3, DFLASH, and DSPARK; compatible fused drafts can share entry storage with target-model KV |
SGLang's usage guide covers configuration, and the design guide provides the backend compatibility matrix and allocator details.
Current limits shape deployment
Unified memory is still experimental and targets hybrid models.
- Unsupported: models with only full-attention layers, MLA paired with sliding-window layers, and DeepSeek-V4.
- Prefill/decode disaggregation requires Mooncake and cannot run alongside pipeline parallelism, speculative decoding or decode context parallelism.
- HiCache cannot run alongside EAGLE or EAGLE3.
- Priority preemption is disabled while unified memory is active.
Replay the traffic mix before rollout
Production evaluation should start in staging with --enable-unified-memory and traces that cover long-context, short-request and high-concurrency traffic. Teams can compare output throughput, time to first token, maximum active requests and retractions against their current split-pool configuration, then check every enabled backend against SGLang's compatibility matrix.