Amazon makes SGLang-Neuron publicly available for serving models on AWS Trainium

Amazon announced on 12.11.2026 that SGLang-Neuron is now publicly available through an AWS Deep Learning Container. The release comes from RadixArk in collaboration with Annapurna Labs. The plugin lets users serve models and generate reinforcement learning (RL) rollouts with SGLang on AWS Trainium while keeping the same SGLang interface they already use. Trainium is a portfolio of AWS AI chips that, according to AWS, delivers strong economics for AI training and inference at scale.

With SGLang-Neuron, running SGLang on Trainium follows the established SGLang workflow: models are launched through SGLang's server interface and the endpoint is invoked with the same generation APIs. Below that interface, SGLang-Neuron executes on native PyTorch. Because it is built as an out-of-tree plugin, Trainium support and the SGLang core can be developed independently and iterate more quickly.

How SGLang runs on Trainium

SGLang-Neuron connects SGLang with AWS Trainium as an out-of-tree plugin. Installing it registers the neuron platform through SGLang's plugin entry points, and setting SGLANG_PLATFORM=neuron selects that platform when the server starts. The SGLang core still supplies the API server, the request lifecycle, RadixAttention, and prefix-cache bookkeeping, while the plugin provides the Trainium-specific models, attention, mixture of experts (MoE), and the graph runner.

TorchNeuron registers Trainium as a native PyTorch device. The model execution path in SGLang-Neuron relies on standard PyTorch operations and adds kernels written with the Neuron Kernel Interface (NKI) for performance. These operations and kernels are compiled into executable programs with torch.compile, which the graph runner prepares and runs during serving.

The plugin also supports PD disaggregation. In that setup, a prefill server computes the prompt KV cache and sends it to a decode server using NIXL over Elastic Fabric Adapter (EFA). The decode server then produces the remaining tokens. Both servers continue to use SGLang's scheduler and request lifecycle, while the plugin handles the Trainium execution and the KV-cache transfer path.

Serving models on AWS Trainium with SGLang-Neuron

The walkthrough in the announcement uses a Trainium3 Amazon Elastic Compute Cloud (Amazon EC2) instance together with a pytorch-inference-sglang-neuronx container. The steps are:

1. Start the container.

docker run --rm -it \
--privileged --ipc=host \
-v /path/to/work:/work \
public.ecr.aws/neuron/pytorch-inference-sglang-neuron:v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04 \
/bin/bash

1. Launch the SGLang model server inside the container.

SGLANG_PLATFORM=neuron \
python -m sglang.launch_server \
--model-path openai/gpt-oss-120b \
--tp-size 8 \
--ep-size 8 \
--kv-cache-dtype fp8_e4m3 \
--context-length 10240 \
--chunked-prefill-size 8192 \
--max-running-requests 64 \
--max-total-tokens 655360 \
--mem-fraction-static 0.8 \
--host 127.0.0.1 \
--port 30000

1. Send a request to the model server.

curl -s http://127.0.0.1:30000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "openai/gpt-oss-120b",
"messages": [
{"role": "user", "content": "Explain why the sky appears blue."}
],
"temperature": 0,
"max_tokens": 128
}'

For more detail, AWS points readers to the SGLang-Neuron getting started guide for setup instructions and to the repository documentation for the list of supported models, features, and validated configurations.

Conclusion

SGLang-Neuron brings SGLang's familiar serving workflow to AWS Trainium on top of native PyTorch. Users can begin by launching the SGLang-Neuron container and following the quickstart to serve a supported model; the announcement also invites readers to try it with their own workloads, contribute through the project repository, and share feedback to help shape future model and feature support.