Overview

SGLang is an open‑source framework that delivers inference for large language models and multimodal models with low latency and high throughput. It supports text, vision‑language, and diffusion models across a range of hardware.

Deploying SGLang is straightforward. Install by pip or docker, launch the server with a single command pointing to the model, and then query through standard OpenAI‑compatible API endpoints. The installation guide lists the required CUDA versions (12.9, 13.0) and provides commands for users with GPU, CPU, or other accelerator support. Once running, the server can be accessed locally or embedded in a larger service stack.

SGLang focuses on performance. It uses disaggregated prefill/decode, speculative decoding, and a zero‑overhead scheduler to make better use of GPU resources. The framework includes optimized GPU kernels and a Rust core for the prefix cache by default. On recent hardware, updates in the 0.5.22 release report up to 2.3× throughput gains on H200 GPUs and 1.9× on AMD MI350X/MI355X for specific models. Long‑prompt workloads achieve more than 27% throughput improvement thanks to prefill‑decode (PD) disaggregation, while KV cache can reside in CPU memory via HiSparse support for EAGLE and MTP.

Hardware support is diverse. SGLang runs on NVIDIA GPUs, AMD GPUs, Google TPUs, CPUs, Ascend NPUs, and XPU servers. It ships with a single engine that can handle multiple model families, including DeepSeek, Qwen, GPT‑OSS, Llama, Mistral, GLM, and newer diffusion models such as Kandinsky 6 and Gemini 2. Developers can mix and match models on the same cluster because the framework abstracts the underlying acceleration platform.

The API remains simple. After the server starts, you send requests to a /v1/chat/completions or /v1/responses endpoint that accepts the same format as OpenAI's API. Automation of batch requests is possible because SGLang can shift between prefill and decode without a restart. Optional response storage can be enabled with the --enable-response-store flag if you need to retrieve past responses by ID.