SGLang Native APIs

Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs:

/generate (text generation model)
/get_model_info
/server_info
/health
/health_generate
/flush_cache
/update_weights
/encode(embedding model)
/v1/rerank(cross encoder rerank model)
/v1/score(decoder-only scoring)
/v1/decisions(typed decisions on a chat model)
/classify(reward model)
/start_expert_distribution_record
/stop_expert_distribution_record
/dump_expert_distribution_record
/tokenize
/detokenize
A full list of these APIs can be found at http_server.py

We mainly use requests to test these APIs in the following examples. You can also use curl.

Launch A Server

Example

from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

server_process, port = launch_server_cmd(
    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
)

wait_for_server(f"http://localhost:{port}", process=server_process)

Generate (text generation model)

Generate completions. This is similar to the /v1/completions in OpenAI API. Detailed parameters can be found in the sampling parameters.

Example

import requests

url = f"http://localhost:{port}/generate"
data = {"text": "What is the capital of France?"}

response = requests.post(url, json=data)
print_highlight(response.json())

Get Model Info

Get the information of the model.

model_path: The path/name of the model.
is_generation: Whether the model is used as generation model or embedding model.
tokenizer_path: The path/name of the tokenizer.
preferred_sampling_params: The default sampling params specified via --preferred-sampling-params. None is returned in this example as we did not explicitly configure it in server args.
weight_version: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters.
has_image_understanding: Whether the model has image-understanding capability.
has_audio_understanding: Whether the model has audio-understanding capability.
model_type: The model type from the HuggingFace config (e.g., “qwen2”, “llama”).
architectures: The model architectures from the HuggingFace config (e.g., [“Qwen2ForCausalLM”]).
embedding: The resolved embedding-serving plan. It includes pooling, normalization, execution and attention style, Matryoshka dimensions, cache policy, and effective BCG prefill settings. This field is available when the model configuration exposes an embedding capability contract.

Example

url = f"http://localhost:{port}/get_model_info"

response = requests.get(url)
response_json = response.json()
print_highlight(response_json)
assert response_json["model_path"] == "qwen/qwen2.5-0.5b-instruct"
assert response_json["is_generation"] is True
assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct"
assert response_json["preferred_sampling_params"] is None
assert response_json.keys() == {
    "model_path",
    "is_generation",
    "tokenizer_path",
    "preferred_sampling_params",
    "weight_version",
    "has_image_understanding",
    "has_audio_understanding",
    "model_type",
    "architectures",
    "embedding",
}

Get Server Info

Gets the server information including CLI arguments, token limits, and memory pool sizes.

Note: get_server_info merges the following deprecated endpoints:
- get_server_args
- get_memory_pool_size
- get_max_total_num_tokens

Example

url = f"http://localhost:{port}/server_info"

response = requests.get(url)
print_highlight(response.text)

Health Check

/health: Check the health of the server.
/health_generate: Check the health of the server by generating one token.

Example

url = f"http://localhost:{port}/health_generate"

response = requests.get(url)
print_highlight(response.text)

Example

url = f"http://localhost:{port}/health"

response = requests.get(url)
print_highlight(response.text)

Flush Cache

Flush the radix cache. It will be automatically triggered when the model weights are updated by the /update_weights API. Parameters:

timeout (query, float, default 0, unit: seconds): Wait time for idle state before flushing. 0 means fail fast if not idle. When HiCache async operations are in-flight, a non-zero timeout allows the server to wait until idle before flushing, avoiding unnecessary 400 errors.

Command

# With timeout (wait up to 30s for idle state)
curl -s -X POST "http://127.0.0.1:30000/flush_cache?timeout=30"

Example

url = f"http://localhost:{port}/flush_cache"

response = requests.post(url)
print_highlight(response.text)

Update Weights From Disk

Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size. SGLang support update_weights_from_disk API for continuous evaluation during training (save checkpoint to disk and update weights from disk).

Example

# successful update with same architecture and size

url = f"http://localhost:{port}/update_weights_from_disk"
data = {"model_path": "qwen/qwen2.5-0.5b-instruct"}

response = requests.post(url, json=data)
print_highlight(response.text)
assert response.json()["success"] is True
assert response.json()["message"] == "Succeeded to update model weights."

Example

# failed update with different parameter size or wrong name

url = f"http://localhost:{port}/update_weights_from_disk"
data = {"model_path": "qwen/qwen2.5-0.5b-instruct-wrong"}

response = requests.post(url, json=data)
response_json = response.json()
print_highlight(response_json)
assert response_json["success"] is False
assert response_json["message"] == (
    "Failed to get weights iterator: "
    "qwen/qwen2.5-0.5b-instruct-wrong"
    " (repository not found)."
)

Example

terminate_process(server_process)

Encode (embedding model)

Encode text into embeddings. Note that this API is only available for embedding models and will raise an error for generation models. Therefore, we launch a new server to server an embedding model.

Example

embedding_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
    --host 0.0.0.0 --is-embedding --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=embedding_process)

Example

# successful encode for embedding model

url = f"http://localhost:{port}/encode"
data = {"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "text": "Once upon a time"}

response = requests.post(url, json=data)
response_json = response.json()
print_highlight(f"Text embedding (first 10): {response_json['embedding'][:10]}")

Example

terminate_process(embedding_process)

v1/rerank (cross encoder rerank model)

Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like BAAI/bge-reranker-v2-m3 with attention-backend triton and torch_native.

Example

reranker_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \
    --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=reranker_process)

Example

# compute rerank scores for query and documents

url = f"http://localhost:{port}/v1/rerank"
data = {
    "model": "BAAI/bge-reranker-v2-m3",
    "query": "what is panda?",
    "documents": [
        "hi",
        "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.",
    ],
}

response = requests.post(url, json=data)
response_json = response.json()
for item in response_json:
    print_highlight(f"Score: {item['score']:.2f} - Document: '{item['document']}'")

Example

terminate_process(reranker_process)

v1/score (decoder-only scoring)

Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities. Parameters:

query: Query text
items: Item text(s) to score
label_token_ids: Token IDs to compute probabilities for
apply_softmax: Whether to apply softmax to get normalized probabilities (default: False)
item_first: Whether items come first in concatenation order (default: False)
model: Model name

query and items also accept token ids, with query set to [] when each item is a complete prompt. label_token_ids can be one list per item. temperature divides the label logits before the softmax and needs apply_softmax, and return_token_logprobs adds token_logprobs, the full-vocabulary log-probabilities of the labels. The response contains scores - a list of probability lists, one per item, each in the order of label_token_ids.

Example

score_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
    --host 0.0.0.0 --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=score_process)

Example

# Score the probability of different completions given a query
query = "The capital of France is"
items = ["Paris", "London", "Berlin"]

url = f"http://localhost:{port}/v1/score"
data = {
    "model": "qwen/qwen2.5-0.5b-instruct",
    "query": query,
    "items": items,
    "label_token_ids": [9454, 2753],  # e.g. "Yes" and "No" token ids
    "apply_softmax": True,  # Normalize probabilities to sum to 1
}

response = requests.post(url, json=data)
response_json = response.json()

# Display scores for each item
for item, scores in zip(items, response_json["scores"]):
    print_highlight(f"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}")

Example

terminate_process(score_process)

v1/decisions (typed decisions on a chat model)

Answer typed questions about an input with per-option probabilities, without generating. The server renders each question with the model’s chat template and thinking turned off, assigns one-token answer labels, and scores them through the same path as v1/score. Parameters:

input: Text, object, or array the questions are about
questions: List of questions, each with a unique id and a type of choice (with options), score (with levels), or yes_no
temperature: Divides the label logits before the softmax (default: 1)
chat_template_kwargs: Extra chat template arguments
prompt_format_version: Optional pin of the server-owned prompt wording
return_prompt_token_ids: Return the scored prompt and label token ids for replay through v1/score (default: False)

The response contains answers keyed by question id, each with type, probabilities, and label_mass, plus choice for a choice question or score for a score question. A yes or no answer is read from probabilities["yes"]. See Decision models for examples, the full reference, and error cases.

Classify (reward model)

SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations.

Example

# Note that SGLang now treats embedding models and reward models as the same type of models.
# This will be updated in the future.

reward_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=reward_process)

Example

from transformers import AutoTokenizer

PROMPT = (
    "What is the range of the numeric output of a sigmoid node in a neural network?"
)

RESPONSE1 = "The output of a sigmoid node is bounded between -1 and 1."
RESPONSE2 = "The output of a sigmoid node is bounded between 0 and 1."

CONVS = [
    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE1}],
    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE2}],
]

tokenizer = AutoTokenizer.from_pretrained("Skywork/Skywork-Reward-Llama-3.1-8B-v0.2")
prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False)

url = f"http://localhost:{port}/classify"
data = {"model": "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2", "text": prompts}

responses = requests.post(url, json=data).json()
for response in responses:
    print_highlight(f"reward: {response['embedding'][0]}")

Example

terminate_process(reward_process)

Capture expert selection distribution in MoE models

SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization. Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.

Example

expert_record_server_process, port = launch_server_cmd(
    "python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning"
)

wait_for_server(f"http://localhost:{port}", process=expert_record_server_process)

Example

response = requests.post(f"http://localhost:{port}/start_expert_distribution_record")
print_highlight(response)

url = f"http://localhost:{port}/generate"
data = {"text": "What is the capital of France?"}

response = requests.post(url, json=data)
print_highlight(response.json())

response = requests.post(f"http://localhost:{port}/stop_expert_distribution_record")
print_highlight(response)

response = requests.post(f"http://localhost:{port}/dump_expert_distribution_record")
print_highlight(response)

Example

terminate_process(expert_record_server_process)

Tokenize/Detokenize Example (Round Trip)

This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization.

Example

tokenizer_free_server_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct
""")

wait_for_server(f"http://localhost:{port}", process=tokenizer_free_server_process)

Example

import requests
from sglang.utils import print_highlight

base_url = f"http://localhost:{port}"
tokenize_url = f"{base_url}/tokenize"
detokenize_url = f"{base_url}/detokenize"

model_name = "qwen/qwen2.5-0.5b-instruct"
input_text = "SGLang provides efficient tokenization endpoints."
print_highlight(f"Original Input Text:\n'{input_text}'")

# --- tokenize the input text ---
tokenize_payload = {
    "model": model_name,
    "prompt": input_text,
    "add_special_tokens": False,
}
try:
    tokenize_response = requests.post(tokenize_url, json=tokenize_payload)
    tokenize_response.raise_for_status()
    tokenization_result = tokenize_response.json()
    token_ids = tokenization_result.get("tokens")

    if not token_ids:
        raise ValueError("Tokenization returned empty tokens.")

    print_highlight(f"\nTokenized Output (IDs):\n{token_ids}")
    print_highlight(f"Token Count: {tokenization_result.get('count')}")
    print_highlight(f"Max Model Length: {tokenization_result.get('max_model_len')}")

    # --- detokenize the obtained token IDs ---
    detokenize_payload = {
        "model": model_name,
        "tokens": token_ids,
        "skip_special_tokens": True,
    }

    detokenize_response = requests.post(detokenize_url, json=detokenize_payload)
    detokenize_response.raise_for_status()
    detokenization_result = detokenize_response.json()
    reconstructed_text = detokenization_result.get("text")

    print_highlight(f"\nDetokenized Output (Text):\n'{reconstructed_text}'")

    if input_text == reconstructed_text:
        print_highlight(
            "\nRound Trip Successful: Original and reconstructed text match."
        )
    else:
        print_highlight(
            "\nRound Trip Mismatch: Original and reconstructed text differ."
        )

except requests.exceptions.RequestException as e:
    print_highlight(f"\nHTTP Request Error: {e}")
except Exception as e:
    print_highlight(f"\nAn error occurred: {e}")

Example

terminate_process(tokenizer_free_server_process)

Basic Usage

Advanced Features

Supported Models

Developer Guide

References

Launch A Server

Generate (text generation model)

Get Model Info

Get Server Info

Health Check

Flush Cache

Update Weights From Disk

Encode (embedding model)

v1/rerank (cross encoder rerank model)

v1/score (decoder-only scoring)

v1/decisions (typed decisions on a chat model)

Classify (reward model)

Capture expert selection distribution in MoE models

Tokenize/Detokenize Example (Round Trip)

​Launch A Server

​Generate (text generation model)

​Get Model Info

​Get Server Info

​Health Check

​Flush Cache

​Update Weights From Disk

​Encode (embedding model)

​v1/rerank (cross encoder rerank model)

​v1/score (decoder-only scoring)

​v1/decisions (typed decisions on a chat model)

​Classify (reward model)

​Capture expert selection distribution in MoE models

​Tokenize/Detokenize Example (Round Trip)

Launch A Server

Generate (text generation model)

Get Model Info

Get Server Info

Health Check

Flush Cache

Update Weights From Disk

Encode (embedding model)

v1/rerank (cross encoder rerank model)

v1/score (decoder-only scoring)

v1/decisions (typed decisions on a chat model)

Classify (reward model)

Capture expert selection distribution in MoE models

Tokenize/Detokenize Example (Round Trip)