vLLM/Recipes
Qwen

Qwen/Qwen3.6-27B

Qwen3.6 dense multimodal model (27B) with gated delta networks hybrid attention, MTP, and 262K context

Qwen3.6 flagship dense — single-GPU FP8 or 2x GPU BF16

View on HuggingFace
dense27B262,144 ctxvLLM 0.17.0+multimodaltext
Guide

Overview

Qwen3.6-27B is the flagship dense model of the Qwen3.6 family. It uses the same gated delta networks hybrid attention as its MoE siblings, supports vision+text input, and natively serves 262K context. MTP (multi-token prediction) is supported out of the box for low-latency decoding.

Prerequisites

  • vLLM version: >= 0.17.0
  • DGX Spark NVFP4 vLLM version: >= 0.24.0
  • Hardware (BF16): 1x H200 or 2x H100
  • Hardware (FP8): single 40 GB GPU (H100/H200/L40S)
  • Hardware (Int4): single 24 GB GPU
  • Hardware (NVFP4): single NVIDIA Blackwell GPU (B200/B300/GB10 DGX Spark)

Install vLLM

uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto

Launching the Server

Single-GPU FP8

vllm serve Qwen/Qwen3.6-27B-FP8 \
  --max-model-len 262144 \
  --reasoning-parser qwen3

BF16 on 2xH100 (TP2)

vllm serve Qwen/Qwen3.6-27B \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --reasoning-parser qwen3

MTP speculative decoding

vllm serve Qwen/Qwen3.6-27B-FP8 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
  --reasoning-parser qwen3

Text-only (skip vision encoder)

vllm serve Qwen/Qwen3.6-27B-FP8 \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-prefix-caching

DGX Spark GB10 NVFP4

The NVIDIA ModelOpt NVFP4 checkpoint is served from nvidia/Qwen3.6-27B-NVFP4 and requires vLLM 0.24.0+.

Single node (TP1)

# Use container: vllm/vllm-openai:v0.24.0-ubuntu2404

vllm serve nvidia/Qwen3.6-27B-NVFP4 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.5 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice

Multi-node (TP2)

Download the checkpoint on all nodes. Start the worker nodes first and then the head node. The head node coordinates the others, so they need to be ready to connect when it comes up.

# COMMON variables example values — same on all nodes
export HEAD_IP="<node1-connectx-ip>"   # The ConnectX IP of Node1 (Head Node)
export ETH_IF="enp1s0f1np1"            # CX7 ethernet interface which is UP
export IB_IF="rocep1s0f1,roceP2p1s0f1" # BOTH matching RoCE interfaces
export MASTER_PORT=29501               # Free port on the Head Node
export CONTAINER_NAME=vllm_node
export IMAGE="vllm/vllm-openai:v0.24.0-ubuntu2404"

# Head Node
docker run --privileged --ulimit nofile=1048576:1048576 --ipc=host \
  --gpus all --rm --network host --name "$CONTAINER_NAME" --entrypoint="" \
  -e MN_IF_NAME="$ETH_IF" \
  -e UCX_NET_DEVICES="$ETH_IF" \
  -e NCCL_SOCKET_IFNAME="$ETH_IF" \
  -e NCCL_IB_HCA="$IB_IF" \
  -e NCCL_IB_DISABLE=0 \
  -e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \
  -e GLOO_SOCKET_IFNAME="$ETH_IF" \
  -e TP_SOCKET_IFNAME="$ETH_IF" \
  -e NCCL_IGNORE_CPU_AFFINITY=1 \
  -e VLLM_FLOAT32_MATMUL_PRECISION=high \
  -e HEAD_IP="$HEAD_IP" \
  -e MASTER_PORT="$MASTER_PORT" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  "$IMAGE" vllm serve nvidia/Qwen3.6-27B-NVFP4 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --gpu-memory-utilization 0.5 \
  --max-model-len 262144 \
  --max-num-seqs 10 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice \
  --tensor-parallel-size 2 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr "$HEAD_IP" \
  --master-port "$MASTER_PORT"

# Worker node
docker run --privileged --ulimit nofile=1048576:1048576 --ipc=host \
  --gpus all --rm --network host --name "$CONTAINER_NAME" --entrypoint="" \
  -e MN_IF_NAME="$ETH_IF" \
  -e UCX_NET_DEVICES="$ETH_IF" \
  -e NCCL_SOCKET_IFNAME="$ETH_IF" \
  -e NCCL_IB_HCA="$IB_IF" \
  -e NCCL_IB_DISABLE=0 \
  -e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \
  -e GLOO_SOCKET_IFNAME="$ETH_IF" \
  -e TP_SOCKET_IFNAME="$ETH_IF" \
  -e NCCL_IGNORE_CPU_AFFINITY=1 \
  -e VLLM_FLOAT32_MATMUL_PRECISION=high \
  -e HEAD_IP="$HEAD_IP" \
  -e MASTER_PORT="$MASTER_PORT" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  "$IMAGE" vllm serve nvidia/Qwen3.6-27B-NVFP4 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --gpu-memory-utilization 0.5 \
  --max-model-len 262144 \
  --max-num-seqs 10 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice \
  --tensor-parallel-size 2 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr "$HEAD_IP" \
  --master-port "$MASTER_PORT" \
  --headless

RTX Pro 6000 NVFP4

The NVIDIA ModelOpt NVFP4 checkpoint is served from nvidia/Qwen3.6-27B-NVFP4 and requires vLLM 0.24.0+.

# Use container: vllm/vllm-openai:v0.24.0-ubuntu2404

vllm serve nvidia/Qwen3.6-27B-NVFP4 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice

NVFP4 on Blackwell

The nvidia/Qwen3.6-27B-NVFP4 checkpoint is NVIDIA's ModelOpt re-quantization: MLP linears drop to NVFP4 (W4A16) while the attention linears and KV cache stay FP8, so the ~22 GB weights fit a single Blackwell GPU. vLLM auto-detects the ModelOpt quantization from the checkpoint, so no --quantization flag is needed — just use a recent vLLM (NVIDIA recommends nightly or a source build with ModelOpt W4A16/NVFP4 support). NVIDIA reports near-lossless accuracy versus the FP8 baseline.

vllm serve nvidia/Qwen3.6-27B-NVFP4 \
  --max-model-len 262144 \
  --reasoning-parser qwen3

Processing Ultra-Long Texts

Qwen3.6-27B natively supports 262,144 tokens. For longer inputs, apply YaRN RoPE scaling via --hf-overrides and raise --max-model-len. Pick factor to match your real workload — 2.0 covers ~524K, 4.0 covers ~1M — since YaRN at higher factors degrades short-context quality.

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 1010000 \
  --reasoning-parser qwen3 \
  --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'

See the model card for the full parameter reference.

Client Usage

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B",
    messages=[{"role": "user", "content": "Write a haiku about gated delta networks."}],
    max_tokens=256,
)
print(resp.choices[0].message.content)

Troubleshooting

  • CUDA graph / Mamba cache size error: reduce --max-cudagraph-capture-size (default 512). See vLLM PR #34571.
  • Disable reasoning: add --default-chat-template-kwargs '{"enable_thinking": false}'.
  • Prefix Caching (Mamba): currently experimental in "align" mode.

References