Spark Bake-off: Qwen3.8-27B on one DGX Spark

Out-of-the-box numbers before any tuning: SGLang cookbook “DGX Spark” recipe, NVFP4 weights, no speculative decoding. Measured 2026-10-31 on spark1 (ASUS Ascent GX10, GB10, 128 GB unified memory).

This is the line to beat. Each team gets one Spark and tunes it for speed and concurrency; the scoring rules are below.

Concurrency sweep (1k-token prompt, 1k-token reply)

All runs

TTFT = time to first token. TPOT = time per output token after the first (per-user tok/s = 1000 / TPOT). Server became ready 242 s after launch. chunked_prefill_size=2048 context_len=262144 max_prefill_tokens=16384 max_running_requests=44 max_total_num_tokens=1162733

Thermal limit hit: the 8k in / 128 out run at 16 users was stopped by the safety watchdog when the hottest CPU sensor reached 92 °C (the GX10 powers itself off around 93–97 °C). After an hour of continuous load, 16 simultaneous 8K-token prompts heated the box from about 81 °C to 92 °C in roughly two and a half minutes, so this point has no stock result. Heavy prompt processing is thermally limited on this hardware; both Sparks need identical placement and airflow.

Accuracy gate

Speed only counts if quality holds. These are the stock server’s scores (lm-evaluation-harness, thinking off, greedy decoding); a tuned server must reach the gate on every test.

The eval client was paused 23 time(s) to let the CPU cool; pausing does not affect scores. ± is one standard error. A tuned score below the gate but within two standard errors of stock is re-run before it counts as a fail.

How the baseline was run

Server, exactly as published in the SGLang cookbook (image pinned to the version it was validated on):

docker run --gpus all --shm-size 32g -p 30000:30000 -v ~/.cache/huggingface:/root/.cache/huggingface --ipc=host \
  lmsysorg/sglang \
  sglang serve \
    --trust-remote-code \
    --model-path RadixArk/Qwen3.8-27B-NVFP4 \
    --kv-cache-dtype fp8_e4m3 \
    --mem-fraction-static 0.80 \
    --attention-backend flashinfer \
    --chunked-prefill-size 2048 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mamba-radix-cache-strategy extra_buffer \
    --mamba-ssm-dtype float32 \
    --host 0.0.0.0 \
    --port 30000

Load generator (SGLang’s own bench_serving, same image, same box, real ShareGPT text trimmed to fixed lengths, replies forced to full length):

python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 30000 --model RadixArk/Qwen3.8-27B-NVFP4 \
  --dataset-name random --random-input-len 1024 --random-output-len 1024 --random-range-ratio 1.0 \
  --num-prompts $((4*C)) --max-concurrency $C --seed 42 --output-file sweep_1k1k.jsonl
# C = 1 2 4 8 16 32 64;  long-prompt shape: --random-input-len 8192 --random-output-len 128, C = 1 4 16

kernel: 7.0.0-1024-nvidia
driver: 580.178.04
dgx_release: 7.6.0
sglang: 0.5.19
image: lmsysorg/sglang@sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9
model_revision: 319f741cce68d7914884900c138a1fbb70a42f30

Proposed rules for the tuning round

1. One referee client. Same pinned benchmark tool and command on both Sparks, same model revision and chat template, temperature 0.
2. Real prompts. Random-token prompts overstate speculative decoding by about 23% (NVIDIA SPEED-Bench), so use ShareGPT or SPEED-Bench text. Reply length fixed by forcing full-length outputs.
3. Two shapes: chat (1k prompt / 1k reply) and long prompt (8k prompt / 1k reply).
4. Three scores: single-user tok/s and TTFT; the 1–64-user sweep (throughput vs per-user speed curve); and goodput, the most users served while p99 TTFT ≤ 2 s and p99 TPOT ≤ 100 ms (stretch tier: ≤ 40 ms), thresholds adapted from MLPerf Inference.
5. Accuracy gate. On the exact tuned server, GSM8K, IFEval and the 16K needle test must each score ≥ 99% of the stock scores above (MLPerf uses the same 99% rule against its reference model). Fail it and the speed doesn’t count.
6. Clean runs. Warm up first, repeat each test 3 times with different seeds and report the median, and log GPU clock and temperature. A GPU stuck below ~1,400 MHz under load means the power brick needs a reset, not a tune.
7. Thermals are part of the contest. Both Sparks get the same placement and airflow. The referee scripts stop a run when the hottest CPU sensor reaches 90–92 °C (the GX10 hard-powers-off at about 93–97 °C); a run stopped by the watchdog doesn’t count.

Context: there is no lightweight official standard for this. MLPerf Inference is the industry benchmark but takes weeks to run compliantly; InferenceX (SemiAnalysis) uses the same throughput-vs-interactivity method on fixed 1k/1k and 8k/1k shapes. These rules borrow from both.