Spark Bake-off: Qwen3.8-27B on one DGX Spark

Out-of-the-box numbers before any tuning: SGLang cookbook “DGX Spark” recipe, NVFP4 weights, no speculative decoding. Measured 2026-10-31 on spark1 (ASUS Ascent GX10, GB10, 128 GB unified memory).

This is the line to beat. Each team gets one Spark and tunes it for speed and concurrency; the scoring rules are below.

Concurrency sweep (1k-token prompt, 1k-token reply)

All runs

TTFT = time to first token. TPOT = time per output token after the first (per-user tok/s = 1000 / TPOT). Server became ready 242 s after launch. chunked_prefill_size=2048 context_len=262144 max_prefill_tokens=16384 max_running_requests=44 max_total_num_tokens=1162733

Accuracy gate

Speed only counts if quality holds. These are the stock server’s scores (lm-evaluation-harness, thinking off, greedy decoding); a tuned server must reach the gate on every test.

Running after the speed sweep: GSM8K (full 1,319), IFEval (full 541) and a 16K-token RULER needle test (100 samples).

How the baseline was run

Server, exactly as published in the SGLang cookbook (image pinned to the version it was validated on):

docker run --gpus all --shm-size 32g -p 30000:30000 -v ~/.cache/huggingface:/root/.cache/huggingface --ipc=host \
  lmsysorg/sglang \
  sglang serve \
    --trust-remote-code \
    --model-path RadixArk/Qwen3.8-27B-NVFP4 \
    --kv-cache-dtype fp8_e4m3 \
    --mem-fraction-static 0.80 \
    --attention-backend flashinfer \
    --chunked-prefill-size 2048 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mamba-radix-cache-strategy extra_buffer \
    --mamba-ssm-dtype float32 \
    --host 0.0.0.0 \
    --port 30000

Load generator (SGLang’s own bench_serving, same image, same box, real ShareGPT text trimmed to fixed lengths, replies forced to full length):

python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 30000 --model RadixArk/Qwen3.8-27B-NVFP4 \
  --dataset-name random --random-input-len 1024 --random-output-len 1024 --random-range-ratio 1.0 \
  --num-prompts $((4*C)) --max-concurrency $C --seed 42 --output-file sweep_1k1k.jsonl
# C = 1 2 4 8 16 32 64;  long-prompt shape: --random-input-len 8192 --random-output-len 128, C = 1 4 16

kernel: 7.0.0-1024-nvidia
driver: 580.178.04
dgx_release: 7.6.0
sglang: 0.5.19
image: lmsysorg/sglang@sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9
model_revision: 319f741cce68d7914884900c138a1fbb70a42f30

Proposed rules for the tuning round

1. One referee client. Same pinned benchmark tool and command on both Sparks, same model revision and chat template, temperature 0.
2. Real prompts. Random-token prompts overstate speculative decoding by about 23% (NVIDIA SPEED-Bench), so use ShareGPT or SPEED-Bench text. Reply length fixed by forcing full-length outputs.
3. Two shapes: chat (1k prompt / 1k reply) and long prompt (8k prompt / 1k reply).
4. Three scores: single-user tok/s and TTFT; the 1–64-user sweep (throughput vs per-user speed curve); and goodput, the most users served while p99 TTFT ≤ 2 s and p99 TPOT ≤ 100 ms (stretch tier: ≤ 40 ms), thresholds adapted from MLPerf Inference.
5. Accuracy gate. On the exact tuned server, GSM8K, IFEval and the 16K needle test must each score ≥ 99% of the stock scores above (MLPerf uses the same 99% rule against its reference model). Fail it and the speed doesn’t count.
6. Clean runs. Warm up first, repeat each test 3 times with different seeds and report the median, and log GPU clock and temperature. A GPU stuck below ~1,400 MHz under load means the power brick needs a reset, not a tune.

Context: there is no lightweight official standard for this. MLPerf Inference is the industry benchmark but takes weeks to run compliantly; InferenceX (SemiAnalysis) uses the same throughput-vs-interactivity method on fixed 1k/1k and 8k/1k shapes. These rules borrow from both.