SGLang Inference System Optimization

LLM serving systems are full-stack performance problems: request scheduling, batching, KV-cache memory management, speculative decoding, CUDA-graph capture, and attention kernels all interact, and the fastest configuration for single sequential requests is rarely the fastest under concurrent load. Optimizing one without regressing the other (while keeping the model's outputs bit-for-bit faithful under greedy decoding) is what production inference teams spend months on.

The starting point here is deliberately not a strawman: the provided SGLang launch configuration already uses an FP8 KV cache, NEXTN/MTP speculative decoding, the extra-buffer mamba scheduler, CUDA graphs, and tuned page and memory settings. Flag-tuning alone is unlikely to yield much; real gains require custom kernels, SGLang or FlashInfer source modifications, or model surgery that preserves output equivalence.

The agent must make an SGLang serving instance with Qwen3.5-4B on a B200 GPU serve requests as fast as possible. Single-request latency across input/output shapes and batched requests at several concurrency levels matter comparably.

• The server is started through /app/server/launch_server.sh, which receives PORT and MODEL_PATH environment variables and must expose the OpenAI-compatible SGLang HTTP API.
• Correctness is a hard requirement: with temperature-0 greedy decoding, the optimized server must keep producing the outputs of the starting configuration, with an average whitespace-token prefix match of at least 0.95.
• Everything the server needs must live inside /app and be reapplied or imported by launch_server.sh: the server is later launched from a fresh copy of the workspace in which only /app carries over, so direct edits to site-packages or in-place edits to /app/model do not survive on their own.
• The design space is wide: scheduler and batching configuration, tuning or replacing the speculative pipeline, custom Triton or CUDA kernels, direct SGLang and FlashInfer source modifications, and model modifications (quantise, prune, fuse), all under the output-equivalence requirement.

A running serving workspace: Qwen3.5-4B weights pre-downloaded at /app/model, the tuned launch_server.sh starting configuration, and a local measurement loop. compare_outputs.pysnapshots the pristine server's greedy outputs on the visible 174-prompt dev set (dev_prompts.jsonl, generated from the same distribution as the held-back set) and diffs after every change; verify_serving.py sanity-checks the server and run_dev_bench.py measures latency across input-length × output-length quadrants plus a concurrent batch.

Preinstalled tooling includes the CUDA 12.8 dev toolkit (nvcc), PyTorch with Triton and torch.compile, the pinned SGLang + sgl_kernel + FlashInfer serving stack in site-packages, and gcc/ninja for native builds. There is no internet access at runtime (no package installs, no model downloads), and a sandbox-timer CLI reports the remaining budget.

Verification runs in a separate clean-room environment on the captured /app. It launches the baseline and candidate servers under the same unprivileged user, collects greedy outputs on a held-back prompt set, and measures an A/B/A benchmark: baseline session, candidate session, then a second baseline session whose samples are pooled with the first into the scored median.

• The correctness gate compares candidate outputs against the baseline's on hidden prompts (normal text, code, math, long-context, degenerate and adversarial inputs); an average whitespace-token prefix-match below 0.95 zeroes the result.
• Speed is measured on five sequential workloads covering the input-length × output-length quadrants plus a reasoning shape, and two concurrent workloads (4-way and 8-way mixed). Per-class geometric-mean speedups are mapped linearly to credit between a live-zero anchor at 1.02x (parity and no-op relaunches score exactly 0) and a 2.0x cap.
• The aggregate is capped at 0.30 + 0.70 × the weaker class's credit, so the top of the range is unreachable without genuine improvement in both sequential and concurrent serving; continuous regression gates on each class and on the weakest single workload collapse the score for real regressions while tolerating measured replay noise.
• The timed path itself is integrity-checked: every benchmark response is compared against the baseline's response to the same salted request, with mean and median prefix-match bars plus a completion-length ratio that catches servers truncating output under load. Failing any of these gates zeroes the result, as does a server that fails to launch.