DSpark in vLLM: Verification Budgeted by Confidence, Not by Configuration

The vLLM Team blogged on 14 August 2026 about adaptive verification for DSpark (arXiv:2607.05147, 6 July 2026), landed in PR #47808 as enable_adaptive_verification. It removes the need to tune num_speculative_tokens per deployment and was measured on DeepSeek-V4-Pro-0813 at TP=8 on 8×B300 (SM100) with vLLM main at 73b8394.

The short version: vLLM no longer wastes work verifying speculative tokens that are unlikely to survive, especially when the GPU is busy. One configuration behaves like a long draft at low load and a short draft at high load.

Finding

Speculative decoding drafts a few tokens ahead and verifies them together to reduce decode steps. At batch size 1 that extra work is close to free because the GPU is memory bound with spare compute. At batch size 256 it competes directly with real tokens for compute, and every rejected draft token wastes throughput.

On DeepSeek-V4-Pro-0813 with a 7-token draft, the chance a drafted token survives verification falls from more than 70 percent at position 1 to less than 10 percent at position 7. A fixed draft length therefore pays for low-value tail tokens on every step. No single length is optimal across concurrencies because the point where the gamble stops paying off moves with load and with workload.

DSpark already scores each draft position with a learned confidence head. vLLM turns those scores into survival probabilities and uses them to decide, each step, how much of the draft to verify.

The scheduler picks a budget B that maximizes expected accepted tokens per microsecond. The numerator is one bonus token per sampling request plus the survival probabilities of the B best draft slots in the batch. The denominator is a step cost looked up from tables profiled once at startup by timing dummy steps and taking the median of five runs per shape. Inside captured CUDA graphs the cost is a staircase — a batch of 121 tokens runs the 128-token graph and pays for 128 — with a sharp jump when work falls out of graphs. The optimizer strongly prefers to stay inside. The curve is forced monotonic to suppress kernel tile noise.

Because survival only decreases with position, the global top-B across the batch naturally yields a contiguous prefix per request. Slots compete across requests, so position 5 of a confident request can outrank position 1 of an uncertain one. The blog illustrates 21 drafted slots trimmed to the best 11.

Two engineering details make this practical. Sizing runs on the CPU while the GPU executes the previous step from a double-buffered confidence array that is one step stale, so no round trip is added. The per-request distribution of the B slots runs on the GPU against current confidences via a Triton kernel from torch.compile. And variable-length verification requires attention kernels that support per-request query lengths. The DSV4 sparse MLA family is naturally varlen and DeepSeek open-sourced a varlen indexer kernel in DeepGEMM in the same PR. Graphs are captured with num_reqs = min(num_tokens, max_num_seqs) and max_query_len = num_speculative_tokens + 1, so one graph serves any mix of 1 to 8 tokens per request when k=7.

The benchmark is 880 prompts at temperature 1.0 with up to 2048 output tokens, swept over concurrency 1 to 256. Adaptive with num_speculative_tokens: 7 stays on the throughput versus interactivity Pareto frontier across the sweep. It reads as a long fixed block at low concurrency and a short one at high concurrency, without retuning.

Scope and limits are explicit. Results are from a single model, accelerator type, and high-temperature prompt distribution. Startup profiling against a synthetic 8192-token KV context is an approximation and will drift from real distributions. Full varlen decode graphs require AttentionCGSupport.ALWAYS, which only the DSV4 sparse MLA, sparse SWA, and indexer backends report on SM100, otherwise the feature is rejected at startup rather than falling back to PIECEWISE. --enforce-eager, LoRA, and pipeline parallelism are not supported, and output logprobs are rejected because verification compacts logits after the forward pass.

Meaning

The change is not better drafts. DSpark already improves draft quality with a semi-autoregressive head. The change is how much of the draft the system pays to verify.

A fixed length is a single operating point tuned for one load and one acceptance rate. Neither is stable in serving. Adaptive verification moves that choice from deployment configuration to a per-step decision driven by two observables: where confidence is high and where the cost curve is flat.

The cost model is what makes it cheap enough to do every step. Two flat lookup tables — verification cost by token count and drafting cost by request count — are populated once at startup. The staircase shape carries the information the scheduler needs: staying within a captured graph is cheap, crossing it is not.

The execution split matters. Sizing on stale confidences while allocating on fresh ones lets the budget decision overlap GPU work. The GPU kernel never asks the host how the budget divides, which avoids the synchronization cost that would otherwise erode the throughput the budget is trying to protect.

The current boundary is hardware-specific. Until more attention backends expose per-request query lengths, the on-by-default claim holds only where the stack reports full varlen graph support. That is a deployment constraint, not a tuning detail.

Connection

This follows a pattern seen whenever a system multiplexes speculative work alongside committed work.

Network congestion control faced the same shift. A fixed window tuned for one link degrades elsewhere because the useful window depends on what the receiver will acknowledge and what the network can carry now. Modern control moved the window from a static setting to a signal derived from observed delivery.

Verification budgeting has the same structure. A draft token is an in-flight packet. Its survival probability predicts whether the verification work will be acknowledged by the target model. The profiled step cost is the time to carry it. The scheduler opens the window when survival is high and the cost curve is flat, and closes it when survival is low or the curve steepens. The global ranking adds one property networks do not: confidence competes across flows, so the budget goes where it is most likely to be acknowledged.

The broader point is that inference serving benefits when its speculative policy observes its own execution cost, not only its prediction quality. Longer drafts, better indexers, and larger capture sets raise the ceiling. A cost-aware budget keeps the system from paying for the ceiling when conditions do not warrant it.

Reproduction notes and server commands are in the original post. The feature is on vLLM main behind speculative_config: { method: "dspark", num_speculative_tokens: 7, enable_adaptive_verification: true } with kv-cache-dtype fp8 and max_cudagraph_capture_size set to (num_speculative_tokens + 1) * max_num_seqs to keep verification inside graphs.