Qwen3.8-27B · EXL3 · batch 4096→8192

Single-variable A/B against run #2: only vllm.max_num_batched_tokens changed (4096 → 8192). Everything else identical (same image, weights, card, port). MTP×3 stays on; power cap unchanged (user deferred).

Status: complete (2026-10-17 ~01:50 UTC), no measurable speedup from the change; see Findings.

Config

Hypothesis

Run #2's startup log warned: max_num_scheduled_tokens is set to 4096 based on the speculative decoding settings … may lead to suboptimal performance. Raising the batch budget to 8192 should give the scheduler room for draft-token slots and larger prefill chunks → expect equal-or-better decode and faster cold prefill; watch for any regression in TTFT or KV pressure (gmu headroom was only ~1.3 GB at peak).

Stats

Startup (from logs, 2026-10-17 01:46–01:49 UTC)

• max_num_batched_tokens=8192 confirmed in non-default args; chunked prefill on at 8192; encoder cache budget 8192 (was 4096)
• Run #2's scheduler warning is gone, no more "max_num_scheduled_tokens is set to 4096 … suboptimal performance"
• Compile range now (1, 8192): backbone 93.08 s + eagle head 20.29 s (fresh AOT artifacts for the new range); engine init 170.98 s (compile 124.72 s); serving ~3.8 min after container start
• KV pool shrank: available 9.1 GiB (was 10.22) → 236,470 tokens (was 264,846); max concurrency at 131,072 tokens/request: 1.80× (was 2.02×). Peak activation 1.91 GiB (was 1.0), non-torch 1.63 (was 1.42)
• Weights unchanged: 17.34 GiB; free at startup 30.98/31.89 GiB

Findings

• No measurable speedup: warm decode 56.6 vs 57.0 tok/s (run #2), warm TTFT 1.40 vs 1.43 s, both within noise. The 4096 batch budget was not the bottleneck for single-request long-context work; the scheduler warning it produced was cosmetic.
• Real cost paid: KV pool −28k tokens (264,846 → 236,470), concurrency at 131k dropped 2.02× → 1.80×, this config can no longer hold two concurrent full-context requests.
• Cold numbers: prefill throughput is literally unchanged (6,438.6 → 6,438.6 tok/s), the batch budget doesn't touch prefill speed on this workload. Cold decode 58.1 tok/s is solidly measured (server windows agree); run #2's 26–51 estimate was JIT-contaminated mid-decode.
• The same 7 Triton kernels re-JIT'd (~40 s spread) despite the on-disk JIT cache, the new compile range (1, 8192) forces fresh kernel shapes. Any config change that moves the compile range pays a one-time cold penalty.
• MTP acceptance slightly better than run #2 (2.79 vs 2.63 cold; ~2.6–2.9 vs 2.54 warm), within noise, possibly scheduler headroom.
• Verdict: revert max_num_batched_tokens to 4096 unless a future multi-request workload shows otherwise.

Recommendations

Quality variant

Carried from run #2 (unchanged by this variable): SC_5.00bpw_H6 mixed-K weights A/B; fp16 prefill (EXL3_INT8_PREFILL=0).

Speed variant

1. Revert max_num_batched_tokens to 4096, no speed gain at 8192, cost −28k KV tokens (concurrency 2.02× → 1.80× at 131k).
2. MTP off / depth-2 A/B, next single-variable test (acceptance decays to ~0.36 at pos 3).
3. Power cap 230 W, deferred by user; doc's biggest lever (+33–42 % decode).
4. APC stays ON; gmu 0.94 for single-client use.

Appendix: docker-compose (verbatim, as provided 2026-10-21)

services:
  exl3xpu:
    image: ghcr.io/0xsero/exl3xpu@sha256:21412bdd7535e9c653eeb3d099dce3bc79a83d440def2cd9556111c79c870fa8
    container_name: b70
    restart: unless-stopped
    ports:
      - "9932:9932"
    devices:
      - /dev/dri
    group_add:
      - "992"
    environment:
      ZE_FLAT_DEVICE_HIERARCHY: COMPOSITE
      HF_HOME: /root/.cache/huggingface
      HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface/hub
    volumes:
      - ./models/qwen3.8-27b-exl3-4.00bpw:/mnt/exl3model:Z
      - ./cache:/root/.cache/huggingface:Z
    command:
      - models/qwen3.8-27b-exl3-4.00bpw
      - --gpu
      - "0"
      - --model-path
      - /mnt/exl3model
      - --set
      - vllm.gpu_memory_utilization=0.94
      - --set
      - vllm.max_model_len=131072
      - --set
      - vllm.max_num_batched_tokens=8192
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_xml
      - --port
      - "9932"
      - --reasoning-parser
      - qwen3
      - --served-model-name 
      - "Qwen3.8-27B"