Qwen3.8-27B · EXL3 4.00bpw · exl3xpu

Status: ready to launch, blockers resolved (b70-vllm stopped, server port fixed to 9932).

Config

Source: https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook/blob/master/docs/qwen38-27b/EXL3-XPU.md

Launch blockers

Both resolved before launch: b70-vllm stopped (frees card2 + host port 9932), server port changed 9931 → 9932 to match the published mapping.

Alternate variant: SC_5.00bpw_H6 mixed-K (the doc's quality pick)

• Weights: same repo @ dfe2f4fd71bf9a1994051d4d47e6460fa96dd723 (~20 GB), branch SC_5.00bpw_H6, mixed trellis rates K=4–7 instead of flat 4.00 bpw
• Image: ghcr.io/sergiiob/exl3xpu-mixedk:20261014@sha256:933eda54bb51a2e8a6669a70f5a2d7696fabd12a5f8b7c469df02fdfecc79d3a
• Same serve config models/qwen3.8-27b-exl3-4.00bpw, model dir passed via --model-path
• Doc numbers: ~0.89× decode of plain 4.00bpw; KL vs BF16 = 0.418×/0.161× of plain (wikitext/chat); context to 196K @ gmu 0.94

Test protocol

Standard: frozen corpus (prompt-corpus.txt, ~65k tokens) + "Summarize the main events of this text in exactly 10 bullet points.", temp=0, max_tokens=512. Cold (req #1) then warm (req #2). Client-side streaming timing; server logs + VRAM pasted after.

Stats

Extra data point: a manual warm test of the same corpus ran ~01:05:54 (TTFT ≈1.5 s, E2E ≈11.1 s, 512 completion tokens, derived from /metrics sums). /metrics totals include it.

Startup (from logs, 2026-10-17 00:51–00:55 UTC)

• vLLM 0.26.1.dev0+g568afb3a1.d20260907, quantization exl3, dtype float16 (bf16→fp16 cast again), APC on, cudagraph FULL_DECODE_ONLY (sizes 1–64)
• Weights 17.34 GiB in memory (checkpoint 15.70 GiB; MTP draft head shares target embedding + lm_head), load 14.9 s
• KV pool: 10.22 GiB → 264,846 tokens; max concurrency for 131,072-token requests: 2.02× (baseline vLLM run: 322k tokens / 2×)
• VRAM budget at start: weights 17.34 + peak activation 1.0 + non-torch 1.42 + cudagraph 0.2 + KV 10.22 ≈ 30.2 GiB; free at startup 30.98/31.89 GiB
• Engine init 164.84 s (torch.compile 119.86 s = backbone 103.04 + eagle head 16.81), graph capture 3 s, MM warmup 11 s → serving ~3.7 min after container start (faster than the doc's 6–7 min claim)
• Warning: model's generation_config.json overrides server sampling defaults (temp 1.0, top_k 20, top_p 0.95), our test pins temp=0 so it's unaffected, but note for any un-pinned use

Findings

• Cold prefill parity with baseline: 6,438.6 tok/s vs ~6,434.7 (vLLM GPTQ run #1), engine/quant choice makes no meaningful difference to cold prefill on card2.
• Steady-state cold TTFT ≈ 10 s (64,388 @ 6,438.6 tok/s). The measured ~43.2 s is inflated by one-time Triton JIT compiles of 7 kernels spread over 00:59:54–01:00:35 (~40 s), including the rejection sampler mid-decode.
• Decode gap vs baseline is confounded: 57.0 effective tok/s (MTP×3) vs 105.8 (MTP×4). Different model (27B dense vs 35B MoE A3B. MoE has far fewer active params/token and should decode faster), different quant, different MTP depth, different token mix. Cannot attribute the gap to EXL3 from this pair; needs a same-model cross-engine or same-config A/B run.
• MTP acceptance lower than baseline: mean length 2.54–2.63 vs 3.07; per-position decay 0.76→0.36 means position 3 adds little, depth 3 may be near the sweet spot for this model.
• Capacity: KV pool 264,846 tokens → 2.02× concurrency at 131k requests (baseline: 322k / ~2×). Weights 17.34 GiB vs 19.9.
• Startup faster than doc claim: serving in ~3.7 min vs the doc's 6–7 min (AOT compile cache on the shared volume helps).
• This build's SpecDecoding metrics appear per-interval (values reset between windows), unlike baseline's cumulative, record verbatim, never sum across windows.

Recommendations

Quality variant

1. SC_5.00bpw_H6 mixed-K weights, doc's quality pick: KL vs BF16 drops 0.42×/0.16× vs plain 4.00bpw at ~0.89× decode cost. A/B against plain is a natural follow-up run.
2. fp16 prefill (--set env.EXL3_INT8_PREFILL=0 on the serve command. NOT docker run -e, the serve config's env: block overrides it), int8 prefill adds KL ≈ 70 % of the whole weight-quantization error; fp16 costs 1.28–1.63× TTFT (less at longer prompts).

Speed variant

1. Raise max_num_batched_tokens above 4096, vLLM itself warns the 4096 default "may lead to suboptimal performance" with spec decoding enabled; it also caps prefill chunking.
2. Power cap at 230 W, most power-limited engine measured in the doc: +33–42 % decode paired medians vs 150 W.
3. Verify MTP is net-positive / tune depth, per-position acceptance decays to 0.36 at position 3; test without spec and at depth 2.
4. Prefix cache stays ON, warm long prompts are nearly free (TTFT 1.43 s vs ~10 s steady-state cold).
5. gmu 0.94 for single-client headless use if context headroom needed (not for concurrent users).

Measurement hygiene

• Doc's published decode figures are MTP chunk-rates (accepted steps), they undercount generated tokens. Our rows use client-side generated tok/s. Never mix the two across INDEX rows.
• Startup JIT compiles contaminate the first request (~40 s here); steady-state cold = rerun after JIT cache is warm.
• This build has no /reset_prefix_cache endpoint (404), a clean cold run requires a container restart.
• Unattributed requests land in the same /metrics sums and window logs (one manual corpus test appeared mid-run), note request provenance when reading server numbers.

Follow-up / next tests

• Same-model cross-engine A/B: Qwen3.8-27B on vLLM (GPTQ/FP8) vs this EXL3 setup, isolates engine from the confounded pair above.
• EXL3 without speculative_config, raw decode, isolates MTP net gain (acceptance decays to 0.36 at pos 3).
• Raise max_num_batched_tokens (4096 → 16384) and re-measure prefill/decode.
• Power cap 230 W, doc's biggest lever (+33–42 % decode); needs user-side setting.

Traps from source doc

• serve.py takes a config positional (models/qwen3.8-27b-exl3-4.00bpw) + --set k=v overrides; bare vLLM flags are rejected.
• Image is ~32 GB, doc mandates an isolated dockerd with its own --data-root on small root disks (containerd snapshotter writes to /var/lib/containerd).
• One agent per card: 4 concurrent agents gave only +16 % total tokens with 4×/7× worst-case latency. max_num_seqs 16.
• W4A16/GPTQ family is CLOSED for this model (arithmetic canary failures + prefix-cache token-salad corruption on the vLLM-XPU compressed-tensors route). EXL3 is the route of record.

Appendix: docker-compose (verbatim, as provided 2026-10-21)

services:
  exl3xpu:
    image: ghcr.io/0xsero/exl3xpu@sha256:21412bdd7535e9c653eeb3d099dce3bc79a83d440def2cd9556111c79c870fa8
    container_name: b70
    restart: unless-stopped
    ports:
      - "9932:9932"
    devices:
      - /dev/dri
    group_add:
      - "992"
    environment:
      ZE_FLAT_DEVICE_HIERARCHY: COMPOSITE
      HF_HOME: /root/.cache/huggingface
      HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface/hub
    volumes:
      - ./models/qwen3.8-27b-exl3-4.00bpw:/mnt/exl3model:Z
      - ./cache:/root/.cache/huggingface:Z
    command:
      - models/qwen3.8-27b-exl3-4.00bpw
      - --gpu
      - "0"
      - --model-path
      - /mnt/exl3model
      - --set
      - vllm.gpu_memory_utilization=0.94
      - --set
      - vllm.max_model_len=131072
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_xml
      - --port
      - "9932"
      - --reasoning-parser
      - qwen3
      - --served-model-name 
      - "Qwen3.8-27B"

Note: appendix reflects the config as launched (server port corrected 9931 → 9932).