Qwen3.8-27B · GGUF IQ3_S · llama.cpp

Third engine in the journal: llama.cpp on the same ASROCK B70 (card2/renderD129), same model family as runs #2/#3 (Qwen3.8-27B) but GGUF quant (IQ3_S GSQ "Intel Arc Tuned") with an embedded MTP draft head (Q4_K). Dual speculative decoding: ngram-mod + draft-mtp.

Status: awaiting launch, blocker: b70-exl3 currently holds port 9932 AND card2; stop it first.

Config

Benchmark protocol note: server sampling defaults (temp 0.7, penalties) differ from runs #1–#3. For comparability, the client request overrides to greedy-equivalent: temperature=0, top_p=1.0, top_k=1, repeat_penalty=1.0, presence_penalty=0.0. Prompt + question identical (64,388 corpus tokens).

Startup stats

• 20.4 GB of 32 GB at startup (user), large headroom: ctx→262k (+~2 GiB q4_0 KV) and f16 KV quality variant both fit comfortably
• Model source: Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF
• SYCL kernels compiled at startup (SYCL_CACHE_PERSISTENT=0), watch post-test logs for mid-inference re-JIT

Stats

Baselines to compare (same corpus): vLLM MoE run #1 warm decode 105.8 tok/s; EXL3 runs #2/#3 warm decode ~57 tok/s, prefill ~6,439 tok/s.

Findings

• Cold prefill is genuinely slow: 1,013.59 tok/s (server-measured, no JIT warnings in the log), 6.3× slower than EXL3/vLLM's ~6,439 tok/s on the same model family. Not a JIT artifact. Prime suspects: --ubatch-size 2048 too small for this hybrid model, and/or --kv-offload pushing KV writes over PCIe. Both are A/B candidates.
• Warm cache reuse works perfectly server-side: only 4 tokens re-prefilled in 351 ms (f_sim_best = 1.000). Yet the client saw TTFT 23.45 s, and cold shows the same constant gap (client TTFT 85.74 vs server prompt eval 63.64 ≈ +22 s; warm ≈ +23 s). A steady ~22–23 s first-token delivery delay independent of server work points at SSE/streaming buffering, not server compute. Follow up with curl -N raw timing.
• Same-model engine A/B (dense Qwen3.8-27B): llama.cpp warm decode 44.9 vs EXL3 57.0 tok/s. EXL3 is ~27% faster on decode. (Run #1's 105.8 was the MoE A3B, not comparable.)
• Draft acceptance is strong: 69–76 % per request (mean len 2.61/2.87), better than EXL3's MTP (~51–54 %), partly because ngram-mod exploits repetitive prose.
• VRAM barely moves under load (20.4 → 20.9 GB for 65k tokens of q4_0 KV), huge headroom confirmed for ctx→262k and f16 KV.
• Serving in ~1 min after container start (vs EXL3's 3.7 min); SYCL graphs reused (167 cold / 381 warm).
• Tokenizer overhead: 64,501 prompt tokens here vs 64,388 on vLLM/exl3xpu (tool definitions + jinja template).

Recommendations

Quality variant

1. KV dtype q4_0 → f16/bf16 (--cache-type-k/v), quantized KV is the main long-context recall tax; ~2× KV VRAM (~+3–4 GiB at 163k, still fits in 32 GB).
2. Higher-quant GGUF (IQ4_XS/IQ5) if IQ3_S quality disappoints, fits with room on one B70.
3. Draft KV already f16 (default), leave it.

Speed variant

1. ubatch-size 2048 → 4096, prefill is the headline weakness (1,014 tok/s vs ~6,439 on other engines); bigger chunks are the first lever.
2. --no-kv-offload A/B, offload is default-on; if KV writes are crossing PCIe during prefill, disabling it (everything fits on GPU) could close most of the prefill gap.
3. ctx-size 163840 → 262144, headroom confirmed: only +0.5 GB VRAM for 65k tokens of q4_0 KV (~8 B/token), so 262k ctx costs ~+2 GiB. No wall in sight.
4. SYCL_CACHE_PERSISTENT=0 → 1, sycl host volume already mapped but persistence off; recompiles every start. A/B after baseline.
5. Spec decode isolation, ngram-mod and draft-mtp both on (acceptance 69–76 %); A/B each alone to see net gain per type. --spec-draft-p-min 0.5 is conservative.
6. Follow-up: SSE first-token delay, constant ~22–23 s gap between server prompt-eval completion and client TTFT on both requests; verify with curl -N raw timing before trusting llama.cpp TTFT numbers.

Appendix: docker-compose (verbatim, as provided 2026-10-21)

services:
  llamacpp:
    container_name: b70
    image: ghcr.io/ggml-org/llama.cpp:full-intel
    restart: unless-stopped
    ports:
      - "9932:9932"
    volumes:
      - ./models:/models
      - ./cache/llamacpp:/root/.cache/llamacpp:Z
      - ./cache/sycl:/root/.cache/sycl:Z
      - ./cache/huggingface:/root/.cache/huggingface:Z
    devices:
      - /dev/dri
    group_add:
      - "992"
    environment:
      GGML_SYCL_ENABLE_GRAPH: 1
      UR_LOADER_USE_LEVEL_ZERO_V2: 0
      ZE_AFFINITY_MASK: 0
      SYCL_DEVICE_FILTER: level_zero:0
      ONEAPI_DEVICE_SELECTOR: level_zero:0
      SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS: "1"
      GGML_SYCL_ENABLE_COPY: "1"
      GGML_SYCL_F16: "1"
      ZES_ENABLE_SYSMAN: "1"
      OMP_NUM_THREADS: "12"
      HF_HOME: /root/.cache/huggingface
      HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface
      LLAMA_CPP_CACHE: /root/.cache/llamacpp
      SYCL_CACHE_PERSISTENT: "0"
      SYCL_CACHE_PATH: /root/.cache/sycl
      SYCL_USE_XMX: 1
    command:
      - --server
      - --model
      - /models/Qwen3.8-27B-GSQ-RCO-IQ3_S-Intel-Arc-Tuned-MTP-Q4_K.gguf
      - --spec-type 
      - "ngram-mod,draft-mtp"
      - --spec-ngram-mod-n-match 
      - "24"
      - --spec-ngram-mod-n-min
      - "1"
      - --spec-ngram-mod-n-max
      - "3"
      - --spec-draft-n-max
      - "3"
      - --spec-draft-p-min
      - "0.5" 
      - --jinja 
      - --chat-template-file 
      - /models/qwen-sharp.jinja
      - --alias 
      - "Qwen3.8-27B"
      - --host
      - 0.0.0.0
      - --port
      - "9932"
      - --n-gpu-layers
      - "all"
      - --flash-attn
      - "on"
      - --fit-target
      - "256"
      - --ctx-size
      - "163840"
      - --n-predict 
      - "-1"
      - --temp 
      - "0.7"
      - --top-p 
      - "0.9"
      - --top-k 
      - "40"
      - --repeat-penalty
      - "1.05"
      - --repeat-last-n
      - "256"
      - --presence-penalty
      - "0.0"
      - --dry-penalty-last-n
      - "512"
      - --threads
      - "8"
      - --threads-batch
      - "12"
      - --batch-size
      - "16384"
      - --ubatch-size
      - "2048"
      - --cors-origins
      - "*"
      - --cache-type-k
      - "q4_0"
      - --cache-type-v
      - "q4_0"
      - --cache-prompt
      - --cache-reuse
      - "2048"
      - --cache-ram
      - "2048"
      - --parallel
      - "1"
      - --kv-unified
      - --kv-offload
      - --reasoning
      - "on"
      - --tools
      - "read_file,file_glob_search,grep_search,write_file,edit_file,get_info"
      - --metrics
      - --slots
      - --warmup
      - --perf