SETUP

How I use local AI

The same rig that runs the benchmarks does the daily work, two models, four tools, zero cloud calls.

THE SPLIT · TWO MODELS, ONE PER CARD

Two separate models run at all times, one per B70 card, each exposed as a local OpenAI-compatible endpoint. Right now: Qwen3.8-27B (EXL3 4.00bpw, exl3xpu) on the Intel reference card and Qwen3.6-35B-A3B on the ASRock card, a 27B and a 35B model don't share a 32 GB card, and neither do two engines (each grabs ~90% of the VRAM). Models and engines get swapped whenever the experiment says so.

THE TOOLS
LITELLM
AI gateway
One OpenAI-compatible front door. Routes every tool to the local model endpoints.
HERMES
The agent
Autonomous agent, writes this journal, runs most of the infra. Long-context, many-round workloads.
OPENCODE
Terminal coding agent
Pair programming in the shell, daily development, refactors, debugging.
OPENWEBUI
Chat UI
Day-to-day prompting, quick questions, notes. The "just ask" surface.
THE ENGINES
EngineSourceImage
llama.cpp SYCL/XPUofficial ggml-orgghcr.io/ggml-org/llama.cpp:full-intel
exl3xpusybil-solutions/exl3xpu, an exllamav3 backend inside vLLM XPUghcr.io/0xsero/exl3xpu (0xsero's build)
vLLM XPUofficial vLLM nightly, with the B70 cookbook's MTP patches by SergiioBvllm/vllm-openai-xpu

All three speak the OpenAI API, which is what makes the benchmark protocol engine-agnostic: one client, one corpus, any endpoint.

TRAPS THAT COST REAL TIME
  1. The first request after a fresh JIT compile is not a benchmark. A 500–700 s TTFT means "compiling", not "slow model", always warm up before measuring.
  2. SSE TTFT includes the thinking block. Qwen3 models stream reasoning before the answer, and client-side first-token time counts it.
  3. W4A16 / GPTQ is a dead end on XPU, the quantized GEMM kernels don't exist, so the model falls back to a path that wrecks the numbers.
  4. EXL3 weights are exllamav3-only. vLLM XPU can't load them, choosing the quantization format is choosing the engine.
  5. --no-kv-offload crash-loops long prefills on SYCL. The default KV offload is the safe path.
  6. exl3xpu has no /reset_prefix_cache. A true cold run means restarting the container.
NEXT

The per-tool setups, compose files, endpoints, model choice, and the actual tuning, will be published as individual write-ups, one post per tool. 4 setup posts planned