Running Qwen 27B on an Intel Arc Pro B70

I wanted the coding agent on my MacBook to talk to a model running on hardware I own. The Intel Arc Pro B70 made that affordable: 32 GB of VRAM for far less than NVIDIA charges for the same memory. The catch is that almost nobody runs inference on this card, so every setting had to be found by measurement. This repository is the record of finding them.

The box is a Threadripper 3970X with 64 GB of RAM, one B70, and Ubuntu with the in-tree xe driver. One fact about the card shapes everything else: its matrix engines have no native FP8.

What runs

Qwen3.8-27B with GPTQ INT4 weights, which take about 14 GB and leave the rest of the card for context. On top of that, an FP8 key-value cache, a small INT4 draft model for speculative decoding that predicts four tokens at a time, and a 131,072-token context window. vLLM on Intel's XPU stack serves it, one user at a time.

For one person with thinking off, warm decode runs at about 87 tokens per second. That is faster than I read. The cold start on a prompt near the 124,000-token ceiling takes 142 seconds. A cached repeat of the same prompt takes 5.

Quality, then context, then speed

That ordering decided most of the configuration. A 27B model at INT4 beats a smaller model at higher precision for the coding work I do. Agent work fills the context with the repository, so 128K was a requirement. Decode speed came last, and 87 tokens per second is more than a single reader needs.

Context was the hard one. A 16-bit KV cache tops out near 110,000 tokens on this card, because the speculative decoding buffers and the model's recurrent state come out of the same budget. Storing the KV cache in FP8 nearly doubles the number of tokens it can hold, and costs about 15 percent on cold time to first token at 31K input. I took that trade.

What did nothing

Doubling the prefill chunk size bought nothing and cost 17,000 tokens of KV capacity. Prefill on this card is limited by per-token work, so grouping tokens differently does not help. Pinning the process to two core complexes moved throughput by fractions of a percent. A new vLLM release passed every check and gained nothing measurable, so the pinned image stayed. The four-token draft did earn its place, with a 6.6 percent gain over three tokens on the same requests.

What broke

The engine dies at startup unless /dev/dri is bind-mounted into the container. Mapping it as a device is not enough.

The GPU monitors I expected to work do not. intel_gpu_top is hard-coded to the old i915 driver's counters, and so is btop's Intel collector. gputop is the one that works under xe.

The GPU occasionally faults. A watchdog notices, saves diagnostics, and restarts the engine, because a crashed inference server should not stay down until I happen to look.

The clients

The same repository holds the MacBook side: configuration for pi, Codex, OpenCode, and Hermes, each checked against the server before it went in. The vLLM recipe and the reference measurements started from SergiioB's Intel Arc Pro B70 Inference Cookbook. This repository adapts that work to one specific machine and writes down what changed.