[email protected]

llm-server

A 27B model with a 128K context window on a 32 GB Intel GPU, serving the coding agents on my MacBook. What I measured, what I changed, and what broke.

published

length4 min read

sourcechriscorbell/llm-server

I wanted the coding agents on my MacBook to use a model running on my own hardware. The Intel Arc Pro B70 made that affordable, since it has 32 GB of VRAM for a lot less than NVIDIA charges for the same amount of memory. The catch is that almost nobody runs inference on this card. I started from SergiioB's Intel Arc Pro B70 Inference Cookbook, then tested every setting on my own machine and wrote down what changed.

The machine is a Threadripper 3970X with 64 GB of RAM, one B70, and Ubuntu Server 26.04 with the in-tree xe driver.

What's running

  • Model: Qwen3.8-27B, quantized to 4-bit (GPTQ INT4). The weights take about 18 GB, which leaves the rest of the card for context.
  • Context: 131,072 tokens. The KV cache, which is the model's working memory for the conversation, is stored in FP8.
  • Speculative decoding: the model's own draft head, also shrunk to INT4, guesses four tokens ahead, and the full model checks them.
  • Engine: vLLM on Intel's XPU stack, one user at a time.
decode, thinking off
87tok/s
decode, thinking on
69tok/s
uncached 126K-token prompt
146s
the same prompt, cached
5.7s

With thinking on, which is how my agents actually use it, decode runs at about 69 tokens per second. The bigger number to watch is how long before the first token. A 126,000-token prompt the server hasn't seen before takes 146 seconds. That almost never happens in practice, because an agent resends the same growing conversation every turn and nearly all of it comes from the prefix cache. The same prompt a second time starts in under 6 seconds.

Why the obvious choice was wrong

The B70's matrix engines handle FP16, BF16, INT8, and INT4, but not FP8. On most cards, Qwen's official FP8 release would be the safe, low-loss pick. On this one, FP8 goes through a slower conversion path, and Intel's math library has an open bug about FP8 matrix math occasionally giving wrong results on these cards. So the weights are INT4.

Quality first, then context, then speed

That order decided most of the config. For coding, I'd rather have a 27B model at INT4 than a smaller model at higher precision. Agents fill the context with the repo, so 128K was a must. Speed came last.

Context was the hard part. With a 16-bit KV cache, the card tops out around 110,000 tokens, because the speculative decoding buffers and the model's recurrent state come out of the same memory. An FP8 cache holds almost twice as many tokens. The cost is prompt processing time: about 15 percent slower at 31K tokens and 20 percent slower at 90K. Cached requests lose a bit too, since the FP8 cache works in bigger blocks and more of each prompt gets reprocessed. I took the trade.

What helped, and what didn't

Two changes made a real difference. Shrinking the draft head to INT4 sped up decode by 26 to 31 percent, with no quality loss on my test tasks. Guessing four tokens ahead instead of three added another 6.6 percent on identical code requests.

The rest didn't:

  • Doubling the prefill chunk size didn't speed anything up and cost 17,000 tokens of cache. Prefill on this card is limited by the work per token, not by how tokens are grouped.
  • Pinning the process to one CPU core complex made no measurable difference.
  • vLLM 0.29.0 passed every check but wasn't faster, needed four patches, and held 1,400 fewer tokens of cache. I stayed on the pinned image.
  • llama.cpp's SYCL backend with a Q6_K build worked at 128K, but decoded at 34 tok/s with thinking on, about half of vLLM.

Why not just run it on the Mac

I ran the same base model on my M5 Pro MacBook with Inco's Splash and sent both machines identical requests. The B70 decoded at 91 tok/s to the Mac's 63. The bigger gap is prompt processing: an uncached 32K prompt took 21 seconds on the B70 and 102 on the Mac. The Mac was faster on cached prompts, 0.27 seconds vs 2.4, but under sustained load it got hot enough to slow down.

What broke

  • The engine crashed at startup with opendir failed: could not open device directory until I bind-mounted /dev/dri into the container. Mapping it as a device isn't enough, even with one GPU.
  • Level Zero rejects any single allocation over 4 GiB by default, and the KV cache is bigger than that. Setting UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1 fixes it.
  • The usual GPU monitors don't work. intel_gpu_top and btop's Intel collector are both hard-coded to the old i915 driver. sudo gputop works with xe.
  • Thinking comes back in message.reasoning, not message.reasoning_content. A client that only reads the older field shows no thinking at all.
  • The GPU faults every so often, so a watchdog catches it, saves diagnostics, and restarts the engine. My first version had a regex bug that matched any log line, so it kept restarting a healthy server over unrelated network messages. The fixed version only reacts to real GPU faults. It passes its tests but hasn't caught a real one yet.

The client side

The repo also has the MacBook side: tested configs for Claude Code, Codex, OpenCode, and Hermes. Claude Code doesn't need a proxy, because vLLM serves Anthropic's Messages API directly. It needs three settings: the key sent as a bearer token, an effort of low, medium, or xhigh (plain "high" gets a 500), and the context limit set explicitly, or it assumes 200K. My pi config lives in its own repo.