Every build we tested

Not just the winners. The crashes, regressions, and abandoned attempts are here too.

Complete build history

The "ROCm stacked" build combined PR #21344 with -ffast-math and hipBLASLt. We expected additive gains. Instead, prefill dropped from 354 to 230 t/s. Not every optimization stacks cleanly.

What broke along the way

Real workload timing

Server-mode tests with actual prompts. 256-token generation, no_think mode, temp=0.3.

Speculative decoding

Target: Qwen3.5-122B-A10B-REAP-20-Q6_K (76 GB). Draft: Qwen3.5-0.8B-Q4_K_M (508 MB). This should not have worked out of the box. It didn't.

Results after the fixes

no_think mode is where spec decode shines: 40% speedup with 90% acceptance. Thinking mode drops acceptance to ~76% because reasoning tokens are harder to predict.

The four bugs we had to fix

Beyond PR #20075, these fixes were required for Qwen3.5 on non-Metal backends:

• Hybrid seq_rm skipped attention cleanup. When recurrent rollback failed, the function returned early and never called mem_attn->seq_rm(). Stale positions accumulated, breaking M-RoPE invariants.
• Soft rollback corrupted positions. The partial removal path moved cell positions backward (cells[i].pos = p0 - 1) instead of erasing them. seq_pos_max then returned stale higher positions.
• Compat check was chicken-and-egg. common_speculative_is_compat() tested seq_rm on a fresh context before any checkpoints existed. It always failed for hybrid models.
• Recurrent reserve size was too small. With -np 1, recurrent_rs_size = max(1, n_seq_max) = 1. Checkpointing needs up to 8 cells per sequence. We bumped it to max(16, n_seq_max * 16).

Context scaling

Prefill slows down as context grows. Decode stays weirdly constant. Here's the full sweep from 64 to 32K tokens.

Prefill speed by context

Decode speed by context

Full context scaling table

ROCm's MMQ prefill advantage fades after 16K. At that point, flash attention dominates over batched matmul. Decode stays flat across all lengths.

Raw llama-bench: combined throughput

Research trail

Profiling data, parallel subagent investigations, and the upstream issues and PRs that came out of them.

KV Cache Compression Frontier

Mission 01 tested 11 cache_type_k × cache_type_v combinations across 4 context lengths (512–32K) on Qwen3.5-122B-A10B-REAP-20-Q6_K.

Pareto-optimal sweet spots

*All asymmetric combos (q8_0/f16, f16/q8_0, q4_0/f16, f16/q4_0, q4_0/q8_0, q8_0/q4_0) either timed out at 32K or showed severe prefill degradation. Avoid mixed K/V quantization on this Vulkan backend.

Upstream contributions

What four parallel subagents found

• Memory: Vulkan's vkAllocateMemory with HOST_VISIBLE_BIT maps the full GTT correctly. ROCm's hipMalloc is boxed into BIOS VRAM unless GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 is set.
• Compiler flags: -ffast-math can give ~4% on some AMD GPUs, but it regressed in our stacked build. ROCm 7.2.1 made the old --amdgpu-unroll-threshold-local workaround unnecessary.
• Flash attention: lhl's rocWMMA FATTN tuning can improve decode at depth by 33-104%, but the patch doesn't apply cleanly to current master.
• MUL_MAT_ID: The routed expert path takes 700+ ms per batch. The most promising fix pattern is Metal's map → batched matmul → unmap approach from PR #13388.

Resources

The PRs, issues, articles, and repos that made this project possible — or that we wish we'd found sooner.

Our contributions

Prior art and reading

Models

Reproduce

Hardware setup, kernel parameters, build commands, and the exact flags we used.

Hardware and OS setup

• AMD Ryzen AI MAX+ 395 (16C/32T Zen 5)
• Radeon 8060S (gfx1151, RDNA 3.5, 40 CU)
• 128 GB LPDDR5X-8000
• Fedora 43, kernel 6.17.1

Required kernel parameters

sudo grubby --update-kernel=ALL --args='amd_iommu=off ttm.pages_limit=335544321 amdgpu.gttsize=122880'

Reboot, then verify GTT:

cat /sys/class/drm/card*/device/mem_info_gtt_total

You should see ~128849018880 (120 GiB). Without amd_iommu=off, any model over 64GB will hang during Vulkan loading.

Vulkan stock baseline

export LD_LIBRARY_PATH=$HOME/.local/opt/llama.cpp/current:$LD_LIBRARY_PATH

$HOME/.local/opt/llama.cpp/current/llama-bench \
  -m $HOME/.local/share/models/gguf/Qwen3.5-122B-A10B-REAP-20-Q6_K.gguf \
  -ngl 99 -fa 1 -c 131072 \
  -p 512,2048,8192,16384,32768,65536,131072 \
  -n 128

ROCm with PR #21344

We used a custom podman container because Fedora's HIP headers conflict with ROCm 7.2.1 runtime.

# Inside the custom container
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1

llama-bench \
  -m /models/Qwen3.5-122B-A10B-REAP-20-Q6_K.gguf \
  -ngl 99 -fa 1 -c 131072 \
  -p 512,2048,8192,16384,32768,65536,131072 \
  -n 128

Do not skip the UMA env var. Without it, ROCm sees only BIOS VRAM and performance collapses.

Speculative decoding

export LD_LIBRARY_PATH=$HOME/.local/opt/llama.cpp/current:$LD_LIBRARY_PATH

$HOME/.local/opt/llama.cpp/current/llama-server \
  -m $HOME/.local/share/models/gguf/Qwen3.5-122B-A10B-REAP-20-Q6_K.gguf \
  -md $HOME/.local/share/models/gguf/Qwen3.5-0.8B-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 99 -ngld 99 -fa 1 -c 4096 \
  --draft-max 8 --parallel 1

For Qwen3.5 and other hybrid SSM/MoE models, you'll need PR #20075 plus the four additional fixes we documented.