During the decoding phase of Mixture-of-Experts (MoE) serving, token routing demonstrates strong temporal locality: consecutive decoding steps within an MoE layer repeatedly activate overlapping or recently utilized experts. Rather than freezing expert placement with static assignments determined at startup, semantic-aware expert caching maintains a shared Least Recently Used (LRU) residency space in GPU VRAM across all layers. As generation proceeds, a cache hit refreshes the recency of an expert, a cache fill admits newly selected experts into VRAM, and eviction removes the least recently demanded expert. This dynamic caching mechanism allows scarce GPU memory to continuously adapt to the model's evolving working set during autoregressive generation.

Semantic-Aware LRU Expert Caching in MoE Decode

When decode-time cache misses occur, bandwidth-adaptive execution splits the set of $m$ missing experts $M$ into two disjoint subsets: a cache-fill set $F$ containing $q = |F|$ experts, and a CPU-execution set $C$ containing $m - q$ experts, such that $M = F \cup C$. Experts assigned to $F$ are transferred over the PCIe bus into GPU cache slots, evaluated on the GPU, and retained in VRAM for future reuse. Concurrently, experts assigned to $C$ execute in place directly from host DRAM by the CPU without modifying GPU cache residency. This concurrent partitioning utilizes the PCIe bus at full bandwidth while simultaneously using residual host memory bandwidth for CPU computation, avoiding GPU idle time and maintaining continuous cache updates.

Bandwidth-Adaptive Miss Partitioning in MoE Decode

In an edge system where direct CPU expert evaluation and direct memory access (DMA) transfers across PCIe draw from the same host DRAM subsystem, a saturated PCIe transfer leaves limited memory bandwidth for CPU processing. If $$B_H$$ denotes the measured host-side expert-processing bandwidth and $$B_P$$ denotes the pinned expert-transfer bandwidth across PCIe, the residual host bandwidth $$B_R$$ available for concurrent in-place CPU expert computation is: $$B_R = \max(B_H - B_P, 0)$$ When $$B_H > B_P$$, the CPU can execute missed experts using this residual bandwidth without throttling the saturated PCIe transfer.

Residual Host Bandwidth Formulation for CPU Expert Execution

When an active expert is not present in GPU VRAM during decoding, the system can either transfer its weights across PCIe to execute on the GPU, or compute it directly on the CPU where the weights reside in host RAM. Neither extreme is universally optimal: exclusively moving weights over PCIe leaves available host DRAM bandwidth and CPU cores idle if host memory can supply more data than the bus can move; conversely, computing entirely on the CPU leaves the PCIe interconnect idle and forfeits future cache hits that a VRAM cache fill provides. Because the balance between host memory bandwidth and PCIe transfer bandwidth varies widely across machines (such as LPDDR5 laptops versus high-speed PCIe desktops), the optimal division of miss handling cannot be statically predetermined and must be tuned to specific hardware resources.

Learn Before

Related

Learn After