University of Michigan - Ann Arbor

In edge MoE serving, prefill double buffers and the decode expert cache are allocated from a single shared global slot pool rather than isolated memory partitions. Because both execution phases draw from the same residency pool, the serving engine eliminates the overhead of a dedicated prefill cache and avoids explicit phase-handoff reallocations. Moreover, expert weights already loaded in VRAM that persist across prefill automatically remain in the shared pool, effectively seeding the GPU expert cache for the subsequent latency-sensitive decode phase.

Unified Slot Pool and Decode Seeding in Edge MoE Serving

Match each memory management concept in edge MoE serving to its correct description:

Based on the memory management principles for edge MoE serving, explain how transitioning to a single shared global slot pool remedies these phase-transition latency bottlenecks.

Learn Before

Related