University of Michigan - Ann Arbor

To streamline heterogeneous execution across hardware components, model-specific Mixture-of-Experts checkpoint layouts are normalized into a unified set of expert banks. Each bank organizes parameters using a flattened layer–expert index $lE + e$ as its leading dimension, where $l$ is the layer index, $e$ is the expert index within that layer, and $E$ is the total number of routed experts per layer. Rows across all banks sharing the same flattened index collectively constitute one full expert. This uniform organization guarantees that both device-side GPU kernels and host-side CPU worker threads reference a shared logical expert identity, irrespective of the underlying tensor representations.

Normalized Expert Bank Representation

In a normalized expert bank representation, which formula defines the leading dimension index for an expert with index $e$ in layer $l$, given $E$ routed experts per layer?

The FreeToken Weight (FTW) storage format is a pre-processed weight file format that stores expert parameters pre-merged into the normalized runtime bank layout ahead of time. By preparing the weights in advance according to the engine's exact runtime memory arrangement, the system completely bypasses tensor discovery, unpacking, and runtime weight repacking during startup. At engine launch, expert weights are loaded via parallel direct I/O as aligned chunks directly into pre-allocated, exact-size host banks, allowing rapid cold bootstrap prior to memory pinning.

Learn Before

Related