University of Michigan - Ann Arbor

In an edge system where direct CPU expert evaluation and direct memory access (DMA) transfers across PCIe draw from the same host DRAM subsystem, a saturated PCIe transfer leaves limited memory bandwidth for CPU processing. If $$B_H$$ denotes the measured host-side expert-processing bandwidth and $$B_P$$ denotes the pinned expert-transfer bandwidth across PCIe, the residual host bandwidth $$B_R$$ available for concurrent in-place CPU expert computation is: $$B_R = \max(B_H - B_P, 0)$$ When $$B_H > B_P$$, the CPU can execute missed experts using this residual bandwidth without throttling the saturated PCIe transfer.

Residual Host Bandwidth Formulation for CPU Expert Execution

Calculate the residual host bandwidth (B_R) available for concurrent in-place CPU computation, and explain whether the CPU can execute missed experts concurrently without contending with or throttling the PCIe bus.

For $$m$$ missing experts of size $$S$$, let $$q$$ experts use the concurrent PCIe cache-fill branch and $$m-q$$ execute on the CPU. When $$B_H>B_P$$, their approximate execution times are $$T_{\text{fill}}(q)\approx\frac{qS}{B_P}$$ and $$T_{\text{cpu}}(m-q)\approx\frac{(m-q)S}{B_H-B_P}$$. Balancing the branches gives $$\frac{q}{m-q}\approx\frac{B_P}{B_H-B_P}$$ and therefore $$q^*\approx m\frac{B_P}{B_H}$$. The runtime rounds $$q^*$$ to an integer and retains at least one cache fill so that the cache continues warming. As $$B_H$$ approaches $$B_P$$ from above, $$q^*$$ approaches $$m$$, reducing the policy to pure on-demand cache fill.

Learn Before

Related