Least-Loaded Expert Parallelism: Load Balancing an Imbalanced Mixture-of-Experts
Published in ICML, 2026
Recommended citation: Xuan-Phi Nguyen, Shrey Pandit, Austin Xu, Caiming Xiong, Shafiq Joty (2026). Least-Loaded Expert Parallelism: Load Balancing an Imbalanced Mixture-of-Experts. ICML 2026.
Paper Link: https://arxiv.org/abs/2601.17111
Abstract
This work introduces an expert-parallelism strategy that addresses load imbalance in Mixture-of-Experts training.