Mixture of Experts

How Mixture of Experts models like Mixtral and DeepSeek-V3 work: sparse expert layers, top-k routing, load balancing, and the cost of expert parallelism.