DeepSeek introduced novel techniques to efficiently train models with a mixture of experts architecture, stabilizing performance and increasing GPU utilization.

DeepSeek's implementation of mixture of experts innovates by changing the routing mechanism, allowing for a high sparsity factor that activates only a small fraction of the model's parameters while ensuring all experts are utilized during training to improve model performance.

Running a sparse mixture of experts model presents complexities, such as load balancing and scheduling communications between idle experts, which DeepSeek has addressed effectively.

DeepSeek's training efficiency stems from two main techniques: a mixture of experts model and a new method called MLA latent attention, which reduces both training and inference costs.

DeepSeek's recent models, including V3, highlight the importance of balancing training and inference compute resources to optimize performance and efficiency.

DeepSeek's V3 model utilizes a mixture of experts architecture, activating only 37 billion out of 671 billion parameters for each token prediction, significantly reducing computation compared to models like Llama 3, which activates all 405 billion parameters.

A crucial enhancement in DeepSeek's models is the fp8 accumulation fix, which helps prevent small numerical errors from compounding during calculations, leading to more efficient training across thousands of GPUs. Additionally, DeepSeek needed to optimize GPU usage due to hardware constraints and export controls on GPU sales to China, as their GPUs were often idle, achieving only about 35% utilization.

DeepSeek has historically leveraged its hedge fund resources to build significant GPU clusters, including a claim of having the largest cluster in China with 10,000 A100 GPUs in 2021.

DeepSeek's model architecture innovations, such as multi-head latent attention, dramatically reduce memory pressure, making their models more efficient.