.png)
January 25, 2025
The development of large language models (LLMs) is progressing rapidly. A current example is provided by ByteDance with its new model Doubao-1.5-pro. Particularly noteworthy is the efficiency of the training process, which has been significantly increased compared to previous models. Experts estimate that the training required significantly less computing power and thus also lower costs than comparable models.
A tweet by renowned AI expert Emad Mostaque provides insights into the impressive figures. Based on his calculations, which are based on publicly available information, the training of the 20 billion parameter model with 9 trillion tokens and 8-bit training at a Matrix-Multiply-Utilize rate (MFU) of 37.5% (equivalent to 750 Teraflops) required an estimated 400,000 H100-hours. This represents a significant advance compared to DeepSeek v3/R1 and amounts to total costs of under one million US dollars for training from scratch.
The increased efficiency is a crucial factor for the democratization of AI technologies. Lower training costs enable more companies and research institutions to participate in the development of LLMs and drive innovation. This opens up new possibilities for the application of AI in various fields, from text generation and translation to the development of chatbots and other AI-based applications.
Doubao-1.5-pro is based on a Mixture-of-Experts (MoE) architecture. This architecture allows the active parameters of the model to be dynamically adapted to the respective task. This allows the performance of the model to be optimized while simultaneously reducing the computational effort. The MoE architecture is considered promising for the development of even larger and more powerful language models.
In benchmarks, Doubao-1.5-pro has already achieved impressive results, surpassing models such as DeepSeek-v3, GPT-4, and Llama 3.1-405B. Particularly noteworthy is the new "Deep Thinking" mode, which shows convincing performance in the AIME benchmark. These results underscore the potential of Doubao-1.5-pro and the progress that ByteDance has made in the field of AI research.
The developments surrounding Doubao-1.5-pro show that the development of LLMs is progressing in great strides. The continuous improvement of training efficiency and the development of new architectures like MoE enable ever more powerful models. It remains exciting to observe what further innovations will emerge in this area in the future and how these will further change the application of AI in our everyday lives.
Bibliography: - https://twitter.com/EMostaque/status/1773025848855068809 - https://www.reddit.com/r/Amd/comments/jo6vwv/can_someone_explain_to_me_how_a_graphics_card/ - https://cs.stackexchange.com/questions/167711/tops-trillion-operations-per-second-to-tokens-per-second - https://www.youtube.com/watch?v=nC-bmwNguQs