Zuckerberg Highlights Meta’s Requirement for Tenfold Increase in Computing Power to Train Llama 4 Compared to Llama 3

Meta, the creator of the widely recognized open-source large language model Llama, foresees a need for a substantial increase in computational resources for future model training.

During the earnings call for the second quarter, Mark Zuckerberg highlighted that the computational demand for developing Llama 4 will be tenfold that of its predecessor, Llama 3. Yet, he emphasizes the importance of expanding Meta’s model training capabilities to stay ahead in the competitive landscape.

“Training Llama 4 is expected to require nearly ten times the computing power we invested in Llama 3, and we anticipate even greater needs for subsequent models,” Zuckerberg explained.

“Predicting future trends in computational requirements is challenging. However, I prefer to prepare by expanding our capacity early on rather than risk falling behind due to the prolonged duration it takes to launch new inference initiatives,” Zuckerberg added.

Meta announced the introduction of Llama 3, equipped with 80 billion parameters, in April. Following this, the company unveiled an enhanced version, Llama 3.1 405B, sporting 405 billion parameters, marking it as Meta’s most ambitious open-source project to date.

Susan Li, Meta’s CFO, discussed the company’s contemplation on various data center initiatives and the expansion of capacities to accommodate the training of forthcoming AI models. She indicated an expected rise in capital expenditure by 2025 as a result of these investments.

The financial implications of developing large language models are significant. Meta’s investments surged by nearly 33%, reaching $8.5 billion in Q2 2024 from $6.4 billion the previous year, attributed to enhancements in servers, data centers, and network infrastructures.

A report by The Information indicates OpenAI’s substantial expenditures on model training and server rentals, amounting to $7 billion in total.

“Our approach to expanding generative AI training involves creating an adaptable infrastructure. This versatility will enable us to allocate resources efficiently between gen AI inference and our fundamental ranking and recommendation tasks, depending on which offers more value at the time,” Li mentioned during the discussion.

In the same discussion, Meta shed light on the user engagement with its Meta AI chatbot, noting India as its prime market. Nevertheless, Li remarked that generative AI products are not anticipated to significantly affect the company’s revenue streams.