100,000 NVIDIA GPUs: Musk's "Great Wall of Chips" in the AI World
Compared to the 25,000 GPUs used by OpenAI to train GPT-4, this number represents a 4-fold increase.
Notably, this world's largest supercomputing center only began construction in March this year, with completion planned for 2025. Surprisingly, it was finished in just four months, well ahead of schedule.
The official launch of the supercomputing center will signal Musk's counteroffensive against OpenAI.
The Costly Supercomputing Center
The fundamental reason for Musk building the supercomputing center is to train xAI's chatbot, Grok.
Musk founded xAI in July 2023, with core founding members from renowned companies like OpenAI, Google DeepMind, Microsoft Research, and Tesla. To compete directly with ChatGPT, Musk has devoted most of his energy to xAI over the past two years.
To ensure the realization of the supercomputing center, Musk publicly claimed he would personally oversee the development of this supercomputer.
Choosing the location was undoubtedly key to building the supercomputing center.
According to Forbes, Musk's team negotiated with seven or eight cities before finalizing Memphis, Tennessee in March, as Memphis could provide sufficient power and rapid construction capabilities.
It's worth noting that the supercomputing center requires extensive resources. Besides basic hardware like NVIDIA chips, the factory may use up to 150 megawatts of electricity per hour, equivalent to the power needed by 100,000 households, which could be a huge burden for the local area.
In addition to power resources, the supercomputing center is expected to need 1.3 million gallons of water daily for cooling, which means drawing from Memphis's main water source every day.
This has sparked dissatisfaction among Memphis residents and environmental organizations, who jointly issued an open letter protesting: "We must consider how an industry using such vast amounts of energy will further impact communities already burdened by pollution and high energy costs."
To complete the project, xAI promised to improve Memphis's public infrastructure to support the supercomputing center's development, including building a new substation and a wastewater treatment facility.
The cost of the supercomputing center is astronomical. Estimating the cost, 100,000 NVIDIA H100 GPUs at $30,000-$40,000 each would total $4 billion (over 29 billion RMB).
Even for Musk, the "world's richest person," this massive financial investment is hard to swallow. So he partnered with Dell and Supermicro to build the cluster and obtained resources for 24,000 H100 GPUs from Oracle.
Despite the enormous cost, Musk remains steadfast in pushing forward with the supercomputing center's construction. He summarized it in one sentence on social media:
"We must take the wheel ourselves."
xAI: Valued at Over $130 Billion
Undeniably, Musk has given his all to make xAI catch up with OpenAI.
Under the halo of the world's richest man, xAI, founded in July 2023, has just turned one year old but already has the capital and confidence to challenge OpenAI.
On April 13 this year, xAI launched its first multimodal large model, Grok-1.5V. In addition to text functionality, Grok can now process various visual information, including documents, charts, graphs, screenshots, photos, and perform multidisciplinary reasoning.
With the birth of Grok-1.5V, xAI has become hot in the capital market.
On May 26, xAI announced the completion of $6 billion in funding, valuing it at $24 billion (about 173.8 billion RMB). The investors in this round of financing are formidable, including Silicon Valley's renowned venture capital firms Sequoia Capital and Andreessen Horowitz, as well as an investment fund closely linked to the Saudi royal family.
OpenAI has raised a total of $14.3 billion in funding over the past decade, while xAI raised $6 billion in just 10 months, equivalent to half of OpenAI's decade-long funding.
This is also one of the highest single financings in the history of the large language model sector, surpassing Anthropic's $4 billion funding on September 12, 2023, and $2 billion on October 28, 2023, as well as InflectionAI's $1.3 billion funding on June 30, 2023. The total funding amount is second only to OpenAI and Anthropic.
xAI's ability to secure the highest funding in AI history is naturally inseparable from Musk's halo. Especially as other companies in Musk's business empire have helped xAI considerably.
According to media reports, when xAI was first established, Tesla dispatched about 50 software engineers to assist with tasks such as code review. Some Tesla engineers even directly joined xAI. At the same time, the vast amount of video data captured by cameras and other sensors equipped in Tesla cars can provide real-time data input for xAI's models, helping them receive training.
Musk's well-known social platform X can also provide massive human-generated data for xAI. This data is an ideal resource for training AI systems on how to better interact with humans.
The combination of these advantages has given birth to an AI unicorn valued at $24 billion in just one year, capable of competing with OpenAI in terms of funding, data, and chips.
On July 1, Musk publicly stated on his social platform that xAI's next step is to remove large language models from internet-trained data, with plans to launch the next-generation upgraded version, Grok-2, in August.
We believe that next month, we'll see Grok-2 battling GPT-4.
The Talent War in the AI World
Besides funding and chips, another crucial aspect of competition between AI large models is talent.
Musk once lamented on his social platform: The AI talent war is the craziest talent battle he has ever seen.
As a "famous internet celebrity" worldwide, Musk's every move can influence the direction of an industry. This influence and fame also provide convenience for Musk to recruit professional talent.
According to media reports, Musk has poached numerous researchers from tech giants like Google and Microsoft. Executives from other companies complain that the compensation packages offered by xAI are highly competitive, putting them at a disadvantage in the recruitment market.
At the same time, as one of the main co-founders of OpenAI and a long-term board member from 2015-2018, Musk not only contributed tens of millions of dollars in funding and provided suggestions on research directions but also played a crucial role in recruiting world-class talent.
Although he left OpenAI due to differences in development philosophy, he is undoubtedly very familiar with OpenAI's leadership.
When founding xAI, Musk poached data scientists Kyle Kosic and Igor Babuschkin from OpenAI with high salaries.
Even with the supercomputing center officially in use, Musk hasn't stopped recruiting talent.
On July 23, xAI launched a global recruitment drive, offering nearly 30 positions including AI engineers and various managerial roles. In addition to competitive salaries and equity incentives, xAI's benefits are quite attractive, even offering unlimited paid leave with approval.
Even though the AI year has passed for over a year, the AI arms race continues to heat up.
Even Musk continues to double down on xAI until it surpasses OpenAI.