On July 24th in the early morning, Meta released Llama 3.1, currently the most powerful open-source large language model. At the same time, Zuckerberg was interviewed by Rowan Cheung, founder of "The Rundown AI", to discuss Llama 3.1 and his views on AI development.
Zuckerberg believes that in the long run, the ecosystem created by open-source models is safer than closed models. He predicts that AI agents will eventually become essential tools for businesses, just like email and social media accounts. He stated: "Eventually, there might even be more AI agents than humans, and people will interact with them in various ways. Obviously, this is also a huge business opportunity."
Here are the main points of the interview:
Rowan Cheung: Mark, thank you very much for doing this interview. Meta released a major AI model today, can you outline what was released and its significance?
Mark Zuckerberg: Today we released Llama 3.1, and we released three models. This is the first time we've released a 405 billion parameter model. It's the most complex open-source model to date. It's competitive with some of the leading closed models, and in some areas, it even outperforms closed models.
I'm really excited to see how people will use it, especially now that our community policy allows people to use Llama as a teacher model for distillation and fine-tuning, basically allowing them to create any other model with it.
In addition, we distilled the 405 billion parameter model into newer, more advanced 70 billion and 8 billion parameter models. These models also perform very well and are cost-effective. I'm looking forward to seeing how everyone uses these models.
I think this is an important moment for open-source AI. I've thought about this for quite a while, and I believe open-source AI will become the industry standard, following the development path of Linux.
Before Linux became popular, many companies wanted their own closed versions of Unix, and there were no complex open-source projects at the time. People thought that closed development was the only way to create complex products.
Initially, Linux gained a foothold based on being cheaper and customizable for developers. As the ecosystem developed, it had more scrutiny, became safer and more advanced. More partners built more features on top of Linux, eventually making it as usable as other closed Unix systems.
Now, I believe Llama 3.1 has the opportunity to become the open-source AI standard, making open-source the industry standard for AI. Even if it hasn't surpassed closed models in performance yet, it has significant advantages in cost and customizability. I think these are advantages that developers will leverage.
We're focusing on building a partner ecosystem around it, and we'll see many different features being built on top of it.
Rowan Cheung: I've seen all the benchmarks, and the results are incredible. Obviously, this is the first 405 billion parameter open-source frontier model. Are there any specific practical use cases that you're particularly excited to see people build with this model?
Mark Zuckerberg: What I'm most excited to see is people using it to distill and fine-tune their own models. As you said, this is the first open-source frontier-level model, but it's not the first frontier-level model. There have been other models with this capability before.
People will want to do inference directly on the 405 billion parameter model because we estimate it's 50% cheaper than GPT-4. This can make some impact for many people.
However, I think what's really novel about this model is that it's open-source weights, allowing you to distill it to any size you want, use it for synthetic data generation, and use it as a teacher model.
For the future, we don't think it will belong to a single entity. Like, OpenAI's vision is to build a "big AI," and Anthropic and Google have similar visions.
But that was never our vision. Our vision is that there should be many different models in the future. I think every startup, large company, every government will want to have their own custom model.
When closed systems were much better than open-source, it was indeed more convenient to use closed models. Although open-source models could be customized, there was still a performance gap.
Now it's different. The performance gap of open-source models has basically caught up. You'll see more people motivated to customize and build models that fit their needs, train with their own data, and are the right scale for them.
They'll also have tools to do this because companies like Amazon's AWS and Databricks are building complete service suites for distilling and fine-tuning open-source models.
In my view, this is the new situation now. We're excited to see how far this trend can be pushed. This is a new capability in the world that has never existed before because no open-source or open-source weight model has ever reached this level of complexity before.
Rowan Cheung: This is indeed a big deal. How will you educate developers to use these tools? More broadly, does Meta have plans or strategies to educate the world about open-source models and their importance?
Mark Zuckerberg: Before Llama 3.1, the fundamental reason Meta invested in this area was to ensure we could use leading models ourselves. Due to our history, especially on mobile, we didn't want to rely on a competitor's foundational technology. So we built models for ourselves.
Before Llama 3.1, we instinctively believed that if we open-sourced it, it would attract a community to grow around it, expand its functionality, and make it more valuable for everyone, including ourselves. Because ultimately, it's not just a technology, it's an ecosystem. To make it more useful for us, it also needs a broad ecosystem.
A big change with Llama 3.1 is that we're no longer just building for ourselves and throwing it out for developers to use, but more actively building partnerships to ensure there's a whole ecosystem of companies that can do interesting things with this model and serve developers in ways we can't.
We're not a cloud service provider, we're not AWS, Google, or Azure, so developers won't come to us to build their stuff, but we want to make sure all these cloud service providers can use this model well.
This involves not just hosting and inference, but also some new features like distillation and fine-tuning, which aren't as easy to do with closed models, so we have to work specifically with partners to enable these features.
At the same time, there will be companies like Groq that focus on ultra-low latency inference. I'm happy to hand it over to Groq, who are now using it to build new things.
There's also a series of enterprises like Dell, Scale AI, Deloitte, or Accenture that work with global enterprises on technology deployment. I think these companies will help build custom models.
Whether it's large companies or governments, many companies want to have their own models and be able to train their own data. Many companies are unwilling to pass data to Google or OpenAI through APIs, not because these companies have privacy issues.
It's more like how people love to use WhatsApp's end-to-end encryption, they want it to be secure by design and keep their data in their own hands.
I think there will be a market built around this as well. I'm very excited about this.
This time we've taken a more proactive approach in building the ecosystem because I think this is how it grows and creates more value for everyone.
Rowan Cheung: I love your close connection with the developer community. I'm part of the community myself and know that people really need these private and local models. Let's talk about your open letter next. In addition to Meta's announcement, you also published a letter, the first part of which focused on the benefits of open-source for developers, which felt very accurate. Can you talk more about the broader impact of open-source AI on society?
Mark Zuckerberg: My view is that open-source is an important factor in achieving a good AI future. AI will bring many improvements in productivity and creativity, and hopefully help us do research and so on.
I think open-source is an important part of ensuring that AI can benefit everyone and be accessible to all, rather than locking AI in the hands of a few large companies.
At the same time, I believe open-source will be a safer and more reliable way to develop AI.
There's now a debate about the safety of open-source - "Is open-source really safe?"
My view is different. I think open-source is not only safe but safer than closed development.
We can divide risks into "unintentional" and "intentional". "Unintentional" risks are when systems go out of control in some way, which is also the scenario of AI going out of control in most science fiction.
I think open-source is safer in this regard because there's more scrutiny and transparency. When all developers use it, there are also safety guidelines and safety tools, as well as a lot of scrutiny and testing pressure, just like traditional open-source software. Compared to closed models, problems will be discovered and solved faster.