Table of Contents
Mark Zuckerberg, the CEO of Meta, revealed in an Instagram post that Llama 3 was now being trained by Meta AI in January 2024. The Llama 1 models—originally stylized as “LLaMA”—released in February 2023 and the Llama 2 models—released in July—were preceded by this most recent iteration of the LLaMa family of large language models (LLMs).
Zuckerberg stated that Meta intends to keep the Llama foundation models open source even though specifics (such as model sizes or multimodal capabilities) have not yet been revealed.
Continue reading to find out what we currently know about Llama 3 and how it could impact the upcoming generation of generative AI model developments.
When will Llama 3 be released?
Although a release date has not been disclosed, it is noteworthy that Llama 1 required three months of training, while Llama 2 required approximately six months. If the schedule for the upcoming generation of vehicles is the same, they should be available by July 2024.
That being stated, it is possible that Meta reserves more time for optimizing and guaranteeing accurate model alignment. Expanding access to generative AI models benefits more than just businesses, startups, and enthusiasts: as open source models become more potent, greater caution is required to lower the possibility that bad actors may exploit them maliciously. Zuckerberg reaffirmed Meta’s dedication to “training [models] responsibly and safely” in his launch video.
Will Llama 3 achieve artificial general intelligence (AGI)?
The long-term objective of Meta, as highlighted in Zuckerberg’s launch video, is to create artificial general intelligence (AGI), a theoretical stage of AI research where models would exhibit a holistic intellect that is on par with or better than human intelligence.
According to Zuckerberg, “it’s become clearer that building full general intelligence is required for the next generation of services.” “Creating the best AI assistants, AIs for artists, AIs for companies, and more—requires advancements in all areas of AI, including memory, coding, reasoning, planning, and other cognitive functions.”
This does not imply that Llama 3 will eventually acquire AGI, or even try to achieve it. However, it does indicate that Meta is consciously doing their other AI research and LLM development in a way that they hope will eventually produce AGI.
Will Llama 3 be open source?
Research institutions with exclusively non commercial use cases were given free access to the Llama 1 models by Meta, but the Llama 2 code and model weights were made available under an open license that allowed any organization with fewer than 700 million monthly active users to use them for commercial purposes. Although the exact technical meaning of “open source” is debatable, Llama 2’s license is commonly referred to as such. There is no proof at this time that Llama 3 will be released in a different way.
Zuckerberg reaffirmed Meta’s dedication to open licenses and democratizing access to artificial intelligence (AI) in his announcement and the press that followed. In an interview with The Verge, Zuckerberg stated, “I tend to think that one of the bigger challenges here will be that if you build something that’s really valuable, then it ends up getting very concentrated.” On the other hand, increasing transparency solves a wide range of problems that may arise from uneven access to opportunities and value. That’s why it’s crucial to the open-source concept.
Will Llama 3 be multimodal?
Multimodal artificial intelligence (AI) refers to models that can comprehend and function across several data forms, often known as modalities. New state-of-the-art models, such as Google’s Gemini or OpenAI’s GPT-4V, and open source entrants like LLaVa (Large Language and Vision Assistant), Adept, or Qwen-VL, can move seamlessly between computer vision and natural language processing (NLP) tasks instead of developing separate models to process text, code, audio, image, or even video data.
Zuckerberg did not specifically mention any multimodal features, even though he did clarify that Llama 3, like Llama 2, will have code-generating skills. He did, however, go into detail in his Llama 3 announcement video about how he sees AI interacting with the Metaverse: Referring to Meta’s Ray-Ban smart glasses, Zuckerberg stated, “glasses are the ideal form factor for letting an AI see what you see and hear what you hear.” Thus, assistance is always ready.
This would seem to suggest that Meta has intentions to integrate visual and audio data into the text and code data that the LLMs now handle for the Llama models, either in the next Llama 3 release or in the generations that follow.
Additionally, it would appear that this is a logical step in the search for AGI. In an interview with The Verge, he stated, “You can quibble about if general intelligence is some far-future super intelligence, or is it akin to human level intelligence, or is it like human-plus.” “But the important thing, in my opinion, is how broad intelligence is—it encompasses all these different capacities where reasoning and intuition are required.”
How will Llama 3 compare to Llama 2?
Zuckerberg also disclosed large expenditures on infrastructure related to training. With the addition of the current GPUs, Meta hopes to have about 350,000 NVIDIA H100 GPUs by the end of 2024, bringing its total available compute capacity to “600,000 H100 equivalents of compute.” Right now, the only company with a comparable cache of computer power is Microsoft.
As a result, even though the Llama 3 models are identical to their predecessors, it is realistic to anticipate that they will deliver significant performance improvements over the Llama 2 models. According to a Deepmind study from March 2022, training smaller models on more data results in better performance than training larger models with fewer data, as Meta’s models and other open-source models, such as those from France-based Mistral, have shown.[iv] Although Llama 2 was pre-trained on 40% more data, it was available in the same sizes as the Llama 1 models, namely in variants with 7 billion, 14 billion, and 70 billion parameters.
Although the Llama 3 model sizes are still unknown, it is expected that they will follow the previous generations’ pattern of improving performance inside models with between seventy and seventy billion parameters. More extensive pre-training for models of any size will undoubtedly be possible because to Meta’s recent infrastructure upgrades.
Additionally, Llama 2 quadrupled Llama 1’s context length, indicating that during inference—that is, during the creation of context or throughout a continuous conversation with a chatbot—Llama 2 can “remember” twice as much information. While unsure, it’s possible that Llama 3 will bring more advancements in this area.
Comparing Llama 3 with OpenAI’s GPT-4: What Are the Differences?
Although the performance of the bigger, 175 billion parameter GPT-3 model was matched or surpassed by the smaller LLaMA and Llama 2 models on some benchmarks, they fell short of the full potential of the GPT-3.5 and GPT-4 models available in ChatGPT.
With their next generation of models, Meta appears determined to provide open source with cutting edge performance. He told The Verge, “Llama 2 wasn’t the best open-source model; it wasn’t an industry-leading model.” “Our goal is to create products that are cutting edge and eventually the top models in the industry starting with Llama 3 and beyond.”
Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me?