A couple indications of just how fast Mistral is coming along.
From Turing Point, January 5, 2024:Mistral AI's Bold Journey
From Paris to Global Stage: The Unconventional Rise of a French AI Unicorn and Its Open-Source Revolution
This French startup, founded in April 2023 with the ambitious goal of challenging the European Union's technological supremacy, has earned both admiration and skepticism. What sets Mistral AI apart is its focus on open-source technology and its bold approach, unapologetically offering models devoid of safety controls. According to a list of 178 questions and answers composed by AI safety researcher Paul Röttger and 404 Media’s testing, Mistral AI’s models have been churning out some rather dicey advice. The content generated by Mistral AI's models has ignited debates on morality, spanning topics from ethnic cleansing to retrograde discrimination, even venturing into unsettling DIY territory.....MORE
In December 2023, only 7 months after their launch, Mistral AI ripped all the charts, becoming a GenAI unicorn with a valuation exceeding $2 billion. They also unconventionally launched an open-sourced model, Mixtral 8x7B, based on the sparse mixture-of-experts technique, via a torrent link! Who are these bold French innovators, what drives them, why the Mixtral model is so efficient, who supports them, and why? Let’s find out.The founders' (or France's?) vision
- Starting point of Mistral AI
- The founders' (or France's?) vision
- Founder’s views toward AI risks
- Financial situation
- It took them four months to roll out their first LLM
- Next step: Mixtral – understanding SMoE architecture and what makes that model so efficient
- How does Mistral make money?
- Conclusion
Recently, prominent French AI leaders such as Yann LeCun of Meta and Clément Delangue of Hugging Face have been actively promoting French tech achievements on Twitter. This effort culminated in a partnership between Meta, Hugging Face, and Scaleway at Paris's Station F, signaling a shift in the global tech landscape. France, with its academic excellence and government support, aims to emerge as a potential open-source AI capital....
And from InfoQ, Jan 23, 2024:
Mistral AI's Open-Source Mixtral 8x7B Outperforms GPT-3.5
Mistral AI recently released Mixtral 8x7B, a sparse mixture of experts (SMoE) large language model (LLM). The model contains 46.7B total parameters, but performs inference at the same speed and cost as models one-third that size. On several LLM benchmarks, it outperformed both Llama 2 70B and GPT-3.5, the model powering ChatGPT.
Mistral 8x7B has a context length of 32k tokens and can accept the Spanish, French, Italian, German, and English language. Besides the base Mixtral 8x7B model, Mistral AI also released a model called Mixtral 8x7B Instruct, which is fine-tuned for instruction-following using direct preference optimisation (DPO). Both models' weights are released under the Apache 2.0 license. Mistral AI also added support for the model to the vLLM open-source project. According to Mistral AI:
Mistral AI continues its mission to deliver the best open models to the developer community. Moving forward in AI requires taking new technological turns beyond reusing well-known architectures and training paradigms. Most importantly, it requires making the community benefit from original models to foster new inventions and usages.
Mixture of Experts (MoE) models are often used in LLMs as a way to increase model size while keeping training and inference time low. The idea dates back to 1991, and Google applied it to Transformer-based LLMs in 2021. In 2022, InfoQ covered Google's Image-Text MoE model LIMoE, which outperformed CLIP. Later that year, InfoQ also covered Meta's NLB-200 MoE translation model, which can translate between any of over 200 languages....
....MUCH MORE
As we said in the outro from February 18's "Venture Capital: "These 12 startups could be France’s next unicorns":
We'll have more on Mistral next week. Previously:
French Tech: "Mistral AI secures €105M in Europe’s largest-ever seed round"
"...Big Tech Alumni building AI startups in Paris"
"France's unicorn start-up Mistral AI embodies its artificial intelligence hopes"
French OpenAI Rival Mistral Nears $2 Billion Valuation With Andreessen Horowitz Backing
This. This is what we've been pitching for the last five years as the way France picks up the economic torch from Germany....
AI to See ‘Major Second Wave,’ NVIDIA CEO Says in Fireside Chat With iliad Group Exec
NVIDIA’s Jensen Huang says sovereign AI a growing need for countries to reflect unique cultural, linguistic, industrial characteristics
European startups will get a massive boost from a new generation of AI infrastructure, NVIDIA founder and CEO Jensen Huang said Friday in a fireside chat with iliad Group Deputy CEO Aude Durand — and it’s coming just in time....
