SeamlessM4T: Meta’s New Multimodal Translation Model

Meta has announced a new machine translation model called SeamlessM4T.

Meta, formerly known as Facebook, has unveiled SeamlessM4T, a multimodal and multilingual AI translation model that can translate speech and text in up to 100 languages. The model is trained on a massive dataset of text and audio, and it can perform text-to-speech, speech-to-text, speech-to-speech, and text-to-text translations.

SeamlessM4T is a significant breakthrough in the field of machine translation. It is the first model that can perform all four types of translation in a single go, without the need for separate systems. This makes it more efficient and accurate than previous models.

SeamlessM4T is also able to handle code-switching, which is when a speaker switches between two or more languages in the same conversation. This is a common occurrence in many multilingual communities, and it can be difficult for traditional machine translation models to handle.

Meta is still working on improving SeamlessM4T, but the model has the potential to revolutionize the way we communicate across languages. It could be used in a variety of applications, such as online translation tools, virtual assistants, and real-time translation devices.

Machine Translation Capabilities of SeamlessM4T 

Language Support 

The system can translate between over 100 different natural languages. This wide coverage allows it to accommodate any language pairing that may be required. Both widely used languages as well as less common ones are represented in its vocabulary. 


In addition to basic text translation, SeamlessM4T includes multimodal functionality. It can convert text to synthesized speech or user-generated speech to text through speech recognition. It also enables direct translation of spoken utterances. Finally, it performs standard text-to-text language conversion. This array of modalities expands the system’s potential uses. 

Training Data 

SeamlessM4T was trained on an exceptionally large composite dataset containing both written and recorded language samples in various languages. Access to such extensive material during development contributed to its overall accuracy. The large training size sets it apart from other translation tools with more limited training corpora. 


For languages that commonly combine elements, like Spanish and English, the system is designed to correctly process code-switching patterns within a single input. This indicates an advanced understanding of natural mixed-language discourse phenomena. 


Meta is making SeamlessM4T available to researchers and developers under the CC BY-NC 4.0 license. This means that anyone can use the model for research or development purposes, as long as they credit Meta and do not use it for commercial purposes.

Photo of author

Cray Zephyr

Cray has a major in philosophy and likes to keep things simple. He tries to keep his opinions to himself but will never shy out of a discussion, except with chickens. A chicken always wins.