Google open-sources MT5, a multilingual model trained on over 101 languages

Updated on 10/29/2020

Good news! One day in the near future, all humans on planet earth will be speaking one language, albeit automatically translated! Is this the beginning of a universal language common to all mankind? One of the holy grails of humanity!

One of the open questions here is whether a single end-to-end multilingual, cross-lingual models is better than e.g. a multilingual ensemble of models each specialized on a single language. Huge multilingual models may discover new relations between or across languages that we would otherwise perhaps miss. Exciting stuff!

Google open-sources MT5, a multilingual model trained on over 101 languages | VentureBeat (mT5 stands for multilingual Text-to-Text Transfer Transformer) Not to be outdone by Facebook and Microsoft, both of whom detailed cutting-edge machine learning language algorithms in late October, Google this week open-sourced a model called MT5 that the company claims achieves state-of-the-art results on a range of English natural processing tasks. MT5, a multilingual variant of Google’s T5 model that was pretrained on a dataset covering 101 languages, contains between 300 million and 13 billion parameters (variables internal to the model used to make predictions) and ostensibly has enough capacity to learn over 100 languages without significant “interference” effects.

Here is the link to the respective research paper:

https://arxiv.org/abs/2010.11934