Showing posts with label natural language processing. Show all posts
Showing posts with label natural language processing. Show all posts

Saturday, July 25, 2026

Language diversity: A historical context for language loss for the past 6,000 years or more

Over human history, many languages have disappeared and continue to disappear!

Why should the evolution of human languages be so different than the biological evolution? Do we need about 7,000 existing and actively spoken languages on our planet?

Hopefully, the Tower of Babel will go away in the 21st century. The world would be a much better place if we all speak the same language (with or without machine translation)!

Is this article another way of trying to indoctrinate with the DEI (Diversity, Equity, and Inclusion) ideology?

The approach taken in this study, i.e. a kind of extrapolation from present-day hunter-gatherers, is certainly questionable if not junk! This assumption is possibly a very strong inductive bias! What if e.g. the first humans spoke only one or a few languages and that for a long time before language diversity emerged maybe with the geographic spread of humans?

Notice that this study does not blame colonialism, but the preceding multinational empire building and expansion for the loss of languages.

"Humanity’s linguistic diversity is in trouble [???]. Roughly half of the 7500 languages signed or spoken today are endangered [???], and at least four go extinct every year. But while preserving the languages that do persist is a top priority, a new study in Science suggests that we are already more than one millennium beyond humanity’s linguistic “golden age.”

Linguists have long debated when the most languages were in use: Some have argued that the total number of languages remained roughly steady from the last ice age until modern-day European colonialism, others suggested that diversity has steadily declined since the dawn of agriculture about 12,000 years ago, and a third camp theorized a high point sometime before large states became established 4000 years ago.
Since writing was not developed until around 6000 years ago, however, understanding dynamics of early language remained challenging.
So, a team of anthropologists applied data on the languages, distributions, and behaviors of present-day hunter-gatherers to estimates of the total human population size and the size and number of individual cultures over the last 12,000 years. This allowed them to create a model of the total number of languages in use at any point since the last ice age ended.

The researchers showed that linguistic diversity rose steadily for around 10,000 years, reaching a “golden age” of tens of thousands of languages around 1000 to 3000 years ago before entering a language loss free-fall that continues to this day. The results reveal that the language crisis is not the result of modern colonialism but rather the culmination of millennia of cultural exchange and homogenization."

"... The golden age was followed by a period of rapid decline in linguistic diversity that coincided with the rise of large states and multinational empires, such as the Roman Empire, the researchers found. The finding challenges a commonly held view that widespread language extinction began later, about 500 years ago, with the onset of European colonial expansion. ..."

"To the Point
  • Fewer languages before agriculture: The researchers estimate that between approximately 4,500 and 6,200 languages were spoken at the beginning of the Holocene, around 12,000 years ago - probably fewer than the roughly 7,500 languages spoken and signed today.
  • A linguistic “golden age”: As the global human population grew, language diversity also increased. The models suggest that tens of thousands of languages may have existed between 1,000 and 3,000 years ago.
  • A deep history of language loss: The decline in linguistic diversity appears to have begun with the expansion of large states and empires, long before modern European colonialism.
  • Today’s languages are survivors [???]: The languages spoken today represent a small and historically biased sample of past linguistic diversity [???], with important implications for how researchers explain global patterns in language and culture.
...
They began with ethnographic data from 171 hunter-gatherer and fisher societies whose traditional subsistence and mobility had not been profoundly transformed by contact with food-producing populations. These data allowed the researchers to estimate the likely distribution of ethnolinguistic group sizes near the beginning of the Holocene.

The team then combined these estimates with reconstructions suggesting that the global human population 12,000 years ago was between approximately 4.4 and seven million. Assuming that ethnolinguistic groups at that time generally corresponded to distinct languages, the models produced an early-Holocene estimate centred on roughly 4,500 to 6,200 languages, although broader plausible estimates ranged from around 3,300 to 7,800. ..."


From the editor's summary and abstract:
"Editor’s summary
More than four languages are lost every year, and this rate is predicted to increase. Blasi et al. investigated how language diversity has changed over the Holocene using a Bayesian modeling approach, ethnographic data, and estimates of the total human population size over time.
They assumed a maximum size of ethnolinguistic groups and that the number of different ethnolinguistic groups is related to the number of languages. Their findings suggested that language loss is not a new phenomenon, and that language diversity has been decreasing rapidly over the past one to three millennia following a peak of an order of magnitude higher diversity than we see today. ...

Abstract
Characterizing the factors that have shaped linguistic diversity is fundamental for understanding human history, culture, and cognition.
In this study, we combined statistical and social computational modeling, ethnographic data, and paleodemographic inference to model trajectories of global linguistic diversity.
Before the onset of plant and animal domestication, the number of languages was smaller than it is today (4500 to 6000 compared with 7500).
Subsequent increases in global population precipitated increased linguistic diversity.
We uncovered a linguistic “golden age” with tens of thousands of languages 3000 to 1000 years ago. Great loss of linguistic diversity did not begin with recent colonial expansion but as multinational empires first spread along with their languages, pathogens, and cultures. Thus, extinction has likely played a much greater role in shaping linguistic and cultural diversity than previously thought."

ScienceAdviser


Study uncovers lost ‘golden age’ of languages (original news release) "A new study coauthored by Yale linguist Claire Bowern suggests that tens of thousands of languages were spoken between 1,000 and 3,000 years ago."






Map of currently spoken and signed languages of the world, colored by endangerment status.


Sunday, June 15, 2025

On Improving large language models with concept-aware fine-tuning. Really!

The abstract of this new paper suggests that ML & AI researchers are reinventing the wheel!

The long existing ambiguity about what tokens are in natural language processing is not helpful! Breaking up words into chunks may even be counterproductive! What unit in the spectrum between a single character, a word chunk, a whole word, a sentence or even a whole paragraph etc. should be used for the training of language models? A combination of fine-grained and coarse-grained units or hierarchical-level units etc.

Caveat: I have not read the paper.

From the abstract:
"Large language models (LLMs) have become the cornerstone of modern AI. However, the existing paradigm of next-token prediction fundamentally limits their ability to form coherent, high-level concepts, making it a critical barrier to human-like understanding and reasoning.
Take the phrase "ribonucleic acid" as an example: an LLM will first decompose it into tokens, i.e., artificial text fragments ("rib", "on", ...), then learn each token sequentially, rather than grasping the phrase as a unified, coherent semantic entity. This fragmented representation hinders deeper conceptual understanding and, ultimately, the development of truly intelligent systems.
In response, we introduce Concept-Aware Fine-Tuning (CAFT), a novel multi-token training method that redefines how LLMs are fine-tuned. By enabling the learning of sequences that span multiple tokens, this method fosters stronger concept-aware learning.
Our experiments demonstrate significant improvements compared to conventional next-token finetuning methods across diverse tasks, including traditional applications like text summarization and domain-specific ones like de novo protein design.
Multi-token prediction was previously only possible in the prohibitively expensive pretraining phase; CAFT, to our knowledge, is the first to bring the multi-token setting to the post-training phase, thus effectively democratizing its benefits for the broader community of practitioners and researchers. ..."

[2506.07833] Improving large language models with concept-aware fine-tuning

Friday, December 23, 2022

Amazon CodeWhisperer added two more programming languages

Good news! Apparently CodeWhisperer supports now a total of 5 programming languages.I have not yet tried any of these new machine learning base code support generators.

There is even a preprint study available for this product titled "Multi-lingual Evaluation of Code Generation Models"! This study even covers 10+ languages.

"... Amazon CodeWhisperer is a new cloud-based capability provided by Amazon Web Services that uses machine learning and large language models to make developers’ lives easier and boost their productivity.
CodeWhisperer works within a developer’s primary workspace, known as an integrated development environment (IDE). As developers build their code, they typically leave notes or comments in natural language describing, for example, the purpose of the next block of code or, indeed, the overall purpose of the program. The system looks at not only the code already produced in the IDE but also the developer’s comments and then, in real time, suggests what it predicts would be a useful next chunk of code. ... "

AWS CodeWhisperer creates computer code from natural language - Amazon Science At re:Invent, AWS announces that the CodeWhisperer preview has added support for two new programming languages.

Tuesday, February 09, 2021

Microsoft details Speller100, an AI system that checks spelling in over 100 languages

Recommendable! Very impressive work by Microsoft! The Tower of Babel no more! It is extremely or critcally  important to correct misspelled search entries. However, this is only one side of the search process. What about misspelled search targets?

"... We believe Speller100 is the most comprehensive spelling correction system ever made in terms of language coverage and accuracy. With this technology, we have improved the search results for all Bing users by expanding accurate spelling correction to over 100 languages. We have observed a double-digit improvement in both spelling correction precision and recall. ...
The number of pages with no results reduced by up to 30%. ...
In search we’ve found about 15% of queries submitted by customers have misspellings. When queries are misspelled, we match the wrong set of documents and trigger incorrect answers, which can produce a suboptimal results page for our customers. Therefore, spelling correction is the very first component in the Bing search stack because searching for the correct spelling of what users mean improves all downstream search components. ...
"

To get an idea how challenging this can be just look at the below example provided for the Germanic languages:

Below is an illustration of orthographic similarities between languages in the Germanic languages.

EnglishDutchAfrikaansGermanLuxembourgish
twotweetweezweizwee
bloodbloedbloedBlutBlutt
fingervingervingerFingerFanger
downloaddownloadenaflaaiherunterladeneroflueden



Microsoft details Speller100, an AI system that checks spelling in over 100 languages | VentureBeat

Here is the underlying Microsoft blog post:

Tuesday, October 27, 2020

Google open-sources MT5, a multilingual model trained on over 101 languages

Updated on 10/29/2020

Good news! One day in the near future, all humans on planet earth will be speaking one language, albeit automatically translated! Is this the beginning of a universal language common to all mankind? One of the holy grails of humanity!

One of the open questions here is whether a single end-to-end multilingual, cross-lingual models is better than e.g. a multilingual ensemble of models each specialized on a single language. Huge multilingual models may discover new relations between or across languages that we would otherwise perhaps miss. Exciting stuff!

Google open-sources MT5, a multilingual model trained on over 101 languages | VentureBeat (mT5 stands for multilingual Text-to-Text Transfer Transformer) Not to be outdone by Facebook and Microsoft, both of whom detailed cutting-edge machine learning language algorithms in late October, Google this week open-sourced a model called MT5 that the company claims achieves state-of-the-art results on a range of English natural processing tasks. MT5, a multilingual variant of Google’s T5 model that was pretrained on a dataset covering 101 languages, contains between 300 million and 13 billion parameters (variables internal to the model used to make predictions) and ostensibly has enough capacity to learn over 100 languages without significant “interference” effects.

Here is the link to the respective research paper:

Friday, July 10, 2020

AI researchers create testing tool to find bugs in NLP from Amazon, Google, and Microsoft

Natural language processing has achieved impressive results! However, ...



"“The [sentiment analysis] failure rate is near 100% for all commercial models when the negation comes at the end of the sentence (e.g. ‘I thought the plane would be awful, but it wasn’t’), or with neutral content between the negation and the sentiment-laden word,” the paper reads."



AI researchers create testing tool to find bugs in NLP from Amazon, Google, and Microsoft | VentureBeat



Here is the underlying research paper: Semantically Equivalent Adversarial Rules for Debugging NLP models

Friday, December 13, 2019

Google Assistant can now interpret 44 languages on smartphones

Very impressive! Not only that Google Assistant will also help you to learn a new language!

Google Assistant can now interpret 44 languages on smartphones | VentureBeat: Google Assistant can now interpret up to 44 different languages on supported iOS and Android smartphones, Google announced.