Tuesday, February 09, 2021

Microsoft details Speller100, an AI system that checks spelling in over 100 languages

Recommendable! Very impressive work by Microsoft! The Tower of Babel no more! It is extremely or critcally  important to correct misspelled search entries. However, this is only one side of the search process. What about misspelled search targets?

"... We believe Speller100 is the most comprehensive spelling correction system ever made in terms of language coverage and accuracy. With this technology, we have improved the search results for all Bing users by expanding accurate spelling correction to over 100 languages. We have observed a double-digit improvement in both spelling correction precision and recall. ...
The number of pages with no results reduced by up to 30%. ...
In search we’ve found about 15% of queries submitted by customers have misspellings. When queries are misspelled, we match the wrong set of documents and trigger incorrect answers, which can produce a suboptimal results page for our customers. Therefore, spelling correction is the very first component in the Bing search stack because searching for the correct spelling of what users mean improves all downstream search components. ...
"

To get an idea how challenging this can be just look at the below example provided for the Germanic languages:

Below is an illustration of orthographic similarities between languages in the Germanic languages.

EnglishDutchAfrikaansGermanLuxembourgish
twotweetweezweizwee
bloodbloedbloedBlutBlutt
fingervingervingerFingerFanger
downloaddownloadenaflaaiherunterladeneroflueden



Microsoft details Speller100, an AI system that checks spelling in over 100 languages | VentureBeat

Here is the underlying Microsoft blog post:

No comments: