This is a monster paper: about 445 authors, 706 references (on 49 pages), 100 pages total!
Unfortunately, the references are indeed severely bloated! This paper is a hodge podge or smorgasbord if you prefer! As they say too many cooks spoil the broth!
About 15 pages list/cover the author contributions!!! This is highly unusual!
"The construction of BIG-bench was organized as an open collaboration on GitHub"! Perhaps that explains some of my remarks here!
The paper is about the Beyond the Imitation Game benchmark (BIG-bench).
It reflects some of the popular ideologies of the day e.g. gender bias, low resource languages. Not exactly what you wan in a research paper about machine learning/artificial intelligence! It's distracting!
"A worrying finding is that model performance on social bias metrics often grows worse with increasing scale" That is something to ponder about! Does this not sound familiar compared to large societies come with increasing social biases (because in smaller societies most everyone knows each other etc.)
E.g. the authors admit "There is no broadly accepted definition of “low-resource” [language] ... Performance of the evaluated models on low-resource languages is generally low. ...". They also included "Kannada, a Dravidian language
spoken by around 40 million people in India". So why do the authors include low resource languages into this benchmark?
Why they included chess game playing tasks is another good question! It is not exactly a linguistic subject? The authors admit "None of the BIG-G models tested can solve this task [check mate moves]."
No comments:
Post a Comment