Posted: 8/26/2018
This is the reference standard for measuring questions & answers task in NLP:
As one can see, there are sometimes multiple state of the art submissions per month. With each submission the score is improved by at least 0.1-0.2 percentage points.
The human score is at 86.8%. The best NLP model is currently at 71.7% (as of 8/26/18) or only a 15.1 percentage points gap. A simple extrapolation at an improvement rate of 0.3 percentage points per months, it would take only about 4 years to reach parity on this task. However, progress in AI is made rather by leaps and bounds. Thus, if the average improvement per months is closer to 1 percentage point, it would only take another 15 months to achieve parity.
Given the rapid progress so far and the fierce competition by very capable, competing groups around the world, perhaps less than one year is possible to reach this milestone.
No comments:
Post a Comment