Not even xlose. This was a small incremental gain (maybe 10% relative reduction in errors). It's cool work but if you're an outsider this specific announcement shouldn't lead to any conclusions.
And this is close talk speech. The holy grail (20 feet away at a party with error rate below human) is decades away.
It would have been more honest to talk about how they increased their conversion rate from N% to M%, this we increased our good rate by Y% or decreased our error rate X% is the oldest trick in the book for academic papers (one that is likely to earn one a well deserved beat down by the PC).
> A nitpick, perhaps, but isn't that three orders of magnitude?
Perhaps the example was a best-case, and the usual improvement is about 10x. (That or 'order of magnitude' has gone the way of 'exponential' in popular use. I don't think I've noticed that elsewhere, though.)
The whole point of the article is that they've (hopefully) improved the accuracy by using recent developments and combining different types of counts, and that method dramatically increases the count.
> The breakthrough didn’t crack 80%, so three cheers for wide credibility intervals with error margin, but I expect the predictor might be feeling slightly more nervous now with one year left to go.
Good call on that by the author; according to this summary paper [1], a model reached ~79-93% accuracy on various Winograd data sets.
> What about the next question – how did the models do? Amazingly well. [...] This means that since 1992, the models have been within 3% of the measurements.
Wow! I wonder though how they account for over fitting of the data. Is it a real solution or a statistical anomaly? I ask because it seems that the progress over the last year(s) has been small increments within the 9-10% range.
What are you referring to here?
reply