The Winograd Schema Challenge
cs.nyu.edu
The Winograd Schema Challenge
1–10 of 14 posts
Re: The Winograd Schema Challenge
#2Summary: The Winograd Schema Challenge was run on July 11, 2016 at IJCAI-16. There were four contestants. The first round of the challenge was a collection of 60 PDP's. The highest score achieved was 58% correct, by Quan Liu, from University of Science and Technology, China.. Hence, by the rules of that challenge, no prizes were awarded, and the challenge did not proceed to the second round.
It is scary to see that all the deep learning methods can be defeated by such simple methos (both in NLP and vision). I am genuinely curious why none of the recent Deep NLP approaches work here.
I heard a researcher describe deep learning as just standard ML with feature engineering replaced by architecture engineering.
If that is the case, we need fundamentally new ML approaches.
Re: The Winograd Schema Challenge
#3Summary: The Winograd Schema Challenge was run on July 11, 2016 at IJCAI-16. There were four contestants. The first round of the challenge was a collection of 60 PDP's. The highest score achieved was 58% correct, by Quan Liu, from University of Science and Technology, China.. Hence, by the rules of that challenge, no prizes were awarded, and the challenge did not proceed to the second round. It is scary to see that a…
> I am genuinely curious why none of the recent Deep NLP approaches work here.
To my understanding, it's because deep learning is fundamentally aimed at pattern-recognition, but makes no attempt to have a model of "reality" underlying that recognition.
I've seen some people argue that these models of reality are, themselves, nothing more than higher orders of pattern-recognition, and therefore ought to be amenable to deep-learning type methodologies. I have my doubts. Deep learning requires reinforcement training from big datasets, and the process of modelling reality does not often accommodate.
The process of learning to see things, navigate through space, etc., is something that you do through reinforcement training -- more or less like a neural network. Newborn babies don't see "things": they see colours and lights and motion, and it takes a couple of years of training before they reliably classify those sensory inputs. That's very much a big data exercise: every instant your eyes are open, you're collecting more data and training on it.
In contrast, consider the schema "The council denied the protesters a permit because they [feared/advocated] violence." The reason you can parse that sentence is not because you have trained yourself on a dataset of thousands of councils and tens of thousands of protests, and can therefore now recognise which might be fearing violence and which might be advocating it. Instead, you've built a model of the world out of vastly sparser data, via structured logical inferences. Which just isn't remotely the way that deep learning works.
So current AI techniques seem to have gotten very good at matching (and surpassing) a subset of cognition, but I'm not convinced that they will scale to cover these kind of schemas. I do think that fundamentally new ML approaches will be needed for these kind of domains. Neural Networks seem to be good at cerebellum and occipital lobe sort of tasks, but we'll need something else for the frontal lobe.
Re: The Winograd Schema Challenge
#4Re: The Winograd Schema Challenge
#5Summary: The Winograd Schema Challenge was run on July 11, 2016 at IJCAI-16. There were four contestants. The first round of the challenge was a collection of 60 PDP's. The highest score achieved was 58% correct, by Quan Liu, from University of Science and Technology, China.. Hence, by the rules of that challenge, no prizes were awarded, and the challenge did not proceed to the second round. It is scary to see that a…
> I am genuinely curious why none of the recent Deep NLP approaches work here. To my understanding, it's because deep learning is fundamentally aimed at pattern-recognition, but makes no attempt to have a model of "reality" underlying that recognition. I've seen some people argue that these models of reality are, themselves, nothing more than higher orders of pattern-recognition, and therefore ought to be amenable to…
> To my understanding, it's because deep learning is fundamentally aimed at pattern-recognition, but makes no attempt to have a model of "reality" underlying that recognition.
Isn't that just the issue that you're essentially making a crude analog of the first layers of the visual cortex and then overtrain them to only recognize specific patterns while ignoring the rest of reality because you simply don't have the higher-order facilities in current networks?
In other words doesn't the critique boil down that ANNs are orders of magnitude simpler than the human brain and thus can only fulfill far more narrow task then many combined abstraction layers?
New techniques are getting developed only as hardware advances in capability to actually run more complex networks.
Re: The Winograd Schema Challenge
#6Summary: The Winograd Schema Challenge was run on July 11, 2016 at IJCAI-16. There were four contestants. The first round of the challenge was a collection of 60 PDP's. The highest score achieved was 58% correct, by Quan Liu, from University of Science and Technology, China.. Hence, by the rules of that challenge, no prizes were awarded, and the challenge did not proceed to the second round. It is scary to see that a…
Nah, no need for new ML approaches. It's just time for the pendulum to swing back again from the current craze of data-driven/"deep learning"/continuous modeling to the alternative of symbolic/"Good Old-Fashioned AI"/discrete modeling.
The switch happens every ~30 years: around the ~1950s, in the late 1980s, and perhaps again soon...
I realize I'm glossing over many nuances of how the most successful AI/ML approaches combine together continuous and discrete modeling, and symbolic and numeric techniques.
Still, this is a real cultural divide for researchers. The "deep learning" types just want to increase their match/success percentages as high as they will go--with little regard for how the sausage is made--while the symbolic types care about whether the model itself provides actual explanatory value.
Re: The Winograd Schema Challenge
#7Earlier quoted context omitted.
> I am genuinely curious why none of the recent Deep NLP approaches work here. To my understanding, it's because deep learning is fundamentally aimed at pattern-recognition, but makes no attempt to have a model of "reality" underlying that recognition. I've seen some people argue that these models of reality are, themselves, nothing more than higher orders of pattern-recognition, and therefore ought to be amenable to…
> To my understanding, it's because deep learning is fundamentally aimed at pattern-recognition, but makes no attempt to have a model of "reality" underlying that recognition. Isn't that just the issue that you're essentially making a crude analog of the first layers of the visual cortex and then overtrain them to only recognize specific patterns while ignoring the rest of reality because you simply don't have the hi…
In other words, current computers are hopelessly slow
Re: The Winograd Schema Challenge
#8Summary: The Winograd Schema Challenge was run on July 11, 2016 at IJCAI-16. There were four contestants. The first round of the challenge was a collection of 60 PDP's. The highest score achieved was 58% correct, by Quan Liu, from University of Science and Technology, China.. Hence, by the rules of that challenge, no prizes were awarded, and the challenge did not proceed to the second round. It is scary to see that a…
Nah, no need for new ML approaches. It's just time for the pendulum to swing back again from the current craze of data-driven/"deep learning"/continuous modeling to the alternative of symbolic/"Good Old-Fashioned AI"/discrete modeling. The switch happens every ~30 years: around the ~1950s, in the late 1980s, and perhaps again soon... I realize I'm glossing over many nuances of how the most successful AI/ML approaches…
Every 30 years of Moore's gives enough oomph for the data driven stuff to step up one level in complexity?
Re: The Winograd Schema Challenge
#9Earlier quoted context omitted.
> To my understanding, it's because deep learning is fundamentally aimed at pattern-recognition, but makes no attempt to have a model of "reality" underlying that recognition. Isn't that just the issue that you're essentially making a crude analog of the first layers of the visual cortex and then overtrain them to only recognize specific patterns while ignoring the rest of reality because you simply don't have the hi…
In other words, current computers are hopelessly slow
Well, neural response in biological creatures is very slow---far slower than the clock cycles of an electrical computer.
Yet somehow, biological animals seem to demonstrate more intelligence and adaptability than machines. Perhaps, the issue is more with the machine architecture, than the speed.
Re: The Winograd Schema Challenge
#10Summary: The Winograd Schema Challenge was run on July 11, 2016 at IJCAI-16. There were four contestants. The first round of the challenge was a collection of 60 PDP's. The highest score achieved was 58% correct, by Quan Liu, from University of Science and Technology, China.. Hence, by the rules of that challenge, no prizes were awarded, and the challenge did not proceed to the second round. It is scary to see that a…
Nah, no need for new ML approaches. It's just time for the pendulum to swing back again from the current craze of data-driven/"deep learning"/continuous modeling to the alternative of symbolic/"Good Old-Fashioned AI"/discrete modeling. The switch happens every ~30 years: around the ~1950s, in the late 1980s, and perhaps again soon... I realize I'm glossing over many nuances of how the most successful AI/ML approaches…
Even long after the 80's, AI meant decision trees and heuristics for graph algorithms.