Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
1–10 of 13 posts
Re: Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
#2This is a fascinating discussion, particularly in light of the human brain architecture. Back in school, my prof used to reference the distinctions between the left and right brains in context of alternative computing approaches used by each half. The right brain seems to be more in line with the statistical machine learning approach, effectively a black box data processor producing intuitive results from large quantities of data. The right brain works great in context of normal data distributions. The left brain is the logical brain and can override the right brain. The left brain is about language based reasoning and can handle the novel and unexpected situations or in other words the statistical outliers that give trouble to the right brain.
Re: Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
#3Peter Norvig is "... now taking a short leave of absence from Google to update his AI textbook."
This is news to me, and potentially quite exciting (even if I have just finally acquired my own copy). Does anyone know anything else about this; what sort of new material, etc?
Re: Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
#4This is a fascinating discussion, particularly in light of the human brain architecture. Back in school, my prof used to reference the distinctions between the left and right brains in context of alternative computing approaches used by each half. The right brain seems to be more in line with the statistical machine learning approach, effectively a black box data processor producing intuitive results from large quant…
the idea that the brain performs several different types of computation on the same data set and then compares the results is interesting and worth following up on.
Re: Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
#5Although this doesn't involve machine-learned models, this story highlights the worth of good human judgment, something that currently isn't replicable in machines.
Re: Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
#6This is a fascinating discussion, particularly in light of the human brain architecture. Back in school, my prof used to reference the distinctions between the left and right brains in context of alternative computing approaches used by each half. The right brain seems to be more in line with the statistical machine learning approach, effectively a black box data processor producing intuitive results from large quant…
Is there any proof for the statements made about left/right brains? Was the prof learned in neuroscience? Or is it just some folk tale stuff?
Re: Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
#7It's possible to think of the process that airline safety regulators have gone through where they have observed crashes and guessed about the cause and then suggested fixes as a type of learning problem.
For goog's case it would be fun to try and build a supervised learning system whose sole purpose is to try and identify the queries which a human observer would consider a "catastrophic failure"
I've also heard of stories where decision trees supposedly outperformed human cardiologists in making diagnoses. (I'm skeptical of this claim but let's assume that it's true) If this type of advance is real then it could save a lot of lives. Unfortunately, if goog's engineering team has this kind of doubt about the machine, then I imagine that it would be easy to persuade a random jury that installing such a poorly understood black box is negligence.
Re: Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
#8This is a fascinating discussion, particularly in light of the human brain architecture. Back in school, my prof used to reference the distinctions between the left and right brains in context of alternative computing approaches used by each half. The right brain seems to be more in line with the statistical machine learning approach, effectively a black box data processor producing intuitive results from large quant…
Is there any proof for the statements made about left/right brains? Was the prof learned in neuroscience? Or is it just some folk tale stuff?
Nothing folksy about left/right brain distinction. Check out this tour de force TED video with a Harvard neuroscientist talking about the differences in brain halves http://www.ted.com/talks/view/id/229
Re: Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
#9Failure would rarely be "catastrophic" in the context of web search as described in the article. If the ML model makes an incorrect prediction on new data you have a bad search result. No big deal - just feed the new data back into the model and learn again.
If the data set is large enough then the ML model may find patterns that escape a human expert. When it comes to finding patterns in very large datasets machines scale much better than humans. Given a large enough dataset a ML approach should be less susceptible to the Black Swan phenomena than human experts.
On the other hand, if failure of the system really could be considered catastrophic then there could always be a human involved. In these cases output from ML models could be one of the inputs that the experts considers before coming to a final decision. E.g. you wouldn't want a ML model doing medical diagnosis by itself but it could be very useful for identifying patients that should be double-checked, scanning diagnosis for errors etc.
Re: Are Human Experts Less Prone to Catastrophic Errors than Machine-Learned Models?
#10A human expert may be better when you're in unknown territory (i.e., something happens that doesn't fall into the domain of expertise) because then external factors acquired over a lifetime may prove to be more useful than textbook or algorithmic knowledge.
But more importantly I _do_ know that statistical models are better at diagnosis and prediction than are human experts. Sad to say, doctors, lawyers, judges and other foolish people keep us from using them.
FuturePundit summary article "Statistical Prediction Rules More Accurate Than Experts": http://www.futurepundit.com/archives/001558.html
The FuturePundit article reviews the paper: "50 Years of Successful Predictive Modeling Should Be Enough: Lessons for Philosophy of Science" by Michael A. Bishop and J. D. Trout http://www.google.com/search?hl=en&q=%2250+Years+of+Succ...