Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning has become alchemy (2017) [video]

youtube.com

Post reply on HN
Print view View original

81–90 of 134 posts

Re: Machine learning has become alchemy (2017) [video]

#81

Post by m0zg »

oldgradstudent wrote: post #79
m0zg wrote: post #76

Earlier quoted context omitted.

But again, my point is, you can't explain why a human works. Nor can you predict how the human will perform in an extreme situation they haven't been through before. To modify your example. Say you have a doctor who haven't killed anyone yet and she reads a paper describing a new high risk / high reward treatment. The same standard of proof is not applied to her, so she can just go ahead and try it. Wasn't there a st…

We can't explain how and why humans work, but we have a lot of experience with them. We know their failure modes pretty well. As to medical treatments, they are done on volunteers under informed consent. If you want to experiment with a car driven by AI on a private track staffed with volunteers, be my guest. It's between you, your conscience, your deity, and maybe OSHA. But for experimentation and deployment on publ…

>> How do you do that with AI/ML-based systems?

Same as with humans: from statistics. I don't see any other way. E.g. autonomous cars will never have zero accident rate, but if they have half as many accidents per million miles as humans, it's a no brainer to me that they should be deployed. We're not going to get there in the next 20 years, so I'm not saying they should be deployed _now_, but eventually it will happen.

Re: Machine learning has become alchemy (2017) [video]

#82

Post by ramblerman »

sonnyblarney wrote: post #23

Making a serious, industrial scale web app in 2000 felt like alchemy. It was all arcane, there were no established patterns, nobody knew how to do it for sure, there were a lot of hustlers, most of them thankfully sincere. When something is new, it feels like a mystery - eventually we'll have a language for wrapping our heads around neural networks, even if it's not as clear cut as we'd like.

I understand what you are getting at, but I would call that a new frontier.

There was ultimately nothing black box about the code we cobbled together back then.

Re: Machine learning has become alchemy (2017) [video]

#83

Post by dang »

pdimitar wrote: post #62
wetpaws wrote: post #58

Earlier quoted context omitted.

Every time there is some progress in AI people would yell "but this is not a real AI 'cause humans can..." ( play go/understand pictures/language/semantics/etc, goalpost is always in motion)

Stop playing the victim. Just make a driverless car that never makes mistakes already -- and I promise you the goalpost won't move for a while. Go and chess mean nothing out there. The world has a lot of very practical and concrete needs that this magical "AI" is nowhere near solving yet. I find the constant defensiveness of the ML proponents tiring and cringe-worthy. Somebody should remind you that what you are doin…

This comment crosses into incivility. Please don't do that on HN, regardless of how right you think you are compared to someone else.

https://news.ycombinator.com/newsguidelines.html

Re: Machine learning has become alchemy (2017) [video]

#84

Post by dang »

mentalhealth wrote: post #51
module0000 wrote: post #42

Earlier quoted context omitted.

>> Those massive gains have yet to considered reliable enough to be considered trusthworthy. We're using them at Generic Health Insurance Megacorp in production - lots of enterprises are. If you are in the IT industry, it might be useful to spend some lab time with ML. Possibly you have a misconception of ML and/or confuse it with AI.

Going all in on the "unintelligible black box" model of health insurance, I see.

Would you please stop being rude to others in HN comments? I'm sure you can express your substantive points without this.

https://news.ycombinator.com/newsguidelines.html

Also, please don't make single-purpose accounts on HN.

Re: Machine learning has become alchemy (2017) [video]

#85

Post by another-one-off »

turingspiritfly wrote: post #31
soared wrote: post #27

Earlier quoted context omitted.

> It barely learns anything given the amount of computation effort an data that goes into it, but it just happens to be good enough to be practically preferable to old symbolic systems. I don’t follow this. Are you implying there haven’t been absolutely massive gains in computer vision, nlg, nlp, etc?

Those massive gains have yet to considered reliable enough to be considered trusthworthy. Would you consider them trusthworthy in court, where lives are at stake? Gains are nice but we are still so far from the essence of AI systems and considering how much resources we are pouring into learning, at this point all of them appear as nothing more than massive fat expensive toys

> Would you consider them trusthworthy in court, where lives are at stake?

Probably. Human intelligence is extremely fallible - based on the statistics the only reason we trust humans to do half the stuff they do is because there is literally no choice.

If we held humans to a high objective engineering standard We wouldn't:

* Let them drive

* Let them present their memories as evidence in a court case

* Entrust them with monitoring jobs

* Allow them to perform surgical operations

Humans are the best we have at those things, but from a "did we secure the best result with the information we had" perspective they are not very reliable. A testable and consistently performing AI with known failure modes might even be able to outperform a human with a higher failure rate (eg, we can reconfigure our road systems if there is just one scenario an AI driver can't handle).

Basically, you might be dead on the money that they are not 'trustworthy enough', but lets not lose sight of the fact that even being an order of magnitude from human performance might be enough after costs and engineering benefits get factored in. The weakest link is the stupidest human, and that is quite a low bar.

Re: Machine learning has become alchemy (2017) [video]

#86

Post by jsinai »

laretluval wrote: post #30
soared wrote: post #27

Earlier quoted context omitted.

> It barely learns anything given the amount of computation effort an data that goes into it, but it just happens to be good enough to be practically preferable to old symbolic systems. I don’t follow this. Are you implying there haven’t been absolutely massive gains in computer vision, nlg, nlp, etc?

There’s going to be people plugging their ears and shouting “but it’s just nonlinear function approximation!” all the way into the singularity.

Classic regression is only intelligible because there were only a few parameters and people could use ANOVA to try and interpret them. IMO ANOVA is alchemy as well and most people trained to use it, don’t fully understand it. Should we leave decisions to those kinds of models?

Not to mention that part of what makes NN such a step forward is precisely the high nonlinearity. When you have millions of parameters, the contribution of each is unimportant and so any kind of analog to ANOVA would be barking up the wrong tree. In a NN all those parameters work in concert to learn a decision boundary to separate data. It’s not intelligible at the minute level but at least we know what it’s doing in the end. The problem of course lies with the outliers and that’s not so much a problem with NN being a black box as it is a problem with the nature of large datasets and our own inability to rationalize each and every datapoint.

I’m going to defend NN as the natural evolution of regression. It’s precisely their high nonlinearity which makes them better. The problem is not that they “are” alchemy but that we treat them “as” magic. Society as no place leaving important decisions to algorithms unintended, NN or not. If major insurance companies left their decisions to logistic regression (which was and is still the case), then would we be making the same arguments? Probably not because someone paid by the insurance company will pull out that other kind of alchemy called ANOVA ...

Re: Machine learning has become alchemy (2017) [video]

#87

Post by notafraudster »

bumby wrote: post #60
solotronics wrote: post #37

Earlier quoted context omitted.

I have the perspective of an informed layman as a programmer who hasn't messed with ML yet. Wouldn't the "safest" solution be a system with multiple algorithms and a consensus mechanism?

I believe some models do precisely that. Random forest ML as an example tallies "votes" on the outcome. I'm not sure how robustly multiple algorithms have been applied to this voting technique, but it would be an interesting read if anyone has information on it.

"Vote tallying" is basically taking a mode/weighted mode response rather than a mean/median/trimmed mean/weighted mean response. There are contexts where this is ideal (for example, classification in multiple unordered classes where mode is the only measure of central tendency that is even reasonable); cases where it's superfluous (in 2-class classification the mode is the median is the sign of the mean); and cases where it's bad (in regression with a continuous outcome where the modal prediction has probability exactly 0). So really it depends on the space where you want to use it.

Typically in a regression setting with an ensemble learner you're using a kind of weighted mean, where the weights are selected based on cross-validation performance. This is sometimes called a "super learner". See van der Laan, Polley, and Hubbard 2007.

Note that this suffers from being similarly awful in terms of theory as a lot of other ML stuff. It is not, for example, the case that two apparently similar datasets or problem domains will produce similar super-learner weights. Which is disturbing, because it's easy to believe that say SVM does better at X and penalized ordered logit at Y, but it's hard to believe that they both do better seemingly at random.

Re: Machine learning has become alchemy (2017) [video]

#88

Post by oldgradstudent »

m0zg wrote: post #81
oldgradstudent wrote: post #79

Earlier quoted context omitted.

We can't explain how and why humans work, but we have a lot of experience with them. We know their failure modes pretty well. As to medical treatments, they are done on volunteers under informed consent. If you want to experiment with a car driven by AI on a private track staffed with volunteers, be my guest. It's between you, your conscience, your deity, and maybe OSHA. But for experimentation and deployment on publ…

>> How do you do that with AI/ML-based systems? Same as with humans: from statistics. I don't see any other way. E.g. autonomous cars will never have zero accident rate, but if they have half as many accidents per million miles as humans, it's a no brainer to me that they should be deployed. We're not going to get there in the next 20 years, so I'm not saying they should be deployed _now_, but eventually it will happ…

Imagine for a moment that you have a working self-driving car in the field, and now you want to make a relatively minor change to the system.

How do you convince yourself or a regulator, not that the technology in general is safe, but that your specific change does not make it unsafe?

What happens if this specific change increases the fatality rate from 1 in 100 million miles to 1 in 50 million miles? Crashes will still be rare and there's enough statistical noise that when you finally understand the change is bad, you may have killed hundreds.

Re: Machine learning has become alchemy (2017) [video]

#89

Post by TeMPOraL »

m0zg wrote: post #81
oldgradstudent wrote: post #79

Earlier quoted context omitted.

We can't explain how and why humans work, but we have a lot of experience with them. We know their failure modes pretty well. As to medical treatments, they are done on volunteers under informed consent. If you want to experiment with a car driven by AI on a private track staffed with volunteers, be my guest. It's between you, your conscience, your deity, and maybe OSHA. But for experimentation and deployment on publ…

>> How do you do that with AI/ML-based systems? Same as with humans: from statistics. I don't see any other way. E.g. autonomous cars will never have zero accident rate, but if they have half as many accidents per million miles as humans, it's a no brainer to me that they should be deployed. We're not going to get there in the next 20 years, so I'm not saying they should be deployed _now_, but eventually it will happ…

> Same as with humans: from statistics. I don't see any other way.

I.e. from body count.

One problem here is that we know humans have common, stable brain architecture, so the limits and failure modes we experienced are stable too, and can be accounted for and worked around. People won't fail you in completely surprising ways.

DNNs are each a different breed; between changing architectures and tuning hyperparameters, I don't see how trust in one instance can be translated to trust un another.

Re: Machine learning has become alchemy (2017) [video]

#90

Post by quanticle »

soared wrote: post #27
Barrin92 wrote: post #21

The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception. It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of in…

> It barely learns anything given the amount of computation effort an data that goes into it, but it just happens to be good enough to be practically preferable to old symbolic systems. I don’t follow this. Are you implying there haven’t been absolutely massive gains in computer vision, nlg, nlp, etc?

Are you implying there haven’t been absolutely massive gains in computer vision, nlg, nlp, etc?

The implication is that the massive gains haven't been the result of any algorithmic breakthroughs, but rather have been due to the application of massive amounts of computational resources, which weren't available when current ML algorithms were invented. So far as I can tell, that's a true accusation. If you look at the papers coming out of Google and Facebook, they talk about throwing thousands of hours of specialized GPU (or even more specialized and expensive TPU) time at some of the problems. The advances have more to do with Moore's Law making brute force feasible than they have to do with algorithmic breakthroughs.

Post reply on HN