Machine learning has become alchemy (2017) [video] • Hacker News Board

Re: Machine learning has become alchemy (2017) [video]

#21

Post by Barrin92 » Wed, Dec 26, 2018, 1:19 AM UTC

The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception.

It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of intelligence.

In fact it basically hits the worst possible spot on the problem solving scale. It barely learns anything given the amount of computation effort an data that goes into it, but it just happens to be good enough to be practically preferable to old symbolic systems.

It is completely mysterious to me how networks that approximate some utility function are a huge step forward to giving insight into cognition, reasoning, modelling, creation of counterfactuals and the sort of mechanisms we actually need to produce human like performance.

Re: Machine learning has become alchemy (2017) [video]

#22

Post by m0zg » Wed, Dec 26, 2018, 1:25 AM UTC

oldgradstudent wrote: post #18

m0zg wrote: post #16

Earlier quoted context omitted.

Do you feel like you understand human limitations in these critical systems? Are humans suitable? Would AI be suitable if it performs statistically better than humans?

> Would AI be suitable if it performs statistically better than humans? In general yes, but it might depend on the pattern of failure - if your self-driving cars hunts me or my family personally, I might have problem with that. But how can you determine that without releasing it to the wild and waiting for bodies? Worse, say you have a safe system, but you need to modify the network (to fix some bug). How can you det…

But any technology can be deadly if you deploy it widely enough. _WhatsApp_ has resulted in "bodies" and it doesn't have any AI in it at all. First airplanes were basically flying coffins. Cars until early 90s had very little chance of survival in collision above 40mph. Many drugs have serious, sometimes deadly side effects. Quarter of a million people die in hospitals in US alone every year due to medical errors. 100% of those errors are currently made by humans.

It's remarkable that AI seems to be held to an arbitrarily high standard, often exceeding that of other technologies.

Re: Machine learning has become alchemy (2017) [video]

#23

Post by sonnyblarney » Wed, Dec 26, 2018, 1:31 AM UTC

Making a serious, industrial scale web app in 2000 felt like alchemy. It was all arcane, there were no established patterns, nobody knew how to do it for sure, there were a lot of hustlers, most of them thankfully sincere.

When something is new, it feels like a mystery - eventually we'll have a language for wrapping our heads around neural networks, even if it's not as clear cut as we'd like.

Re: Machine learning has become alchemy (2017) [video]

#24

Post by andrewflnr » Wed, Dec 26, 2018, 1:40 AM UTC

chrishare wrote: post #15

rwilson4 wrote: post #2

Science is a combination of theory and experiment. Sometimes theory advances faster than experiment, sometimes vice versa. Right now in ML, experiment aka practice is advancing faster than theory. Theory will eventually catch up.

I think a valid concern is that ML methods are being applied in critical, real-life scenarios without some practitioners being aware of flaws (bias, adversarial attacks, privacy issues) and without any theoretic safetynet that helps them reason about how these systems will behave. James Mickens discussed this recently in a keynote: https://www.usenix.org/conference/usenixsecurity18/presentat... Maybe the only way to…

We don't rely on the brain though, at least not on any single one. Any system that relies on human brains alone without cross-checking or, ideally, much simpler automatic systems, will eventually malfunction terribly. A large organization never wants to rely on a single person's judgment for anything, a programmer wants automated systems checking their work, etc.

Re: Machine learning has become alchemy (2017) [video]

#25

Post by typon » Wed, Dec 26, 2018, 1:58 AM UTC

Barrin92 wrote: post #21

The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception. It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of in…

The annoying part is that ML is not sold to the world like this. I would say the truth (I believe) in this comment is the "dirty little secret" of our industry. Everyone working on it knows this but the research and VC dollars are flowing in so no one wants to talk about it too much.

Re: Machine learning has become alchemy (2017) [video]

#26

Post by [deleted] » Wed, Dec 26, 2018, 1:59 AM UTC

joe_the_user wrote: post #3

It is simple to say deep learning is based on "alchemy" or "engineering" or whatever it is that isn't strong theory. And it's reasonable to say deep learning has a lot of mathematical and statistical intuitions but doesn't have a strong theory - maybe just doesn't yet have a strong theory or maybe can never get one. So this is by now a standard argument. The standard answers I think have been: 1) Well, we are discove…

[deleted]

Re: Machine learning has become alchemy (2017) [video]

#27

Post by soared » Wed, Dec 26, 2018, 2:00 AM UTC

Barrin92 wrote: post #21

The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception. It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of in…

> It barely learns anything given the amount of computation effort an data that goes into it, but it just happens to be good enough to be practically preferable to old symbolic systems.

I don’t follow this. Are you implying there haven’t been absolutely massive gains in computer vision, nlg, nlp, etc?

Re: Machine learning has become alchemy (2017) [video]

#28

Post by laretluval » Wed, Dec 26, 2018, 2:02 AM UTC

Barrin92 wrote: post #21

The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception. It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of in…

Maybe the concepts of “cognition, reasoning, modelling, creation of counterfactuals” etc. were not well-formed to begin with. Maybe intelligence, like gravity, is nothing like what we came up with using intuition. The fact is that black-box function approximators regularly beat systems designed around intuitive notions of cognition on observable metrics. Yes, the function approximators are bad. But the fact that they do better than more intuitively-satisfying symbolic systems suggests that our intuitive ideas about intelligence are even worse.

Re: Machine learning has become alchemy (2017) [video]

#29

Post by brain5ide » Wed, Dec 26, 2018, 2:02 AM UTC

sonnyblarney wrote: post #23

Making a serious, industrial scale web app in 2000 felt like alchemy. It was all arcane, there were no established patterns, nobody knew how to do it for sure, there were a lot of hustlers, most of them thankfully sincere. When something is new, it feels like a mystery - eventually we'll have a language for wrapping our heads around neural networks, even if it's not as clear cut as we'd like.

We had the neural networks, and the language. The problem is with the rebranding and the amount of marketing bullshit comming with it. The null hypothesis is that apl of that most of it is a pile of crap, for an overengineered, overoptimized solutions that are probably applied at abstraction layer different from one they are marketed on. There may come solution out of it, but it's more wishfull thinking than not.

Re: Machine learning has become alchemy (2017) [video]

#30

Post by laretluval » Wed, Dec 26, 2018, 2:04 AM UTC

soared wrote: post #27

Barrin92 wrote: post #21

The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception. It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of in…

> It barely learns anything given the amount of computation effort an data that goes into it, but it just happens to be good enough to be practically preferable to old symbolic systems. I don’t follow this. Are you implying there haven’t been absolutely massive gains in computer vision, nlg, nlp, etc?

There’s going to be people plugging their ears and shouting “but it’s just nonlinear function approximation!” all the way into the singularity.