Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning has become alchemy (2017) [video]

youtube.com

Post reply on HN
Print view View original

71–80 of 134 posts

Re: Machine learning has become alchemy (2017) [video]

#71

Post by yesenadam »

Barrin92 wrote: post #21

The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception. It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of in…

ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception.

It seems that an application like cancer diagnosis - feeding in test results and medical information to a NN, it finding patterns better than any human could, has nothing to do with human perception systems. It's just much better and faster at detecting patterns in complex things than humans are.

Re: Machine learning has become alchemy (2017) [video]

#72

Post by oldgradstudent »

protonfish wrote: post #44
Barrin92 wrote: post #21

The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception. It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of in…

It is a misconception that ANNs are based on human visual processing. Maybe "inspired by" is a better term. The 1959 paper "What the Frog's Eye Tells the Frogs Brain" disproved the belief that ANNs have anything to do real neural networks.

But it's a great brand name.

Re: Machine learning has become alchemy (2017) [video]

#73

Post by digitalzombie »

imperio59 wrote: post #47

The way I understand neural networks to work, they are actually series of connected infinitely valued logic gates, where given a numerical input from neg infinity to positive infinity, it spits out another number from neg infinity to positive infinity that feeds into the next set of logic gates, and at the end gives you a confidence interval from 0 to 1 of whether there was a pattern match or not. To me it's very sim…

> That's just marketing speak.

Pretty sure it was the initial thinking when Neural Network was created and it have move beyond that. I think people who know surface level only repeating this tadbit that's out of dated.

It's even in the official wikipedia article (https://en.wikipedia.org/wiki/Artificial_neural_network).

Everything you've stated was basically a personal opinion that could have been verified via google...

Re: Machine learning has become alchemy (2017) [video]

#74

Post by oldgradstudent »

whoisnnamdi wrote: post #12

Sure - perhaps alchemy in the sense that many practitioners are simply throwing things against the wall and seeing what sticks, but not in the sense that there isn't anything real behind all the math or engineering. Many advancements in machine learning have significant backing in theoretical proofs that a given algorithm will result in unbiased estimates, or will converge to such and such value etc. On some level, t…

From the talk, their main point about alchemy:

> Now alchemy is OK, alchemy is not bad, there is a place for alchemy, alchemy worked, alchemists invented metallurgy, [inaudible] dyed textiles, our modern glass making processes and medications.

> Then again, alchemists also believed they could cure diseases with leeches and transmute base metals into gold.

> For the physics and chemistry of the 1700s to usher in the sea change in our understanding of the universe that we now experience, scientists had to dismantle 2,000 years worth of alchemical theories.

> If you're building photo-sharing systems, alchemy is okay. But we're beyond that.

> Now we're building systems that govern health care, and mediate our civic dialogue. We influence elections.

> I would like to live in a society whose systems are built on top of verifiable rigorous thorough knowledge and not on alchemy.

Re: Machine learning has become alchemy (2017) [video]

#75

Post by chrstphrhrt »

typon wrote: post #25
Barrin92 wrote: post #21

The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception. It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of in…

The annoying part is that ML is not sold to the world like this. I would say the truth (I believe) in this comment is the "dirty little secret" of our industry. Everyone working on it knows this but the research and VC dollars are flowing in so no one wants to talk about it too much.

Can confirm, worked at an AI hype company for over a year and built all their systems outside the NN internals. There is going to be a correction.

Re: Machine learning has become alchemy (2017) [video]

#76

Post by m0zg »

oldgradstudent wrote: post #70
m0zg wrote: post #22

Earlier quoted context omitted.

But any technology can be deadly if you deploy it widely enough. _WhatsApp_ has resulted in "bodies" and it doesn't have any AI in it at all. First airplanes were basically flying coffins. Cars until early 90s had very little chance of survival in collision above 40mph. Many drugs have serious, sometimes deadly side effects. Quarter of a million people die in hospitals in US alone every year due to medical errors. 10…

> It's remarkable that AI seems to be held to an arbitrarily high standard, often exceeding that of other technologies. Because safety critical systems require trust. One way of establishing trust is to explain or prove why the system works. This cannot be done with machine learning. Say that you already have a safe system. Now you make a change. How do you demonstrate to a skeptical audience that the new system is s…

But again, my point is, you can't explain why a human works. Nor can you predict how the human will perform in an extreme situation they haven't been through before.

To modify your example. Say you have a doctor who haven't killed anyone yet and she reads a paper describing a new high risk / high reward treatment. The same standard of proof is not applied to her, so she can just go ahead and try it. Wasn't there a story just the other day where doctors were injecting people with some proteins linked to Alzheimer's? They weren't required to be "proven that they're safe" before they did that.

Re: Machine learning has become alchemy (2017) [video]

#77

Post by digitalzombie »

puzzledobserver wrote: post #10

Can someone give a concrete example of the kind of theoretical properties they desire of ``new-style'' machine learning? The kinds of properties that ``old-style'' learning methods guaranteed? People often complain about interpretability: in what sense is an SVM interpretable that a deep neural network is not? Or is the worry about gradient descent not finding global optima? But why is the global optimum a satisfacto…

Most machine learning algorithms that aren't statistical base doesn't give a CI. From a statistical stand point it doesn't give a sense of how good your prediction is. You can get a general sense with just CV.

Also your parameter is not inferable like in statistical algorithm. This is where I see people saying Deep Learning isn't interprable and there are research into this area. If you compare time series stat forecast algorithm with deep learning you at least get a CI on stat algorithm.

Randomly dropping node is pretty magic in my mind.

While I don't know much about SVM I know it's mathematically proven so there should be a way to interpret SVM fitted model.

I sure as hell wouldn't use ML in clinical trial for drugs. That's why biostat is a thing.

Re: Machine learning has become alchemy (2017) [video]

#78

Post by chartpath »

solotronics wrote: post #37
turingspiritfly wrote: post #31

Earlier quoted context omitted.

Those massive gains have yet to considered reliable enough to be considered trusthworthy. Would you consider them trusthworthy in court, where lives are at stake? Gains are nice but we are still so far from the essence of AI systems and considering how much resources we are pouring into learning, at this point all of them appear as nothing more than massive fat expensive toys

I have the perspective of an informed layman as a programmer who hasn't messed with ML yet. Wouldn't the "safest" solution be a system with multiple algorithms and a consensus mechanism?

Yeah. I'm coming from the same background, but with some experience orchestrating these systems in production. AFAIK a bunch of the submissions to the various "AGI" awards like Alexa prize use ensembles of models and some way of weighing each one based on a context in order to choose which classifier to trust in a particular scenario. E.g. MILABOT.

There is so much more than one monolithic NN since those are easy to saturate in terms of precision and recall with enough training data and features, but are not enough to provide good UX in any complex domain/ontology. So it makes more sense to have many different models trained on each subdomain/taxonomy so that each can be specialized and then combined orthogonally.

Then the question becomes "how do we orchestrate them?" Well, there is a lot of research from the 80s and 90s that kinda got left by the wayside due to hype cycles (see the last "AI winter"). My faves are Collagen and Ravenclaw. And there is a lot of literature around topic frame stack modelling, which can be combined with various expert systems or other logics. I am currently using CLIPS (PyKnow) with custom Ravenclaw implementation. I believe b4.ai is doing something similar without the logic/rules engine, and actually applying ML to topic selection as well. My systems are goal oriented so I like to give them a teleology for business reasons, which would not suffice for AGI ambitions.

TLDR data science isn't enough on its own. We need engineers to architect things properly to solve problems.

Re: Machine learning has become alchemy (2017) [video]

#79

Post by oldgradstudent »

m0zg wrote: post #76
oldgradstudent wrote: post #70

Earlier quoted context omitted.

> It's remarkable that AI seems to be held to an arbitrarily high standard, often exceeding that of other technologies. Because safety critical systems require trust. One way of establishing trust is to explain or prove why the system works. This cannot be done with machine learning. Say that you already have a safe system. Now you make a change. How do you demonstrate to a skeptical audience that the new system is s…

But again, my point is, you can't explain why a human works. Nor can you predict how the human will perform in an extreme situation they haven't been through before. To modify your example. Say you have a doctor who haven't killed anyone yet and she reads a paper describing a new high risk / high reward treatment. The same standard of proof is not applied to her, so she can just go ahead and try it. Wasn't there a st…

We can't explain how and why humans work, but we have a lot of experience with them. We know their failure modes pretty well.

As to medical treatments, they are done on volunteers under informed consent. If you want to experiment with a car driven by AI on a private track staffed with volunteers, be my guest. It's between you, your conscience, your deity, and maybe OSHA.

But for experimentation and deployment on public roads, you need to convince other, possibly skeptical, people that your technology works better than humans. How do you do that with AI/ML-based systems?

Re: Machine learning has become alchemy (2017) [video]

#80

Post by dontreact »

andrewflnr wrote: post #24
chrishare wrote: post #15

Earlier quoted context omitted.

I think a valid concern is that ML methods are being applied in critical, real-life scenarios without some practitioners being aware of flaws (bias, adversarial attacks, privacy issues) and without any theoretic safetynet that helps them reason about how these systems will behave. James Mickens discussed this recently in a keynote: https://www.usenix.org/conference/usenixsecurity18/presentat... Maybe the only way to…

We don't rely on the brain though, at least not on any single one. Any system that relies on human brains alone without cross-checking or, ideally, much simpler automatic systems, will eventually malfunction terribly. A large organization never wants to rely on a single person's judgment for anything, a programmer wants automated systems checking their work, etc.

We rely on a single human to drive a car.

In medicine, machine learning systems will work alongside other brains.

What are the instances you’re imagining where a group of brains running an important system are replaced by a single machine learning algorithm running in isolation?

Post reply on HN