Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning has become alchemy (2017) [video]

youtube.com

Post reply on HN
Print view View original

121–130 of 134 posts

Re: Machine learning has become alchemy (2017) [video]

#121

Post by andrewflnr »

dontreact wrote: post #80
andrewflnr wrote: post #24

Earlier quoted context omitted.

We don't rely on the brain though, at least not on any single one. Any system that relies on human brains alone without cross-checking or, ideally, much simpler automatic systems, will eventually malfunction terribly. A large organization never wants to rely on a single person's judgment for anything, a programmer wants automated systems checking their work, etc.

We rely on a single human to drive a car. In medicine, machine learning systems will work alongside other brains. What are the instances you’re imagining where a group of brains running an important system are replaced by a single machine learning algorithm running in isolation?

Yeah, and around 800 people a day die in auto accidents in the US. Bringing up cars bolsters my point, which is that "we rely on the brain without understanding it and that works out fine" is not a good argument. So let's be careful. I'm mystified as to why that seems to be controversial.

Re: Machine learning has become alchemy (2017) [video]

#122

Post by rahimiali »

puzzledobserver wrote: post #10

Can someone give a concrete example of the kind of theoretical properties they desire of ``new-style'' machine learning? The kinds of properties that ``old-style'' learning methods guaranteed? People often complain about interpretability: in what sense is an SVM interpretable that a deep neural network is not? Or is the worry about gradient descent not finding global optima? But why is the global optimum a satisfacto…

this is a followup post i wrote to answer exactly your question: http://www.argmin.net/2018/01/25/optics/

Re: Machine learning has become alchemy (2017) [video]

#123

Post by candiodari »

RA_Fisher wrote: post #106
typon wrote: post #25

Earlier quoted context omitted.

The annoying part is that ML is not sold to the world like this. I would say the truth (I believe) in this comment is the "dirty little secret" of our industry. Everyone working on it knows this but the research and VC dollars are flowing in so no one wants to talk about it too much.

Statistics is quite valuable and grounded in theory. It's a trusty tool if used with respect (like a knife).

Yeah, too bad the theory requires a number of things to be true for data and models ... that are essentially never true. Once you think about

1) is the independance assumption really applicable here ?

2) does the underlying model satisfy the law of large numbers ? (NOT your model, the underlying model, ie. "reality")

3) am I really only interested in predictions within the data range ?

The answer for all these questions is almost universally "no", and the theory says that statistics is not guaranteed to work under those circumstances.

Correct statistical answers:

1) which average percentage of borrowers will default ? There is no answer to that question, because many non-random influences will make borrowers default in synchronized, and therefore very much non-independent ways (say, an economic crisis, an earthquake, an epidemic). Therefore you cannot correctly even calculate an average.

2) which average percentage of borrowers will default ? A model that describes whether an individual will default is going to be non-differentiable, and therefore certainly violate the law of large numbers (there will be sudden jumps in the default rate everywhere because of a million random reasons. An idiot tv anchor declares now the time to sell, and half the town finds out their loan is underwater, say). So underlying models involving humans essentially never satisfy the law of large numbers.

3) which average percentage of borrowers will default ? The number of factors changing that value do not match between the period your data is from (ie. the past) and the period you're predicting. Because there are so many real-world factors that affect your variable you can never avoid this situation. Therefore you cannot predict.

All of these issues allow one to construct realistic scenarios where even trivial statistics will fail spectacularly.

Re: Machine learning has become alchemy (2017) [video]

#124

Post by mentalhealth »

dang wrote: post #84
mentalhealth wrote: post #51

Earlier quoted context omitted.

Going all in on the "unintelligible black box" model of health insurance, I see.

Would you please stop being rude to others in HN comments? I'm sure you can express your substantive points without this. https://news.ycombinator.com/newsguidelines.html Also, please don't make single-purpose accounts on HN.

I agree that this was a bit flippant (though accurate), but if you took the time to look through my posting history over the last nearly five years, you'd learn why I made this account in the first place: https://news.ycombinator.com/item?id=7004159

Because I was literally harassed out of a previous job for discussing my views here on HN. I am an actual medical professional; discussions of this industry on HN are full of people with zero knowledge or experience hypothesizing on the topic; you regularly fail to censure them, but censure the people who are involved and are able to conduct substantive discourse.

I'm disabling this account and leaving. Enjoy your echo chamber, and know that you persist it through your direct action.

Re: Machine learning has become alchemy (2017) [video]

#125

Post by EvilTerran »

notafraudster wrote: post #118
EvilTerran wrote: post #93

Earlier quoted context omitted.

For the benefit of laypeople like myself: - IID: "independent and identically distributed", https://en.wikipedia.org/wiki/Independent_and_identically_di... - OLS: "ordinary least squares", https://en.wikipedia.org/wiki/Ordinary_least_squares (I think)

Yeah, to be clear, the discussion you're responding to is pretty out in the weeds. The great-grandparent to your content raised the imho well founded objection that many uses of machine learning "work" (produce good out-of-sample predictive accuracy) but we don't know "why". We have generated a distinct lack of theory. We know very little about assumptions and how they are violated. Separately, we know very little ab…

That's a huge help, thank you!

Re: Machine learning has become alchemy (2017) [video]

#126

Post by RA_Fisher »

candiodari wrote: post #123
RA_Fisher wrote: post #106

Earlier quoted context omitted.

Statistics is quite valuable and grounded in theory. It's a trusty tool if used with respect (like a knife).

Yeah, too bad the theory requires a number of things to be true for data and models ... that are essentially never true. Once you think about 1) is the independance assumption really applicable here ? 2) does the underlying model satisfy the law of large numbers ? (NOT your model, the underlying model, ie. "reality") 3) am I really only interested in predictions within the data range ? The answer for all these questi…

It's true, statistics can't answer every question. There are some limits.

Re: Machine learning has become alchemy (2017) [video]

#127

Post by chrstphrhrt »

kuhhk wrote: post #119
chrstphrhrt wrote: post #75

Earlier quoted context omitted.

Can confirm, worked at an AI hype company for over a year and built all their systems outside the NN internals. There is going to be a correction.

May I ask if this insight is why you left? What do you work on now?

Actually left due to anti-engineering culture (no tech leaders), succumbed to the distortions of Conway's law due to imbalance between professional services and core/platform.

That said, the nature of the lack of leadership was around the MBA mindset that data science is the be-all end-all of AI, which IME is so far from the truth.

There needs to be many layers/spirals of naive methods (i.e. straight ahead engineering) as a vanguard on the front lines with the DS/NNs/NLP bringing up the flanks for aspects of the domains that become more well understood over time.

So yeah, tail wagging the dog, cart before horse etc. If you just barrel ahead with a monolithic NN pretending that all classes are created equal (structureless blob), which is what our DS PhDs were doing, it will quickly ossify. In our case it was intent prediction so we got to pretty high accuracy, but the labels were indicating many different categories that actually had relations in nature that could not be expressed. Ironically it took some regular engineers to research HMC (hierarchical multi-label classification) and ensembles and implement a new model training framework to support them. Not sure what they teach in school but it doesn't seem to be very practical.

EDIT: now working on messaging (SMS/MMS/RCS/ABC) with a focus on dialog management and logical rules with layering-in of progressively less naive stats methods strategically rather than blindly. It helps to have an existing revenue stream that can be leveraged to create even more value with AI features rather than the backwards hail Mary of not iterating from a foothold of existing traction.

Re: Machine learning has become alchemy (2017) [video]

#128

Post by quanticle »

EmbarrassedFuel wrote: post #97
quanticle wrote: post #90

Earlier quoted context omitted.

Are you implying there haven’t been absolutely massive gains in computer vision, nlg, nlp, etc? The implication is that the massive gains haven't been the result of any algorithmic breakthroughs, but rather have been due to the application of massive amounts of computational resources, which weren't available when current ML algorithms were invented. So far as I can tell, that's a true accusation. If you look at the…

A small bone of contention: it’s not about Moore’s law per se, which has been ‘dead’ since about 2013, coincidentally when the deep learning revival started. It’s matrix multiplication ASIC development that is driving the progress. GPUs already existed when the idea to use them to make the feasible size of neural nets larger came about. For a long time the drive for the increase of GPU compute power was still gaming/…

I'm not saying that neural nets are brute force. I'm saying that there haven't been any algorithmic improvements in neural nets to make them more computationally feasible than they were when they were first invented. Instead, we have specialized hardware which can just do the necessary computation quickly enough to make neural nets feasible.

It's not like neural nets are a new technology. They've been known since the '80s, at least. It's just that they were considered a dead end, because we didn't have the computational resources to run deep neural nets, nor did we have sufficient training data to make neural-net approaches feasible. Once those preconditions were met, neural nets took off in short order.

Re: Machine learning has become alchemy (2017) [video]

#129

Post by pdimitar »

dang wrote: post #83
pdimitar wrote: post #62

Earlier quoted context omitted.

Stop playing the victim. Just make a driverless car that never makes mistakes already -- and I promise you the goalpost won't move for a while. Go and chess mean nothing out there. The world has a lot of very practical and concrete needs that this magical "AI" is nowhere near solving yet. I find the constant defensiveness of the ML proponents tiring and cringe-worthy. Somebody should remind you that what you are doin…

This comment crosses into incivility. Please don't do that on HN, regardless of how right you think you are compared to someone else. https://news.ycombinator.com/newsguidelines.html

Understood and point taken. Thank you.

Re: Machine learning has become alchemy (2017) [video]

#130

Post by dontreact »

andrewflnr wrote: post #121
dontreact wrote: post #80

Earlier quoted context omitted.

We rely on a single human to drive a car. In medicine, machine learning systems will work alongside other brains. What are the instances you’re imagining where a group of brains running an important system are replaced by a single machine learning algorithm running in isolation?

Yeah, and around 800 people a day die in auto accidents in the US. Bringing up cars bolsters my point, which is that "we rely on the brain without understanding it and that works out fine" is not a good argument. So let's be careful. I'm mystified as to why that seems to be controversial.

Are you saying that we should require 2 drivers per car? It seems like at the moment the risk/reward ratio of allowing people to drive a car after getting a license seems to work out well enough. 1 death per 100 million miles driven, and falling.
Post reply on HN