Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning Crash Course: The Bias-Variance Dilemma

ml.berkeley.edu

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#41

Post by platz »

sgt101 wrote: post #39
platz wrote: post #20

Earlier quoted context omitted.

classic statistics is much more interested in the explanatory power of models to describe phenomena. ML is mainly interested in prediction (correlation instead of causation), typically over some data that just fell in your lap.

I simply don't recognise that characterisation of ML. I think that "data driven AI" fits far better. ML emerged in a number of ways over the years, but a strong driver of the last iteration was the knowledge engineering bottleneck encountered in fifth generation computing and surrounding the demise of the last turing center. I invite you to read Chris Bishop's or Stephen Muggleton's books. Anyone who works with data…

data that "fell into your lap" is a euphemism for a process in which the data was not generated by a controlled experiment in which a hypothesis is created first, and then the data is gathered according to a procedure using the proper controls and testing.

"fell into your lap" has nothing to do with how hard the work is, it's about the difference between a controlled experiment and an observational study. the bulk of ML is observational in nature (focusing on prediction) and therefore has nothing to say about causation or understanding the causal variables of the underlying reality.

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#42

Post by verifex »

therajiv wrote: post #5

Wow, the discussion on the Fukushima civil engineering decision was pretty interesting. However, I find it surprising that the engineers simply overlooked the linearity of the law and used a nonlinear model. I wonder if there were any economic / other incentives at play, and the model shown was just used to justify the decision? Regardless, that post was a great read.

I would like to believe that the engineers working on building a nuclear reactor were not so easily fooled by an over-fitting problem; if that were the case, wouldn't that mean we now have actionable information on whether other nuclear power plants are designed correctly?

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#43

Post by joshuamorton »

cromd wrote: post #24
taeric wrote: post #4

This seems to ultimately come down to an idea that folks have a hard time shaking. It is entirely possible that you cannot recover the original signal using machine learning. This is, fundamentally, what separates this field from digital sampling. And this is not unique to machine learning, per se. https://fivethirtyeight.com/features/trump-noncitizen-voters... has a great widget that shows that as you get more data,…

If it helps anyone, the FiveThirtyEight article describes a scenario where people take a survey about immigration status and voting. Most legal citizens will correctly identify themselves, but some will accidentally check the wrong box and say they are an illegal immigrant. If you have a billion citizens and 10 illegal immigrants truly taking the survey, and people check the wrong box 1 in 1000 times, your "percentag…

The other common example of the same phenomena is a test for a deadly genetic defect with a 1% false positive rate. If the incidence of the defect is .01% and you test positive, its actually more likely that you don't have the disease. (although this can be solved with bayesianism over frequentism).

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#44

Post by gnaritas »

kensoh wrote: post #30
jorgemf wrote: post #28

Earlier quoted context omitted.

trading systems are very difficult to model because there are more variable than you can imagine. So basically you are creating a model with less information from the domain than you rather need. But this is not the case in all domain. With AlphaGo there are several things you have to consider. It is not only using deep learning it also uses monte carlo tree search, basically this algorithms is good at exploring sear…

For the branch of trading technical analysis which assumes that all information is factored into price and price action, the variables will drastically be reduced probably to something like volume, low, high, open, close etc. That makes it ideal to develop algos. But as the fundamentalists would say, assuming that everything (demand from insider news etc) has been factored into price is an overly simplistic way of lo…

> For the branch of trading technical analysis which assumes that all information is factored into price and price action, the variables will drastically be reduced probably to something like volume, low, high, open, close etc.

I'd call that quantitative analysis, not technical analysis. The difference between them being the difference between astronomy and astrology. Technical analysis refers to classic trading strategies of visual patterns in the price charts like double top or bottom, head and shoulders, etc; these patterns and their traders suffer from massive hindsight bias. Feeding prices into computers looking for patterns is not technical analysis unless you've programmed the computer to look for said human found patterns. If the computer is actually searching for real patterns and you're testing them properly with forward tests on fresh data, you're doing quantitative analysis not technical analysis.

TA is practiced by manual traders, Quants are generally automated traders or traders doing proper statistics rather than relying on the visual patterns manual traders think they see.

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#45

Post by gpawl »

joshuamorton wrote: post #43
cromd wrote: post #24

Earlier quoted context omitted.

If it helps anyone, the FiveThirtyEight article describes a scenario where people take a survey about immigration status and voting. Most legal citizens will correctly identify themselves, but some will accidentally check the wrong box and say they are an illegal immigrant. If you have a billion citizens and 10 illegal immigrants truly taking the survey, and people check the wrong box 1 in 1000 times, your "percentag…

The other common example of the same phenomena is a test for a deadly genetic defect with a 1% false positive rate. If the incidence of the defect is .01% and you test positive, its actually more likely that you don't have the disease. (although this can be solved with bayesianism over frequentism).

This is a pernicious misunderstanding of "frequentism", often found among people who studied statistics mainly by reading comics.

http://web.archive.org/web/20130117080920/http://andrewgelma...

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#47

Post by gpawl »

amelius wrote: post #6

The whole problem of overfitting or underfitting exists because you're not trying to understand the underlying model, but you're trying to "cheat" by inventing some formula that happens to work in most cases.

In what scenario is it ever possible to understand the underlying model, if the model was naturally occurring and not invented by the scientist?

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#48

Post by 0xbear »

corey_moncure wrote: post #34
cromd wrote: post #24

Earlier quoted context omitted.

If it helps anyone, the FiveThirtyEight article describes a scenario where people take a survey about immigration status and voting. Most legal citizens will correctly identify themselves, but some will accidentally check the wrong box and say they are an illegal immigrant. If you have a billion citizens and 10 illegal immigrants truly taking the survey, and people check the wrong box 1 in 1000 times, your "percentag…

Doesn't this make the assumption that "illegal immigrants" won't check the wrong box, intentionally or by accident?

Illegal immigrants have no business being anywhere near that particular checkbox.

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#49

Post by taeric »

nightski wrote: post #19
taeric wrote: post #4

This seems to ultimately come down to an idea that folks have a hard time shaking. It is entirely possible that you cannot recover the original signal using machine learning. This is, fundamentally, what separates this field from digital sampling. And this is not unique to machine learning, per se. https://fivethirtyeight.com/features/trump-noncitizen-voters... has a great widget that shows that as you get more data,…

The irreducible error is unrelated to the bias/variance trade-off. It's part of the overall error of course, but the bias/variance error is in addition to that. Unless I am misunderstanding your point.

Sorry, I did not mean to be that they were the same. And, to that point, I was greatly projecting based on my beliefs that have grown. The number one thing I keep having to re-stress and learn is that these are probabilistic models.

So, if anything, I only meant they were related in that they are both good targets to internalize when working with ML. If there are better targets, I'm definitely interested in learning more.

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#50

Post by foota »

dopamean wrote: post #26
cromd wrote: post #24

Earlier quoted context omitted.

If it helps anyone, the FiveThirtyEight article describes a scenario where people take a survey about immigration status and voting. Most legal citizens will correctly identify themselves, but some will accidentally check the wrong box and say they are an illegal immigrant. If you have a billion citizens and 10 illegal immigrants truly taking the survey, and people check the wrong box 1 in 1000 times, your "percentag…

If you could find a link to the FiveThirtyEight article I'd really appreciate it. Thanks.

It's linked in the grandparent's comment. https://fivethirtyeight.com/features/trump-noncitizen-voters...
Post reply on HN