Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning Crash Course: The Bias-Variance Dilemma

ml.berkeley.edu

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#31

Post by creeble »

kensoh wrote: post #30
jorgemf wrote: post #28

Earlier quoted context omitted.

trading systems are very difficult to model because there are more variable than you can imagine. So basically you are creating a model with less information from the domain than you rather need. But this is not the case in all domain. With AlphaGo there are several things you have to consider. It is not only using deep learning it also uses monte carlo tree search, basically this algorithms is good at exploring sear…

For the branch of trading technical analysis which assumes that all information is factored into price and price action, the variables will drastically be reduced probably to something like volume, low, high, open, close etc. That makes it ideal to develop algos. But as the fundamentalists would say, assuming that everything (demand from insider news etc) has been factored into price is an overly simplistic way of lo…

I also think that, even if all the information is efficiently baked into price, price isn't the predictor - the information doing the baking is. If you use the "predictant" as the predictor, you won't get very good results.

Not to say that people don't, but as mentioned, they don't last. How long they do last is an even more difficult prediction problem.

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#32

Post by jorgemf »

kensoh wrote: post #30
jorgemf wrote: post #28

Earlier quoted context omitted.

trading systems are very difficult to model because there are more variable than you can imagine. So basically you are creating a model with less information from the domain than you rather need. But this is not the case in all domain. With AlphaGo there are several things you have to consider. It is not only using deep learning it also uses monte carlo tree search, basically this algorithms is good at exploring sear…

For the branch of trading technical analysis which assumes that all information is factored into price and price action, the variables will drastically be reduced probably to something like volume, low, high, open, close etc. That makes it ideal to develop algos. But as the fundamentalists would say, assuming that everything (demand from insider news etc) has been factored into price is an overly simplistic way of lo…

Yes I meant that.

I find very interesting how some trade algorithms are using deep learning to extract features form news apart from the variables you mentioned.

About the search, in case you are interested. You have the basic tree search, then you have A* (a-start) which uses an heuristic to decide which is the next node to expand (very used in path finding in games, where the heuristic is the euclidean distance to the target point). For games you use a search where in one state you maximize the heuristic (your move) and in the next one you minimize it (your opponent move) (sorry I don't remember the name). And monte carlo tree search what it does is to no evaluate some states, it just do some random moves and evaluates the final state, this way it tries to improve the exploration/exploitation of the search (and works pretty well).

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#33

Post by mljoe »

taeric wrote: post #4

This seems to ultimately come down to an idea that folks have a hard time shaking. It is entirely possible that you cannot recover the original signal using machine learning. This is, fundamentally, what separates this field from digital sampling. And this is not unique to machine learning, per se. https://fivethirtyeight.com/features/trump-noncitizen-voters... has a great widget that shows that as you get more data,…

Fundamentally this is related to the induction fallacy. The doge meme image actually illustrates it pretty well.

https://en.wikipedia.org/wiki/Problem_of_induction

Another closely related thing is the No Free Lunch Theorem.

https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op...

These are concepts I believe are very important to internalize if you work with machine learning. Fundamentally we are making predictions (ie. guessing) on the nature of entirely unknown information. So there is a certain inherent impossibility to the task in the general sense. It shouldn't always work.

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#34

Post by corey_moncure »

cromd wrote: post #24
taeric wrote: post #4

This seems to ultimately come down to an idea that folks have a hard time shaking. It is entirely possible that you cannot recover the original signal using machine learning. This is, fundamentally, what separates this field from digital sampling. And this is not unique to machine learning, per se. https://fivethirtyeight.com/features/trump-noncitizen-voters... has a great widget that shows that as you get more data,…

If it helps anyone, the FiveThirtyEight article describes a scenario where people take a survey about immigration status and voting. Most legal citizens will correctly identify themselves, but some will accidentally check the wrong box and say they are an illegal immigrant. If you have a billion citizens and 10 illegal immigrants truly taking the survey, and people check the wrong box 1 in 1000 times, your "percentag…

Doesn't this make the assumption that "illegal immigrants" won't check the wrong box, intentionally or by accident?

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#35

Post by Double_Cast »

corey_moncure wrote: post #34
cromd wrote: post #24

Earlier quoted context omitted.

If it helps anyone, the FiveThirtyEight article describes a scenario where people take a survey about immigration status and voting. Most legal citizens will correctly identify themselves, but some will accidentally check the wrong box and say they are an illegal immigrant. If you have a billion citizens and 10 illegal immigrants truly taking the survey, and people check the wrong box 1 in 1000 times, your "percentag…

Doesn't this make the assumption that "illegal immigrants" won't check the wrong box, intentionally or by accident?

citizens labeled illegals: 1,000,000

illegals labeled citizens: 0.01

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#36

Post by sgt101 »

ssivark wrote: post #14
amelius wrote: post #6

The whole problem of overfitting or underfitting exists because you're not trying to understand the underlying model, but you're trying to "cheat" by inventing some formula that happens to work in most cases.

That's is a great and pithy characterization. To expand on that a little: If there existed an underlying model with a finite number of parameters, then you could potentially find it will a finite number of measurements. But if there is no underlying model, the only viable approach is using non-parametric estimation. If you try to truncate a non-parametric model to a finite/fixed dimensional model, you are introducing…

Yes, interestingly the idea of preference for simplicity is a medieval invention - not a universal truth. It served the rational program well for eight hundred years, but now we have infernal engines we can abandon it. I think that the late enlightenment test of theories that generate a clutch of non trivial testable predictions is going to prove more useful in the future.

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#37

Post by sgt101 »

petters wrote: post #12
amelius wrote: post #6

The whole problem of overfitting or underfitting exists because you're not trying to understand the underlying model, but you're trying to "cheat" by inventing some formula that happens to work in most cases.

Yes, this is both uninteresting and true. The whole field of ML exists precisely because many things are too complex to model directly.

too complex for humans to create the models unassisted?

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#38

Post by [deleted] »

corey_moncure wrote: post #34
cromd wrote: post #24

Earlier quoted context omitted.

If it helps anyone, the FiveThirtyEight article describes a scenario where people take a survey about immigration status and voting. Most legal citizens will correctly identify themselves, but some will accidentally check the wrong box and say they are an illegal immigrant. If you have a billion citizens and 10 illegal immigrants truly taking the survey, and people check the wrong box 1 in 1000 times, your "percentag…

Doesn't this make the assumption that "illegal immigrants" won't check the wrong box, intentionally or by accident?

[deleted]

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#39

Post by sgt101 »

platz wrote: post #20
pakl wrote: post #10

Earlier quoted context omitted.

May I ask how you reached this insight? What field do you work in?

classic statistics is much more interested in the explanatory power of models to describe phenomena. ML is mainly interested in prediction (correlation instead of causation), typically over some data that just fell in your lap.

I simply don't recognise that characterisation of ML. I think that "data driven AI" fits far better. ML emerged in a number of ways over the years, but a strong driver of the last iteration was the knowledge engineering bottleneck encountered in fifth generation computing and surrounding the demise of the last turing center.

I invite you to read Chris Bishop's or Stephen Muggleton's books.

Anyone who works with data will find it hard to imagine data that "fell into your lap", all the data I've ever used successfully required slogging and grinding.

Re: Machine Learning Crash Course: The Bias-Variance Dilemma

#40

Post by sgt101 »

banned1 wrote: post #29
gaius wrote: post #23

Earlier quoted context omitted.

In a sense Data Science is like the Cult of the MBA. MBAs believe a trained manager can manage anything because management skills are generic. A data scientist believes they can analyse anything because analysis is generic. Both fail in the real world because they discount domain knowledge.

Is there a field that does not discount domain knowledge? Or is that just "judgment" and custom analysis? I am trying to understand how all fields map together. Thank you.

The divisions are very confused. I think that sensible people all wish to use domain knowledge if possible. There are two tiers of this, firstly the use of domain knowledge in the manual or procedural construction of the insight system. Secondly the use of formalised knowledge in the creation of models that can then be fused with data.

The first case is where data science has got a bad name; people swing into domains and companies full of cocksure ideas, produce insights that are risible or obvious and get ejected. Sometimes it takes years for sufficient knowledge to be acquired by analysts to deal with difficult domains.

Lots of people use Bayesian inference to do the second. Tools like Stan and PyMC3 are really popular and effective.

Post reply on HN