Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MNIST for ML Beginners: The Bayesian Way

alpha-i.co

Post reply on HN
Print view View original

31–37 of 37 posts

Re: MNIST for ML Beginners: The Bayesian Way

#31

Post by fnbr »

kmangutov wrote: post #27
Eliezer wrote: post #22

Never use predicting stock prices as an example of anything. Pick literally anything other than stock prices to display confidence bounds. This helpful example diagram violates the EMH even worse by showing a large predictable directional change. You might as well illustrate physics with a chart of bowling balls falling upward.

Could you elaborate why predicting stock prices is an invalid example?

It's almost impossible, for two reasons:

1) It's incredibly difficult to create strategies that work because there are thousands of highly trained, highly paid people working to create strategies.

2) Any strategy that you do create will self-correct over time and become useless, as strategies find examples where the market has misjudged prices; by exploiting the strategy, the market should, over time, stop misjudging the prices.

The "EMH" that the parent is referring to is the Efficient Market Hypothesis, which states that the market price is the "correct" price, and that it is impossible to predict a better price, as the current price incorporates all possible information.

I, personally, don't subscribe to that version of the EMH; I believe the one implied by 1), which is that it's difficult to compete with the thousands of math PhDs working on Wall Street.

Re: MNIST for ML Beginners: The Bayesian Way

#33

Post by doktrin »

yorwba wrote: post #23
doktrin wrote: post #19

Earlier quoted context omitted.

> This strategy is equally effective for most things in life. Not if we understand 'effective' to also mean 'cost & time effective' You'd learn a lot by building a nuclear reactor from first principles, but it's not the most effective way to develop an intuition about how one operates.

I think you want to talk about whether the strategy is efficient , which I agree it is not. However, if you already tried understanding several general descriptions and it didn't work out, implementing something from scratch is an inefficient but effective way of really grokking it.

> I think you want to talk about whether the strategy is efficient

effective and efficient are synonyms in this context

http://www.thesaurus.com/browse/efficient?s=t

http://www.dictionary.com/browse/efficient?s=t

http://www.dictionary.com/browse/effective?s=t

Re: MNIST for ML Beginners: The Bayesian Way

#34

Post by panarky »

fnbr wrote: post #31
kmangutov wrote: post #27

Earlier quoted context omitted.

Could you elaborate why predicting stock prices is an invalid example?

It's almost impossible, for two reasons: 1) It's incredibly difficult to create strategies that work because there are thousands of highly trained, highly paid people working to create strategies. 2) Any strategy that you do create will self-correct over time and become useless, as strategies find examples where the market has misjudged prices; by exploiting the strategy, the market should, over time, stop misjudging…

> It's almost impossible ...

You'd think so, and yet ML is transforming investment and trading strategies.

From The Economist:

  Castle Ridge Asset Management, a Toronto-based upstart, has achieved
  annual average returns of 32% since its founding in 2013. It
  uses a sophisticated machine-learning system, like those used
  to model evolutionary biology, to make investment decisions.

  It is so sensitive, claims the firm’s chief executive, Adrian
  de Valois-Franklin, that it picked up 24 acquisitions before
  they were even announced.
Source: http://www.economist.com/news/finance-and-economics/21722685...

Re: MNIST for ML Beginners: The Bayesian Way

#35

Post by candiodari »

srean wrote: post #4
harel wrote: post #3

I wish someone would come up with a tutorial for ML for the mathematically challenged. Something more practical and less theoretical.

I understand and am sympathetic towards the sentiment, but honestly, its a bit like asking for a tutorial on swimming that does not involve water. Something like that can be written, but it wont be very useful. Most of the simple stuff, the non-mathematical parts would get automated away. You don't want to be in a position where you are competing with someone's commodity script (sometimes just a for loop), unless the…

Given that the difference between any 2 ML implementations is going to be 2 things, everything else is going to be eliminated by a hyperparameter search. Neither of these require more than basic mathematical knowledge. Certainly nothing that wouldn't be covered in an undergraduate statistics course.

And of course, it's loads and loads of work.

What makes the difference in success for machine learning projects (this is assuming you have some process to avoid screwups in place):

1) quality of how the data is input into the network. You can almost never "usefully" put raw data in front of a neural network. Call it "data represantation" (for instance, for comparing stock prices do a neural net DNN predictor on the prices. Now do the same on the deltas (today - yesterday). Works 1000x better (still not good enough, but very clear difference))

It may not compare to the feature engineering of SVMs, but it's very present (it kinda does imho, but people tend to get very defensive when suggesting that)

2) your cost/loss function. There are tricks like GANs which are simple in concept and avoid some of the issues, but even there you have your image comparator you're still going to need. You can massively improve image comparisons (e.g. mean square difference of the image scaled 4x4, times 1e12, plus same for 16x16 times 1e6, plus mean square difference at real res beats the crap out of just taking mean square difference. Very good results have also been obtained for MNIST by comparing sorted lists of black pixels, instead of actual images)

Many things depend on your cost/loss function, and you especially have the eternal problem : I want it to improve in 10 different ways. How do I balance those well into a single number ? Robot should grab the teddy bear, and shouldn't hit anything else. Those are easy to balance, but how would you balance grabbing the teddy bear at all versus not damaging it ? It will make a huge difference in whether your neural net converges at all.

Re: MNIST for ML Beginners: The Bayesian Way

#36

Post by annnnd »

srean wrote: post #9
jclos wrote: post #8

Earlier quoted context omitted.

Honestly, the best thing you can do is try to implement your own shitty neural net with only Python + Numpy, from scratch, with only a basic understanding of the math. It will make most of the math very concrete very fast.

Cannot upvote this enough. I think this is the only way to really grok backpropagation. The hours of staring at the update formula till your eyes glaze over the subscripts and superscripts and the summations would not give you as good an understanding as implementing a toy neural net with just a single hidden layer. Its actually a whole lot easier than parsing those low-level notation. It can be done better with high…

I respectfully disagree. Of course you need to really understand backpropagation for any advanced stuff, but it is easier to ignore it at the beginning. Take Keras container, copy some MNIST example from somewhere and tweak it. Then, when you have a general feel of how training NNs works, gradually learn about each concept - BP should of course be one of the first. By the time you get into math stuff it will probably make much more sense because you will understand how it applies to your case.

But I guess the approach depends on how you best learn, so there is no wrong answer. Just jump in!

Re: MNIST for ML Beginners: The Bayesian Way

#37

Post by Eliezer »

fnbr wrote: post #31
kmangutov wrote: post #27

Earlier quoted context omitted.

Could you elaborate why predicting stock prices is an invalid example?

It's almost impossible, for two reasons: 1) It's incredibly difficult to create strategies that work because there are thousands of highly trained, highly paid people working to create strategies. 2) Any strategy that you do create will self-correct over time and become useless, as strategies find examples where the market has misjudged prices; by exploiting the strategy, the market should, over time, stop misjudging…

Does anyone believe in the straw form of the EMH? Not I, surely. Standard weak form suffices to imply that nobody should be showing an ML-derived chart showing an expected tripling of price.
Post reply on HN