Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't use deep learning when your data isn't that big

simplystatistics.org

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Don't use deep learning when your data isn't that big

#31

Post by zeroxfe »

AndrewKemendo wrote: post #2

Commonly understood in the field^ is that 60,000 examples is the sweet spot for training and validation data, 50k for training 10k for test/validation. This is largely because the MNIST set is exactly that size and is so commonly used successfully. Get very high accuracy and reduces instances of overfitting. That said you can do a lot with a relatively little set. This 2012 paper puts the range between 80-570 samples…

> Commonly understood in the field^ is that 60,000 examples is the sweet spot for training and validation data, 50k for training 10k for test/validation.

I don't understand how this could be true. Shouldn't the sweet spot be a function of the dimensionality of the data?

Re: Don't use deep learning when your data isn't that big

#32

Post by amelius »

zeroxfe wrote: post #31
AndrewKemendo wrote: post #2

Commonly understood in the field^ is that 60,000 examples is the sweet spot for training and validation data, 50k for training 10k for test/validation. This is largely because the MNIST set is exactly that size and is so commonly used successfully. Get very high accuracy and reduces instances of overfitting. That said you can do a lot with a relatively little set. This 2012 paper puts the range between 80-570 samples…

> Commonly understood in the field^ is that 60,000 examples is the sweet spot for training and validation data, 50k for training 10k for test/validation. I don't understand how this could be true. Shouldn't the sweet spot be a function of the dimensionality of the data?

No, the deep learning network should be smart enough to reduce the data to its essence (and that's what it ultimately does).

If DL would need more training data for higher dimensional inputs, then DL would lose against a simple pattern matching (correlation) algorithm at some point.

Re: Don't use deep learning when your data isn't that big

#33

Post by curiousgal »

krosaen wrote: post #17

The benefits of deep learning have more to do with the number of features than the size of the dataset; e.g when you are dealing with million pixel images, you need a deep net to extract useful higher level features automatically. From there, yes, more data is better, but a better post in this vein would be, "Don't use deep learning your data doesn't have that many features".

>From there, yes, more data is better

Data augmentation is also a thing.

Re: Don't use deep learning when your data isn't that big

#34

Post by elchief »

krosaen wrote: post #17

The benefits of deep learning have more to do with the number of features than the size of the dataset; e.g when you are dealing with million pixel images, you need a deep net to extract useful higher level features automatically. From there, yes, more data is better, but a better post in this vein would be, "Don't use deep learning your data doesn't have that many features".

Of course you need a lot of rows if you have a lot of columns...

Re: Don't use deep learning when your data isn't that big

#35

Post by cityhall »

krosaen wrote: post #17

The benefits of deep learning have more to do with the number of features than the size of the dataset; e.g when you are dealing with million pixel images, you need a deep net to extract useful higher level features automatically. From there, yes, more data is better, but a better post in this vein would be, "Don't use deep learning your data doesn't have that many features".

I'd say the benefits come from the information content of the features or lack thereof. When you have uninformative features like pixel colors or word identities, there's nothing for traditional methods to work with. You have to start with feature engineering and pruning before decision trees or linear classifiers have a chance.

Most of the wins under the "deep learning" umbrella come from extracting meaning from homogenous features like "the pixel at x-2,y+1 has red=123" or "the word at n+1 is 'king'". That's why we see latent variable embeddings like word2vec come from the DL world even though they're not deep.

When you want to include highly informative features in a deep network, it's often better to feed them into a separate logistic model, as shown in the Tensorflow wide-deep tutorials.

Re: Don't use deep learning when your data isn't that big

#36

Post by landon32 »

amelius wrote: post #32
zeroxfe wrote: post #31

Earlier quoted context omitted.

> Commonly understood in the field^ is that 60,000 examples is the sweet spot for training and validation data, 50k for training 10k for test/validation. I don't understand how this could be true. Shouldn't the sweet spot be a function of the dimensionality of the data?

No, the deep learning network should be smart enough to reduce the data to its essence (and that's what it ultimately does). If DL would need more training data for higher dimensional inputs, then DL would lose against a simple pattern matching (correlation) algorithm at some point.

Imagine you are trying to use a neural network to classify single bit data as either being 1 or 0 (I know you obviously don't need a neural net for this, but it's an example). Aside from not needing a very deep network, you would not need much training data.

Then imagine classifying the color of a single pixel as "light" or "dark". There's three dimensions-- red, green, and blue. You would also need much less training data here than if you were trying to train a network to recognize a car, right?

I think this is what zeroxfe is referring to

Re: Don't use deep learning when your data isn't that big

#38

Post by billf1953 »

dbecker wrote: post #22

If you are Google, Amazon, or Facebook and have near infinite data it makes sense to deep learn. But if you have a more modest sample size you may not be buying any accuracy The author explores sample sizes up to 85, and then suggests this is the relevant range except at Google, Amazon, Facebook, etc. But the VAST majority of people considering deep learning have sample sizes between those extremes. Results on small…

Yeah what a dog argument. Basically either shows the author has little comprehension of ml or he happened to design a terrible demonstration of small data issues.

Re: Don't use deep learning when your data isn't that big

#39

Post by mrmaximus »

saosebastiao wrote: post #24
eachro wrote: post #21

Earlier quoted context omitted.

What is your domain?

Transportation, Logistics, Supply Chain Management, (physical) Operations. I suspect the reason why deep learning has done so poorly in my domain is that the underlying data is a result of things that are very poorly abstracted as a "function". We have lots of discrete events, stateful buffering, hard non-linearities, discontinuities, numerical bounds, etc. It's more like learning business rules and physical process…

+1 For real world business problems that I most frequently encounter doing consulting, it's hard to beat Random Forests and/or Gradient Boosting. Truth be told, most business problems I encounter turn out to be largely helped by good old linear models.

Re: Don't use deep learning when your data isn't that big

#40

Post by cttet »

saosebastiao wrote: post #24
eachro wrote: post #21

Earlier quoted context omitted.

What is your domain?

Transportation, Logistics, Supply Chain Management, (physical) Operations. I suspect the reason why deep learning has done so poorly in my domain is that the underlying data is a result of things that are very poorly abstracted as a "function". We have lots of discrete events, stateful buffering, hard non-linearities, discontinuities, numerical bounds, etc. It's more like learning business rules and physical process…

Is there any example of PGM applying to this field?
Post reply on HN