Don't use deep learning when your data isn't that big
simplystatistics.org
Don't use deep learning when your data isn't that big
1–10 of 41 posts
Re: Don't use deep learning when your data isn't that big
#2Commonly understood in the field^ is that 60,000 examples is the sweet spot for training and validation data, 50k for training 10k for test/validation. This is largely because the MNIST set is exactly that size and is so commonly used successfully. Get very high accuracy and reduces instances of overfitting.
That said you can do a lot with a relatively little set. This 2012 paper puts the range between 80-570 samples [1] again depending on model and required outcomes. Leslie Smith at NRL has been working on this problem as well and showing some great progress on really small sample sets as well.
Major takeaway here is that there is such a thing as too big, and too small of data sets for classification accuracy, but those definitions are rapidly changing.
^Your mileage may vary depending on model, fine tuning, transfer learning etc...
Re: Don't use deep learning when your data isn't that big
#3> For low training set sample sizes it looks like the simpler method (just picking the top 10 and using a linear model) slightly outperforms the more complicated methods.
This is a very bad argument for the given clickbaity headline. A methodology that works well for one dataset with few observations will not necessarily work well for another dataset.
You can do almost whatever you want with small datasets, it's just harder than with big data (and is necessary if obtaining data is expensive, e.g. medical trials). Specifically, you'll want to do bootstrapping to simulate additional data and reduce the uncertainty due to a low amount of data.
The "almost" is that you can't have hundreds of features if you have a small dataset (Curse of Dimensionality: https://en.wikipedia.org/wiki/Curse_of_dimensionality)
Re: Don't use deep learning when your data isn't that big
#4The article is spot on, but also misses a simple thing: like in all hypes, DL hype is built on human irrationality. Most people do not understand DL well (or at all) but they see some high profile teams boasting about their success over and over. So, if I would just use that magical DL tool, just like them, maybe I would do those awesome things. Of course, there is also a simpler, more mundane explanation: beefing up the CV with yet another hyped technology. Hadoop: check; blockchain: check; Deep Learning: check. Maybe even deep learning in the blockchain distributed over a million-node Hadoop cluster. Keep them coming!
Re: Don't use deep learning when your data isn't that big
#5[deleted]
Re: Don't use deep learning when your data isn't that big
#6It's true that simple models often outperform more complex ones on small datasets. But the comparison seems rather unfair in this case: the "deep learning" model employed seems to be a simple feedforward discriminative classifier, and these are known to perform badly on small datasets. There are other "deep learning" models that would likely perform much better on small sample sizes. I've written a blog post about one idea [1]. If you prefer published per-reviewed research (and you should of course) then e.g. Semi-Supervised Learning with Deep Generative Models [2] is a good starting point.
1. http://www.openias.org/hybrid-generative-discriminative
2. https://pdfs.semanticscholar.org/b6b9/39ffc9920cd8521299a6fe...
Re: Don't use deep learning when your data isn't that big
#7There are so many issues with this post, let me enumerate:
1. Straw man tweet by some non-practitioner which is used to set up the straw-man argument.
2. The whole Digits example is ridiculous, statisticians "love" toy problems to prove theorems & make "arguments" etc. ML is empirical and not just the performance but the entire pipeline from data to application matters.
Let me illustrate: If your aim is to predict 1 vs 0 from images of digits. As an ML researcher I would write a program to synthesize images in all different combinations of fonts/font-color/background color/ location available. The data would easily be more than ~100,000 images. At this point one cannot use LASSO on top 10 pixels (due to jittering), and a Deep Models would be necessary. But in reality my model will outperforms because the thinking process as an ML researcher was not to make an "argument" but to "solve" the problem of detecting 1 vs 0.
3. But the biggest flaw is the following argument """The sample size matters. If you are Google, Amazon, or Facebook and have near infinite data it makes sense to deep learn."""
This is an another issue with Biostatisticians (The author of this post is Bio-Stats professor), is that they are fundamentally unable to recognize importance of programming and ability to collect data. Even if you are not Google, Amazon, Facebook you can easily collect data, even labeled data in scale of terabytes can be collected in within days or a week. Every single PhD student I know is not limited by size of the data but rather computational power and storage available to them. I personally have several terabytes of video and data from YFCC 100M that I would love to process and build models on but I am only limited by the computational power & AWS costs. If you want a concrete example, see the Google PlaNet paper [1] I today have enough data (~5 Tb) to replicate it and build open source geolocation model, the only hurdles are storage and computation costs.
Re: Don't use deep learning when your data isn't that big
#8> But I’ve always thought that the major advantage of using deep learning over simpler models is that if you have a massive amount of data you can fit a massive number of parameters.
The major advantage of deep learning is not that it works better on more data. It's that it automatically learns features that would otherwise take expert humans a lot of time and energy to figure out and hardcode into the system.
Re: Don't use deep learning when your data isn't that big
#9The main issue is that the VC dimension of a deep network is very high (iirc it grows proportionally to the number of edges of the network, which grows combinatorically with its depth), and for any dataset smaller than that the network can just learn the dataset and achieve 100% accuracy. However regularizing the network usually solves that problem.
Re: Don't use deep learning when your data isn't that big
#10> But I’ve always thought that the major advantage of using deep learning over simpler models is that if you have a massive amount of data you can fit a massive number of parameters. The major advantage of deep learning is not that it works better on more data. It's that it automatically learns features that would otherwise take expert humans a lot of time and energy to figure out and hardcode into the system.
That's an advantage of convolutional nets. Deep fully connected nets don't do that afaik.