21–30 of 41 posts
I've been baffled by this as well. I can understand why deep learning has done well in fields that can roughly be described as sensory perception, but it has never improved on basic random forests or SVMs for problems in my domain. And we have lots of data (at least more than can fit in an R instance). Even taking data size out of the picture, functionally it is not there yet for most tasks. Maybe it will be in the f…
The author explores sample sizes up to 85, and then suggests this is the relevant range except at Google, Amazon, Facebook, etc.
But the VAST majority of people considering deep learning have sample sizes between those extremes. Results on small samples are interesting, but it's disingenuous to market this as typical of the world outside Google.
> But I’ve always thought that the major advantage of using deep learning over simpler models is that if you have a massive amount of data you can fit a massive number of parameters. The major advantage of deep learning is not that it works better on more data. It's that it automatically learns features that would otherwise take expert humans a lot of time and energy to figure out and hardcode into the system.
That's an advantage of convolutional nets. Deep fully connected nets don't do that afaik.
I've been baffled by this as well. I can understand why deep learning has done well in fields that can roughly be described as sensory perception, but it has never improved on basic random forests or SVMs for problems in my domain. And we have lots of data (at least more than can fit in an R instance). Even taking data size out of the picture, functionally it is not there yet for most tasks. Maybe it will be in the f…
What is your domain?
I suspect the reason why deep learning has done so poorly in my domain is that the underlying data is a result of things that are very poorly abstracted as a "function". We have lots of discrete events, stateful buffering, hard non-linearities, discontinuities, numerical bounds, etc. It's more like learning business rules and physical process design than learning a mathematical function. This is part of the reason I don't see deep learning being a holistic solution for self driving cars...once you get past sensory perception and simple 2d path planning, driving is more of a rule based process than anything.
That being said, ML tends to be a pretty niche technique for us anyway. If a process and its components are well known and understood, we tend towards solutions that come from Operations Research over Machine Learning. It is only when things are poorly understood that we use ML (example: predicting product demand fluctuations based on media coverage or predicting truck arrival times given severe weather patterns and traffic backups). PGMs do really well here, but are far more difficult to understand, formulate, and train...for most tasks Random Forests are almost always Good Enough(TM).
The benefits of deep learning have more to do with the number of features than the size of the dataset; e.g when you are dealing with million pixel images, you need a deep net to extract useful higher level features automatically. From there, yes, more data is better, but a better post in this vein would be, "Don't use deep learning your data doesn't have that many features".
Hmm I would refrain from saying DL is useful only when approaching 1 megapixel images. State of the art performance on MNIST is held by a 6 layer convnet (4 layers convolutional, 2 layers FC). MNIST is just 28 x 28 grayscale images, so 768 dimensions. There are many more datasets on the same order of dimensionality. CIFAR 10/100 (32 x 32 pixel images) is also dominated by DL convnets, AFAIK.
There are so many issues with this post, let me enumerate: 1. Straw man tweet by some non-practitioner which is used to set up the straw-man argument. 2. The whole Digits example is ridiculous, statisticians "love" toy problems to prove theorems & make "arguments" etc. ML is empirical and not just the performance but the entire pipeline from data to application matters. Let me illustrate: If your aim is to predict 1…
> 3. How much of this is students doing research where they already have access to big data, which makes sense if your goal is to do deep learning research, vs being given a problem a business wants to solve? Can you make the same statement for the average problem at your average small-medium sized business? Can you really get big data that is relevant to the local, non-chain coffee shop down the street? If you can i…
There's a classical quote from Tukey "The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data." - yes, it's quite likely that an average small-medium sized business has no problems where the possible benefit of ML-driven insights won't match the costs required to analyze whatever data they have.
However, if a small-medium business has some problem with a large enough likely payback to justify making some ML system, it is quite likely that deep learning may be applicable on their data.
A big issue is transfer learning - in many domains while you may have a small amount of data, you'd want a system that has learned to generalize on a huge quantity of similar external data, and just tuned on your data. For example, if a cookie bakery needs analysis of cookie pictures or reviews of cookies, and has limited data samples, it would be reasonable to include e.g. ImageNet data or Amazon review corpus. You'd "teach" the system how pictures/internet reviews/English language/whatever else works on the biggest data available, and just retrain/adapt it to your particular problem afterwards.
The benefits of deep learning have more to do with the number of features than the size of the dataset; e.g when you are dealing with million pixel images, you need a deep net to extract useful higher level features automatically. From there, yes, more data is better, but a better post in this vein would be, "Don't use deep learning your data doesn't have that many features".
If you are Google, Amazon, or Facebook and have near infinite data it makes sense to deep learn. But if you have a more modest sample size you may not be buying any accuracy The author explores sample sizes up to 85, and then suggests this is the relevant range except at Google, Amazon, Facebook, etc. But the VAST majority of people considering deep learning have sample sizes between those extremes. Results on small…
I've been baffled by this as well. I can understand why deep learning has done well in fields that can roughly be described as sensory perception, but it has never improved on basic random forests or SVMs for problems in my domain. And we have lots of data (at least more than can fit in an R instance). Even taking data size out of the picture, functionally it is not there yet for most tasks. Maybe it will be in the f…
Deep learning is already a Scruffy fad. It basically says, "Hey, let's use a really huge hypothesis space of circuits that often includes a heavy prior towards convolutions." Gradient descent is a Neat principle, but the whole point of things like improved training methods, new objective functions, and convolutions was to deal with the exploding-gradient problem.
Deep learning didn't come up with its own Neat principle, it invented Scruffy methods to apply a Neat principle to a really fucking huge hypothesis space, so long as you've got a pretty big dataset.