Don't use deep learning when your data isn't that big • Hacker News Board

Re: Don't use deep learning when your data isn't that big

#11

Post by IanCal » Wed, May 31, 2017, 6:18 PM UTC

jclos wrote: post #10

anurag wrote: post #8

> But I’ve always thought that the major advantage of using deep learning over simpler models is that if you have a massive amount of data you can fit a massive number of parameters. The major advantage of deep learning is not that it works better on more data. It's that it automatically learns features that would otherwise take expert humans a lot of time and energy to figure out and hardcode into the system.

That's an advantage of convolutional nets. Deep fully connected nets don't do that afaik.

They do, at least as far as I understand the statement. Historically the big benefit was training them layer by layer, which was like training a feature detector then a feature of features detector etc. If that's still how they're trained (been nearly a decade for me now) then they discover features rather than you engineering them.

This meant that you could train on large unlabelled data and then small amounts of labelled data.

Re: Don't use deep learning when your data isn't that big

#12

Post by saosebastiao » Wed, May 31, 2017, 6:18 PM UTC

I've been baffled by this as well. I can understand why deep learning has done well in fields that can roughly be described as sensory perception, but it has never improved on basic random forests or SVMs for problems in my domain. And we have lots of data (at least more than can fit in an R instance).

Even taking data size out of the picture, functionally it is not there yet for most tasks. Maybe it will be in the future, but the big problem with it is that with n neurons, you have n^n possible topologies, and finding the right neural topology is a major optimization problem that we're only barely learning basic human heuristics for.

I'm willing to bet the deep learning thing is just one more Neat fad that will eventually cause disillusionment at its lack of results, reverting us back to the Scruffy view that intelligence is far too complex to be described holistically by small sets of simple algorithms. The great thing about the Scruffy philosophy is that it isn't derogatory...deep learning will always have a place as a tool in its tool set. It merely doesn't hold unreasonable expectations.

Re: Don't use deep learning when your data isn't that big

#13

Post by [deleted] » Wed, May 31, 2017, 6:20 PM UTC

saosebastiao wrote: post #12

I've been baffled by this as well. I can understand why deep learning has done well in fields that can roughly be described as sensory perception, but it has never improved on basic random forests or SVMs for problems in my domain. And we have lots of data (at least more than can fit in an R instance). Even taking data size out of the picture, functionally it is not there yet for most tasks. Maybe it will be in the f…

[deleted]

Re: Don't use deep learning when your data isn't that big

#14

Post by sbov » Wed, May 31, 2017, 6:22 PM UTC

aub3bhat wrote: post #7

There are so many issues with this post, let me enumerate: 1. Straw man tweet by some non-practitioner which is used to set up the straw-man argument. 2. The whole Digits example is ridiculous, statisticians "love" toy problems to prove theorems & make "arguments" etc. ML is empirical and not just the performance but the entire pipeline from data to application matters. Let me illustrate: If your aim is to predict 1…

> 3.

How much of this is students doing research where they already have access to big data, which makes sense if your goal is to do deep learning research, vs being given a problem a business wants to solve? Can you make the same statement for the average problem at your average small-medium sized business? Can you really get big data that is relevant to the local, non-chain coffee shop down the street?

If you can it seems like an amazing business opportunity - to bring Google level insights to businesses that don't directly have Google-level data.

Re: Don't use deep learning when your data isn't that big

#15

Post by amelius » Wed, May 31, 2017, 6:23 PM UTC

jclos wrote: post #10

anurag wrote: post #8

> But I’ve always thought that the major advantage of using deep learning over simpler models is that if you have a massive amount of data you can fit a massive number of parameters. The major advantage of deep learning is not that it works better on more data. It's that it automatically learns features that would otherwise take expert humans a lot of time and energy to figure out and hardcode into the system.

That's an advantage of convolutional nets. Deep fully connected nets don't do that afaik.

Loosely speaking, convolutional nets are just a smart way of computing a function that would otherwise take the computational load of a fully connected net.

Re: Don't use deep learning when your data isn't that big

#16

Post by bluetwo » Wed, May 31, 2017, 6:30 PM UTC

anurag wrote: post #8

> But I’ve always thought that the major advantage of using deep learning over simpler models is that if you have a massive amount of data you can fit a massive number of parameters. The major advantage of deep learning is not that it works better on more data. It's that it automatically learns features that would otherwise take expert humans a lot of time and energy to figure out and hardcode into the system.

I don't really think of it as learning. I think of it as statistics on crack.

Re: Don't use deep learning when your data isn't that big

#17

Post by krosaen » Wed, May 31, 2017, 6:36 PM UTC

The benefits of deep learning have more to do with the number of features than the size of the dataset; e.g when you are dealing with million pixel images, you need a deep net to extract useful higher level features automatically. From there, yes, more data is better, but a better post in this vein would be, "Don't use deep learning your data doesn't have that many features".

Re: Don't use deep learning when your data isn't that big

#18

Post by [deleted] » Wed, May 31, 2017, 6:36 PM UTC

sbov wrote: post #14

aub3bhat wrote: post #7

There are so many issues with this post, let me enumerate: 1. Straw man tweet by some non-practitioner which is used to set up the straw-man argument. 2. The whole Digits example is ridiculous, statisticians "love" toy problems to prove theorems & make "arguments" etc. ML is empirical and not just the performance but the entire pipeline from data to application matters. Let me illustrate: If your aim is to predict 1…

> 3. How much of this is students doing research where they already have access to big data, which makes sense if your goal is to do deep learning research, vs being given a problem a business wants to solve? Can you make the same statement for the average problem at your average small-medium sized business? Can you really get big data that is relevant to the local, non-chain coffee shop down the street? If you can i…

[deleted]

Re: Don't use deep learning when your data isn't that big

#19

Post by jclos » Wed, May 31, 2017, 6:40 PM UTC

IanCal wrote: post #11

jclos wrote: post #10

Earlier quoted context omitted.

That's an advantage of convolutional nets. Deep fully connected nets don't do that afaik.

They do, at least as far as I understand the statement. Historically the big benefit was training them layer by layer, which was like training a feature detector then a feature of features detector etc. If that's still how they're trained (been nearly a decade for me now) then they discover features rather than you engineering them. This meant that you could train on large unlabelled data and then small amounts of la…

Yeah now that I think about it my statement didn't make any sense, since each intermediate layer computes a projection of the previous one, which is technically feature learning. I still disagree with the original comment though, because the intermediate representations of the data computed by a fully connected network are nothing like the ones that would be built by a human doing feature engineering. The ones learned by a convolutional layer would be closer to human-understandable features.

Re: Don't use deep learning when your data isn't that big

#20

Post by ovi256 » Wed, May 31, 2017, 6:50 PM UTC

krosaen wrote: post #17

The benefits of deep learning have more to do with the number of features than the size of the dataset; e.g when you are dealing with million pixel images, you need a deep net to extract useful higher level features automatically. From there, yes, more data is better, but a better post in this vein would be, "Don't use deep learning your data doesn't have that many features".

Hmm I would refrain from saying DL is useful only when approaching 1 megapixel images.

State of the art performance on MNIST is held by a 6 layer convnet (4 layers convolutional, 2 layers FC). MNIST is just 28 x 28 grayscale images, so 768 dimensions. There are many more datasets on the same order of dimensionality. CIFAR 10/100 (32 x 32 pixel images) is also dominated by DL convnets, AFAIK.