Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to use Deep Learning when you have Limited Data

medium.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: How to use Deep Learning when you have Limited Data

#11

Post by dfan »

amelius wrote: post #10
dfan wrote: post #9

Earlier quoted context omitted.

> which is a weakness in the theory because how can more neurons cause more problems? In exactly the same way that adding more terms to a polynomial fit causes more problems. The is one of the most fundamental results in the theory of statistical learning in general; don't blame Deep Learning for it.

Yes I know, it was a rhetorical question. Imho, if having more parameters causes problems, then the system should simply not use those extra parameters. But the theory is not there yet.

That's what regularization is for. You probably know that too, so pretend that was just for the benefit of the onlookers.

Re: How to use Deep Learning when you have Limited Data

#12

Post by elyase »

dfan wrote: post #9
amelius wrote: post #7

I think Deep Learning is very frustrating to work with at the moment. First, there is the problem of overfitting, which shows up typically after you've already been training for hours. So you have to tweak things (basically this is just guessing), and start from scratch. If your network has too many neurons, then overfitting may more easily occur, which is a weakness in the theory because how can more neurons cause m…

> which is a weakness in the theory because how can more neurons cause more problems? In exactly the same way that adding more terms to a polynomial fit causes more problems. The is one of the most fundamental results in the theory of statistical learning in general; don't blame Deep Learning for it.

It can be argued that some algorithms like Random Forests don't impose a generalization penalty as you increase the number of parameters (forests).

Re: How to use Deep Learning when you have Limited Data

#14

Post by jorgemf »

amelius wrote: post #7

I think Deep Learning is very frustrating to work with at the moment. First, there is the problem of overfitting, which shows up typically after you've already been training for hours. So you have to tweak things (basically this is just guessing), and start from scratch. If your network has too many neurons, then overfitting may more easily occur, which is a weakness in the theory because how can more neurons cause m…

There are a lot of solutions for the problems you mention. For overfitting you can do data augmentation, normalization, dropout and early stop with the test set. (and probably improve your dataset)

More neurons means more parameters to adjust to your data, so overfitting is more likely to happen. It is like interpolation a function, the more parameters you use the more overfitting you have.

> if your data is somewhat different from your training data in what humans would call an insignificant way, your network may easily start to fail

humans call it insignificant because we have a deep knowledge of a lot of domains, meanwhile a network has been trained for an specific domain. So if you train the network with a distribution and then test it with another distribution it is not going to work. That is like quite obvious I think

Deep learning works incredible well. It works so well that it outperform humans in some domains. So may want to rethink what are you doing, because I think (but I may be wrong) the reason you are failing applying deep learning is something related with your process and not with deep learning

Re: How to use Deep Learning when you have Limited Data

#17

Post by sytelus »

Another less know but really promising approach is program synthesis (also called "program generation"). One can build fairly robust model just using 2-5 examples and that too in just seconds. Implementation of this approach is already shipped in to Excel where you just enter few example of formatting and "Flash Fill" will learn what to do: http://research.microsoft.com/en-us/um/people/sumitg/flashfi...

Paper: http://research.microsoft.com/en-us/um/people/sumitg/pubs/ca...

Re: How to use Deep Learning when you have Limited Data

#18

Post by prats226 »

sytelus wrote: post #17

Another less know but really promising approach is program synthesis (also called "program generation"). One can build fairly robust model just using 2-5 examples and that too in just seconds. Implementation of this approach is already shipped in to Excel where you just enter few example of formatting and "Flash Fill" will learn what to do: http://research.microsoft.com/en-us/um/people/sumitg/flashfi... Paper: http:/…

This is cool. From what I understand from paper, its DSL has set of algorithms as building blocks that learn the input/output function. Deep learning algos are trying to do the same but with more generic blocks where assumption is that a lot of these blocks will be able to learn algorithms too. Deep learning is trying to build with a more generic approach in which transfer learning is helping to reduce number of examples needed by reusing algorithms learned.

Re: How to use Deep Learning when you have Limited Data

#19

Post by scottyli »

sarthakjain wrote: post #2

Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunatel…

What are the downsides of transfer learning? How can it fail?

And do you just arbitrarily select the "cut off output layer" for the pretrained model when retraining with your own data on new layers?

Re: How to use Deep Learning when you have Limited Data

#20

Post by prats226 »

scottyli wrote: post #19
sarthakjain wrote: post #2

Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunatel…

What are the downsides of transfer learning? How can it fail? And do you just arbitrarily select the "cut off output layer" for the pretrained model when retraining with your own data on new layers?

One way it can fail is that your model might overfit your data if number of parameters you are training are way more than data you are using for training and if regularization techniques are not used. We don't arbitrarily cut-off layers from pretrained model. We check at what layer, output features are not specific to problem for which pretrained model was trained. At this layer, you can stitch a nanonet and build a model for your data
Post reply on HN