How to use Deep Learning when you have Limited Data • Hacker News Board

Re: How to use Deep Learning when you have Limited Data

#51

Post by deepnotderp » Fri, Feb 03, 2017, 7:57 PM UTC

sarthakjain wrote: post #39

deepnotderp wrote: post #38

Earlier quoted context omitted.

I'm inclined to agree with the other commentators here, this is pretty much learnt in "Deep Learning 101" (cs231n)

Great feedback, I really appreciate you taking time to read the post. Though I would like to make a few notes regarding what we are trying to achieve and some improvements we've made: 1) Our target audience is someone who hasn't taken Deep Learning 101 but wants to solve a problem 2) We are focusing on users who don't want to setup their own deep learning machines and don't want to learn how to use tensorflow/kerras/…

Fair enough, that makes sense, but I kind of have an allergic reaction to AI hype after being burned a few times :)

OTOH, your webpage makes it pretty clear what you actually do, so props to you for that!

Re: How to use Deep Learning when you have Limited Data

#52

Post by iraphael » Fri, Feb 03, 2017, 11:54 PM UTC

sarthakjain wrote: post #29

iraphael wrote: post #27

It should be noted that transfer learning is an umbrella term for many ideas that revolve around transferring what one model has learnt into another model. The method described here is a type of transfer learning called fine tuning.

Yes transfer learning is a fairly umbrella term encompassing a lot of different approaches. We tried to give an example of the one most commonly used in NNs almost exclusively with regards to feature extraction. Do you have some resource that lists a variety of transfer learning approaches? Happy to work with you in creating a aggregated list.

Well for starters, fine-tuning can be done in a variety of different ways. You can pretrain your model with a larger, different dataset, or you can train an autoencoder that learns some useful representation of that larger dataset and use the encoder as a base for fine tuning.

Another approach I've seen that was really cool is Model Distillation [0], which is basically the training of a smaller NN with the inputs and outputs of a larger NN (where the output is slightly modified to increase gradients and make training faster).

[0] https://arxiv.org/pdf/1503.02531v1.pdf

Re: How to use Deep Learning when you have Limited Data

#53

Post by amelius » Sat, Feb 04, 2017, 3:03 AM UTC

jorgemf wrote: post #14

amelius wrote: post #7

I think Deep Learning is very frustrating to work with at the moment. First, there is the problem of overfitting, which shows up typically after you've already been training for hours. So you have to tweak things (basically this is just guessing), and start from scratch. If your network has too many neurons, then overfitting may more easily occur, which is a weakness in the theory because how can more neurons cause m…

There are a lot of solutions for the problems you mention. For overfitting you can do data augmentation, normalization, dropout and early stop with the test set. (and probably improve your dataset) More neurons means more parameters to adjust to your data, so overfitting is more likely to happen. It is like interpolation a function, the more parameters you use the more overfitting you have. > if your data is somewhat…

> There are a lot of solutions for the problems you mention.

Yes, that's a problem right there, because often only one solution actually works, and you don't know which one unless you spend hours on training.

Re: How to use Deep Learning when you have Limited Data

#54

Post by jorgemf » Fri, Feb 10, 2017, 5:30 PM UTC

amelius wrote: post #53

jorgemf wrote: post #14

Earlier quoted context omitted.

There are a lot of solutions for the problems you mention. For overfitting you can do data augmentation, normalization, dropout and early stop with the test set. (and probably improve your dataset) More neurons means more parameters to adjust to your data, so overfitting is more likely to happen. It is like interpolation a function, the more parameters you use the more overfitting you have. > if your data is somewhat…

> There are a lot of solutions for the problems you mention. Yes, that's a problem right there, because often only one solution actually works, and you don't know which one unless you spend hours on training.

actually you do all of them, all of them help to overfitting