How to use Deep Learning when you have Limited Data • Hacker News Board

Re: How to use Deep Learning when you have Limited Data

#31

Post by prats226 » Fri, Feb 03, 2017, 3:58 AM UTC

cityhall wrote: post #30

dfan wrote: post #11

Earlier quoted context omitted.

That's what regularization is for. You probably know that too, so pretend that was just for the benefit of the onlookers.

I think his point is no one can tell you from theory which regularization methods to apply to a particular problem to get the best results. You need expert knowledge, experience, and hyperparameter tuning.

Transfer learning helps with overfitting too. It is proven to get more generalized model if you use transfer learning that if you train a model on your own with same data (even with large datasets). You need expertise in deep learning but the good thing is that you don't need a lot of expertise in domain of the problem.

Re: How to use Deep Learning when you have Limited Data

#32

Post by apathy » Fri, Feb 03, 2017, 4:30 AM UTC

elyase wrote: post #12

dfan wrote: post #9

Earlier quoted context omitted.

> which is a weakness in the theory because how can more neurons cause more problems? In exactly the same way that adding more terms to a polynomial fit causes more problems. The is one of the most fundamental results in the theory of statistical learning in general; don't blame Deep Learning for it.

It can be argued that some algorithms like Random Forests don't impose a generalization penalty as you increase the number of parameters (forests).

RF is appallingly difficult to re-use for inference, though. At least with a DNN or CNN you can pop open the hood and see what the model is doing at various points.

Tradeoffs, tradeoffs everywhere. It's almost like traditional mathematical statistics has something to offer them fancy machine learners. (Breiman was a professor of statistics, after all... ahead of his time, but no less a statistician.)

Re: How to use Deep Learning when you have Limited Data

#33

Post by nl » Fri, Feb 03, 2017, 5:28 AM UTC

scottyli wrote: post #19

sarthakjain wrote: post #2

Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunatel…

What are the downsides of transfer learning? How can it fail? And do you just arbitrarily select the "cut off output layer" for the pretrained model when retraining with your own data on new layers?

Transfer learning works pretty well for image classification related tasks.

Some other areas are much more challenging. For example, in natural language processing tasks you will sometimes see some benefit from using pretrained embeddings, but it is very task and model specific. There's some exciting work going on in this area though.

Re: How to use Deep Learning when you have Limited Data

#34

Post by qxcv » Fri, Feb 03, 2017, 5:31 AM UTC

sarthakjain wrote: post #6

chriskanan wrote: post #4

Earlier quoted context omitted.

Is transfer learning really not widely known by people doing AI? In my field, computer vision, it is used by most of the papers in the past three years in CVPR, etc. All of the students that take either my deep learning or my computer vision courses have to do assignments on transfer learning with deep neural networks.

Totally agree, everybody in the industry knows about it. However if you look at https://www.google.com/trends/explore?date=2014-01-01%202017... nobody outside seems to know. I might be wrong but a lot of people outside the ML community seem to be hesitant to using ml because they don't have enough data, trying to remove the misconception if it exists

Google Trends can tell you a lot of things :)

https://www.google.com/trends/explore?date=2014-01-01%202017...

More seriously though: others have pointed out that finetuning is pretty popular in some subfields, but it's just one hammer in a of a whole toolbox of techniques which are necessary to make neural nets train (even when you have a tonne of data). Standardisation, choice of initialisation, and choice of learning rate schedule all come to mind as other factors which seem simple, but which can have a huge impact in practice.

Of course, each tool has its limitations. The most obvious limitation of finetuning is that you need a network that's already been trained on vaguely similar data. Pretraining on ImageNet is probably not going to help you solve problems where the size of objects matters, for example, because most ImageNet performance tends to benefit from scale invariance.

I wish you luck with nanonets.ai, but I think it's irresponsible to market this as the "1 weird trick" to bring data efficiency to neural nets.

Re: How to use Deep Learning when you have Limited Data

#35

Post by prats226 » Fri, Feb 03, 2017, 5:42 AM UTC

nl wrote: post #33

scottyli wrote: post #19

Earlier quoted context omitted.

What are the downsides of transfer learning? How can it fail? And do you just arbitrarily select the "cut off output layer" for the pretrained model when retraining with your own data on new layers?

Transfer learning works pretty well for image classification related tasks. Some other areas are much more challenging. For example, in natural language processing tasks you will sometimes see some benefit from using pretrained embeddings, but it is very task and model specific. There's some exciting work going on in this area though.

Yeah correct. Even with text, there is some exciting work going on. We are in process of building a text based model and should put it up in this week that should work for multiple problems.

Re: How to use Deep Learning when you have Limited Data

#36

Post by Ayyar » Fri, Feb 03, 2017, 6:06 AM UTC

sarthakjain wrote: post #2

Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunatel…

This is Deep Learning 101 level material, rather than an advanced insider technique.

It's even listed in the Tensorflow tutorials: https://codelabs.developers.google.com/codelabs/tensorflow-f...

Re: How to use Deep Learning when you have Limited Data

#37

Post by prats226 » Fri, Feb 03, 2017, 6:19 AM UTC

Ayyar wrote: post #36

sarthakjain wrote: post #2

Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunatel…

This is Deep Learning 101 level material, rather than an advanced insider technique. It's even listed in the Tensorflow tutorials: https://codelabs.developers.google.com/codelabs/tensorflow-f...

Transfer learning is fairly well studied. However we have seen lot of companies facing problems that can be solved using this technique but can't because of lack of knowledge, pretrained models availability, engineering challenges involved. We are just trying to make the process easier for them.

Re: How to use Deep Learning when you have Limited Data

#38

Post by deepnotderp » Fri, Feb 03, 2017, 7:01 AM UTC

sarthakjain wrote: post #2

Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunatel…

I'm inclined to agree with the other commentators here, this is pretty much learnt in "Deep Learning 101" (cs231n)

Re: How to use Deep Learning when you have Limited Data

#39

Post by sarthakjain » Fri, Feb 03, 2017, 7:15 AM UTC

deepnotderp wrote: post #38

sarthakjain wrote: post #2

Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunatel…

I'm inclined to agree with the other commentators here, this is pretty much learnt in "Deep Learning 101" (cs231n)

Great feedback, I really appreciate you taking time to read the post. Though I would like to make a few notes regarding what we are trying to achieve and some improvements we've made:

1) Our target audience is someone who hasn't taken Deep Learning 101 but wants to solve a problem

2) We are focusing on users who don't want to setup their own deep learning machines and don't want to learn how to use tensorflow/kerras/caffe/theano and spend time maintaining their own boxes, they don't want to spend engineering effort in ensuring slas and uptime along with scalability

3) We have made improvements in both the model we use for our product and the way it is retrained. It's not the same as the tensorflow example

4)The model has a different dataset than ImageNet and provides additional value in being better suited to certain tasks.

Worst case we have something nobody wants and that's valuable insight in itself. In the best case we have made something that people learn in Deep Learning 101 that can now be used by anybody without spending time and get straight to solving problems.

Re: How to use Deep Learning when you have Limited Data

#40

Post by sarthakjain » Fri, Feb 03, 2017, 7:39 AM UTC

Ayyar wrote: post #36

sarthakjain wrote: post #2

Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunatel…

This is Deep Learning 101 level material, rather than an advanced insider technique. It's even listed in the Tensorflow tutorials: https://codelabs.developers.google.com/codelabs/tensorflow-f...

While I agree that you could build a demo in Deep Learning 101 that could work for some small set of examples, I disagree that this is 101 level material.

Facebook just released: https://techcrunch.com/2017/02/02/facebooks-ai-unlocks-the-a...

You could also call this Deep Learning 101. But it really isn't because building a usable platform that works at scale actually delivers performance and solves problems is a lot tougher than what can be taught in an intro to Deep Learning 101 course.