Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to use Deep Learning when you have Limited Data

medium.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: How to use Deep Learning when you have Limited Data

#21

Post by averagewall »

sarthakjain wrote: post #6
chriskanan wrote: post #4

Earlier quoted context omitted.

Is transfer learning really not widely known by people doing AI? In my field, computer vision, it is used by most of the papers in the past three years in CVPR, etc. All of the students that take either my deep learning or my computer vision courses have to do assignments on transfer learning with deep neural networks.

Totally agree, everybody in the industry knows about it. However if you look at https://www.google.com/trends/explore?date=2014-01-01%202017... nobody outside seems to know. I might be wrong but a lot of people outside the ML community seem to be hesitant to using ml because they don't have enough data, trying to remove the misconception if it exists

That graph might be more because it's not as exciting. Google searches are probably dominated by hobbyists and casually interested people. If they're not trying to achieve a specific goal, then they might prefer to work out the basics and make it themselves instead of just taking part of someone else's work and reusing it. If you were going to do that for fun, why not go the whole hog and reuse an entire pretrained network?

Personally, I'm a hobbyist and I don't want to know about these shortcuts until I start to need them - which is a stage I might never reach. People who've progressed far enough to need them are probably far fewer than those who are just curious what these words mean.

Another possibility is the words "transfer learning" might be more generally meaningful outside the ML field than the other search terms on the graph, so most of the searches for it are really schoolteachers or something else.

Re: How to use Deep Learning when you have Limited Data

#22

Post by kyloon »

chriskanan wrote: post #4
sarthakjain wrote: post #2

Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunatel…

Is transfer learning really not widely known by people doing AI? In my field, computer vision, it is used by most of the papers in the past three years in CVPR, etc. All of the students that take either my deep learning or my computer vision courses have to do assignments on transfer learning with deep neural networks.

It is probably transfer learning has become more of a technique used for training neural network models (such as Adam being one of the most commonly used optimizer) whereas most of the excitement stems from new and/or complex neural network architectures rather than the technique or tools that made training those neural networks possible.

Re: How to use Deep Learning when you have Limited Data

#23

Post by prats226 »

kyloon wrote: post #22
chriskanan wrote: post #4

Earlier quoted context omitted.

Is transfer learning really not widely known by people doing AI? In my field, computer vision, it is used by most of the papers in the past three years in CVPR, etc. All of the students that take either my deep learning or my computer vision courses have to do assignments on transfer learning with deep neural networks.

It is probably transfer learning has become more of a technique used for training neural network models (such as Adam being one of the most commonly used optimizer) whereas most of the excitement stems from new and/or complex neural network architectures rather than the technique or tools that made training those neural networks possible.

True that most excitement in the field has been in finding new architectures and building general purpose AI. However I would like to point out that people often underestimate potential of transfer learning. We have started solving real problems companies are facing right now because of lack of enough data to be able to train an accurate model themselves.

Re: How to use Deep Learning when you have Limited Data

#24

Post by sarthakjain »

averagewall wrote: post #21
sarthakjain wrote: post #6

Earlier quoted context omitted.

Totally agree, everybody in the industry knows about it. However if you look at https://www.google.com/trends/explore?date=2014-01-01%202017... nobody outside seems to know. I might be wrong but a lot of people outside the ML community seem to be hesitant to using ml because they don't have enough data, trying to remove the misconception if it exists

That graph might be more because it's not as exciting. Google searches are probably dominated by hobbyists and casually interested people. If they're not trying to achieve a specific goal, then they might prefer to work out the basics and make it themselves instead of just taking part of someone else's work and reusing it. If you were going to do that for fun, why not go the whole hog and reuse an entire pretrained n…

Spot on. If you take that argument a step further that means an average developer who is not a data scientist or ml researcher might not know about it. Which implies a super easy to use dead simple technique which is used by most researchers is not available to the common developer even though it is easy enough for them to use.

Re: How to use Deep Learning when you have Limited Data

#25

Post by [deleted] »

averagewall wrote: post #21
sarthakjain wrote: post #6

Earlier quoted context omitted.

Totally agree, everybody in the industry knows about it. However if you look at https://www.google.com/trends/explore?date=2014-01-01%202017... nobody outside seems to know. I might be wrong but a lot of people outside the ML community seem to be hesitant to using ml because they don't have enough data, trying to remove the misconception if it exists

That graph might be more because it's not as exciting. Google searches are probably dominated by hobbyists and casually interested people. If they're not trying to achieve a specific goal, then they might prefer to work out the basics and make it themselves instead of just taking part of someone else's work and reusing it. If you were going to do that for fun, why not go the whole hog and reuse an entire pretrained n…

[deleted]

Re: How to use Deep Learning when you have Limited Data

#26

Post by sarthakjain »

kyloon wrote: post #22
chriskanan wrote: post #4

Earlier quoted context omitted.

Is transfer learning really not widely known by people doing AI? In my field, computer vision, it is used by most of the papers in the past three years in CVPR, etc. All of the students that take either my deep learning or my computer vision courses have to do assignments on transfer learning with deep neural networks.

It is probably transfer learning has become more of a technique used for training neural network models (such as Adam being one of the most commonly used optimizer) whereas most of the excitement stems from new and/or complex neural network architectures rather than the technique or tools that made training those neural networks possible.

Your right about excitement been generated from new techniques in the research community. Also if you look at Prisma, Google photos or any number of applications that have gained mass popularity. They are just implementations of existing networks and techniques. The paper on style transfer made Prisma possible, commercialization usually has been a few steps behind research.

Re: How to use Deep Learning when you have Limited Data

#28

Post by prats226 »

iraphael wrote: post #27

It should be noted that transfer learning is an umbrella term for many ideas that revolve around transferring what one model has learnt into another model. The method described here is a type of transfer learning called fine tuning.

Correct. There are multiple different ways to transfer knowledge in between tasks. We are talking here about transfer learning with deep neural networks where it is proven to work with several advantages over training a model end to end on your own. There are multiple decisions you need to make even with transfer learning like which layer to use for transfer, how much fine-tuning should be done, based on how much data you have which we are trying to automate.

Re: How to use Deep Learning when you have Limited Data

#29

Post by sarthakjain »

iraphael wrote: post #27

It should be noted that transfer learning is an umbrella term for many ideas that revolve around transferring what one model has learnt into another model. The method described here is a type of transfer learning called fine tuning.

Yes transfer learning is a fairly umbrella term encompassing a lot of different approaches. We tried to give an example of the one most commonly used in NNs almost exclusively with regards to feature extraction. Do you have some resource that lists a variety of transfer learning approaches? Happy to work with you in creating a aggregated list.

Re: How to use Deep Learning when you have Limited Data

#30

Post by cityhall »

dfan wrote: post #11
amelius wrote: post #10

Earlier quoted context omitted.

Yes I know, it was a rhetorical question. Imho, if having more parameters causes problems, then the system should simply not use those extra parameters. But the theory is not there yet.

That's what regularization is for. You probably know that too, so pretend that was just for the benefit of the onlookers.

I think his point is no one can tell you from theory which regularization methods to apply to a particular problem to get the best results. You need expert knowledge, experience, and hyperparameter tuning.
Post reply on HN