Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Less than one”-shot learning

technologyreview.com

Post reply on HN
Print view View original

41–46 of 46 posts

Re: “Less than one”-shot learning

#41

Post by YeGoblynQueenne »

breck wrote: post #35
YeGoblynQueenne wrote: post #30

Earlier quoted context omitted.

>> If you look at a horse for a minute it could be the equivalent of training on 1 million images of a horse. If you trained a neural net with 1 million images of the same horse, it would learn to recognise that horse... but no other horse. Neural net datasets try to include as many variants of the target concept as possible in order to capture as many of the common features of instances of that concept as possible.…

> Also, a child can learn to recognise horses from a caricature of a horse The way the chain of connection goes from the eye through the brain is like a tree, and when a child "sees" a horse, a number of those pathways are being activated, some of which are more general, and some of which are specific. > which is a single image of a horse, Although it may be a "single" image in the sense that is a single file on disk…

But those 60 images per second are the same static, unchanging caricature. It makes more sense to say that the child sees the same image multiple times than to say she sees multiple images.

So how does the child learn to recognise a fully 3-dimensional real-life horse from a single caricature of a horse seen any number of times? Can you explain?

Also, if you were to train a nerual net with a single insance of a caricature of a horse, even if you copied it a million times to create an example set of a million copies, the neural net would still only be able to elarn how to recognise that particular caricature, if that- it would certainly not be able to extract any useful features to recognise real-life horses from that single caricature.

Re: “Less than one”-shot learning

#42

Post by YeGoblynQueenne »

breck wrote: post #39
dmurray wrote: post #32

Earlier quoted context omitted.

Transfer learning is a thing though. If you train your network with thousands of pictures of cats, dogs, chickens, birds, goats but no horses, it's possible you can teach it to recognise a horse with a single additional image. Not sure exactly how good the state of the art is compared to human children on that task, but it can't be too far off.

Ah, the transfer learning is the more important point, dominating my point on augmentation. I can show a 2 year old a single picture of a horse and have them learn it, but only after training them on 5 billion pictures of non-horses.

But how exactly do you "train" a 2-year old on 5 billion pictuers of non-horses? And at what point does the child learn to recognise images of anything?

I mean, how do you know it takes 5 billion images of non-horses before the child can learn to recognise a horse? Even if the child sees 5 billion images of non-horses, who says that they haven't learned to recognise the non-horses and the horses from the first of those 5 billion images? After all, I don't know that children stay transfixed for hours staring at horses and non-horses while their brains process all those billions of images until they finally get it.

I think you are making an effort to explain away something that is not very well understood by modern science, by analogy with machine learning, even though there is no good reason to suppose that the two are connected in any way.

Re: “Less than one”-shot learning

#43

Post by breck »

YeGoblynQueenne wrote: post #42
breck wrote: post #39

Earlier quoted context omitted.

Ah, the transfer learning is the more important point, dominating my point on augmentation. I can show a 2 year old a single picture of a horse and have them learn it, but only after training them on 5 billion pictures of non-horses.

But how exactly do you "train" a 2-year old on 5 billion pictuers of non-horses? And at what point does the child learn to recognise images of anything? I mean, how do you know it takes 5 billion images of non-horses before the child can learn to recognise a horse? Even if the child sees 5 billion images of non-horses, who says that they haven't learned to recognise the non-horses and the horses from the first of tho…

24 hours a day times 3600 seconds per hour times 60hz times 1000 = approximately 5 billion.

Just a ball park figure for how many “pictures” they have seen. Probably off by a few OOM, but point holds.

I think there is a very good reason to think machine learning and human learning are connected. First, I’d say it’s largely true the people who have made the most impact in the former have all studied the way the mind works (as far as science allows), and tried to emulate that in machinery. Second, experimentally the phenomena observed are increasingly similar (deep dream, for example).

Re: “Less than one”-shot learning

#44

Post by ummonk »

breck wrote: post #40
ummonk wrote: post #36

Earlier quoted context omitted.

That isn’t seeing many versions of the animal. It is merely processing the one training sample over several seconds, because it takes a few hundred milliseconds for the human brain and visual system to process an image, and several more seconds for the brain to analyze and memorize key features of the sample. Standard machine learning systems don’t work like this. No matter how much time they’re allowed to process a…

All machine learning models that I've trained have involved a data augmentation step. These are artificial augmentations though, so not as effective as more real data. Looking at the same horse live from 100 different perspectives might be better for learning what a horse is than training on 100 static images of different horses.

Yes, data augmentation is standard. It isn't new data though, just different variations of the same data. You also tend to do multiple training passes over the data, even after augmentation.

None of that is the point though: a child can look at a single static image (not 100 different perspectives - just one perspective) of a horse, and learn to recognize a horse. A standard machine learning model cannot, no matter how much you augment the image.

Re: “Less than one”-shot learning

#45

Post by YeGoblynQueenne »

breck wrote: post #43
YeGoblynQueenne wrote: post #42

Earlier quoted context omitted.

But how exactly do you "train" a 2-year old on 5 billion pictuers of non-horses? And at what point does the child learn to recognise images of anything? I mean, how do you know it takes 5 billion images of non-horses before the child can learn to recognise a horse? Even if the child sees 5 billion images of non-horses, who says that they haven't learned to recognise the non-horses and the horses from the first of tho…

24 hours a day times 3600 seconds per hour times 60hz times 1000 = approximately 5 billion. Just a ball park figure for how many “pictures” they have seen. Probably off by a few OOM, but point holds. I think there is a very good reason to think machine learning and human learning are connected. First, I’d say it’s largely true the people who have made the most impact in the former have all studied the way the mind wo…

I like to quote Yann LeCun on how neural nets work like our minds:

IEEE Spectrum: We read about Deep Learning in the news a lot these days. What’s your least favorite definition of the term that you see in these stories?

Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver.

https://spectrum.ieee.org/automaton/artificial-intelligence/...

Also, like I say above, it doesn't matter how many images a human sees- what matters is how many she needs to see before learnign to recognise a thing. The example of an unchanging, two-dimensional caricature of a horse is evidence enough that, even if we do see billions of "images" as you say (I'm not sure it makes sense to speak of "images" int the sense you use it) we don't need to see all those billions of them before we learn what things look like.

Re: “Less than one”-shot learning

#46

Post by breck »

ummonk wrote: post #44
breck wrote: post #40

Earlier quoted context omitted.

All machine learning models that I've trained have involved a data augmentation step. These are artificial augmentations though, so not as effective as more real data. Looking at the same horse live from 100 different perspectives might be better for learning what a horse is than training on 100 static images of different horses.

Yes, data augmentation is standard. It isn't new data though, just different variations of the same data. You also tend to do multiple training passes over the data, even after augmentation. None of that is the point though: a child can look at a single static image (not 100 different perspectives - just one perspective) of a horse, and learn to recognize a horse. A standard machine learning model cannot, no matter h…

> a child can look at a single static image

But in an important way, they can't.

You need to define "look". How many nanoseconds? How much is the lighting changing during that time? How still is the photo? How still is the person's head? In DL a training example in a batch is perfectly defined in bits. Once you try to define a training example for a human you see that a "single static image" means totally different things. A human seeing a static image is the equivalent of training a model on at least thousands, if not millions+ of training images.

Post reply on HN