Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Less than one”-shot learning

technologyreview.com

Post reply on HN
Print view View original

31–40 of 46 posts

Re: “Less than one”-shot learning

#31

Post by dmurray »

bonoboTP wrote: post #22
echelon wrote: post #12

I've started to view Technology Review as a PR puff piece for MIT. They often overstate claims or leave out critical details. As an example, the media lab is still citing innovation with deep fakes, claiming entirely novel results people are shocked to see. They hype their own researchers even though there are kids on YouTube who that have been making similar content up to a year prior to Technology Review's publicat…

In this case it's clear since this is a publication of MIT and they hype up their own research of course. But even when you're note reading a university publication/PR piece, you still see this effect. One big player like a famous lab at Stanford or MIT publishes an incremental paper that is a followup on a well known existing research direction, where several groups are working in parallel on very similar things, an…

> I'm sure I fall victim to this when reading outside my expertise as well, i.e. when reading about genetics stuff or quantum computing.

I try to avoid this by focusing on the content, not who the researchers are. If they can do some cool new thing with mosquito genetics, now I know that. I don't need to know whether the specific paper being hyped is novel or whether 95% of the content was proved by someone else: that's a task for the Nobel committee and I wouldn't recognise the researchers' names again anyway.

Re: “Less than one”-shot learning

#32

Post by dmurray »

YeGoblynQueenne wrote: post #30
breck wrote: post #23

> ...very different from human learning. A child often needs to see just a few examples of an object, or even only one, before being able to recognize it for life. I see this a lot. It's completely wrong. I'm not trying to pick on the author here, I think 95%+ of people share this misunderstanding of deep learning. If you see "only one" horse, say for even a second, you really are seeing a huge number of horses, from…

>> If you look at a horse for a minute it could be the equivalent of training on 1 million images of a horse. If you trained a neural net with 1 million images of the same horse, it would learn to recognise that horse... but no other horse. Neural net datasets try to include as many variants of the target concept as possible in order to capture as many of the common features of instances of that concept as possible.…

Transfer learning is a thing though. If you train your network with thousands of pictures of cats, dogs, chickens, birds, goats but no horses, it's possible you can teach it to recognise a horse with a single additional image.

Not sure exactly how good the state of the art is compared to human children on that task, but it can't be too far off.

Re: “Less than one”-shot learning

#33

Post by smt1 »

Remember that any meta-virtualization (aka software-defined anything, but is really figuring how how to use less energy to control hardware with better regularity). Remember all 1-bit ReLU (linear non-linear rectifier) are just are one way to think about the same thing as say a dirac function using the language of mathmatical physics. Better to pretend that something like all ad networks or dynamic pieces of web browsers could be influenced to do stuff using other pieces of hardware influenced software that apparently already coded to do that. Solve the "well known" abc or apple google microsoft (lua) = use the wisdom of the crowds to look at symbolic differentiation. Or look at it in 2 elementary bit boolean l-r modular problems so solve the calculus of arbitrary excitations/fluxtions in inductive-coinductive in the calculus of categories of ends and coends using modern elementary mathmatical physics just by using noetherian algebras (understsnding why things don't commute rather than associative properties create transitive closures). Understand why noether's theorem and abelian categories are more powerful aways of computing and energy saving by energy scavenging power. Sorry for the sameless plugin and hit me up at sashmit@gmail.com if you want to know more or collaborate. All this can be understood more clearly by treating neural networks as more elementary computer vision applied to more mature algos with more ethical ways of using it.

Re: “Less than one”-shot learning

#34

Post by breck »

ummonk wrote: post #27
breck wrote: post #23

> ...very different from human learning. A child often needs to see just a few examples of an object, or even only one, before being able to recognize it for life. I see this a lot. It's completely wrong. I'm not trying to pick on the author here, I think 95%+ of people share this misunderstanding of deep learning. If you see "only one" horse, say for even a second, you really are seeing a huge number of horses, from…

Not really. A child can see a single static picture of an unfamiliar animal and learn to recognize the animal from that picture alone.

No, they can’t. Flash an image of an animal for a microsecond in front of a child. They won’t “see” anything. One needs to “see” many, many versions of something before you “see” anything at all, nevermind learn something.

Re: “Less than one”-shot learning

#35

Post by breck »

YeGoblynQueenne wrote: post #30
breck wrote: post #23

> ...very different from human learning. A child often needs to see just a few examples of an object, or even only one, before being able to recognize it for life. I see this a lot. It's completely wrong. I'm not trying to pick on the author here, I think 95%+ of people share this misunderstanding of deep learning. If you see "only one" horse, say for even a second, you really are seeing a huge number of horses, from…

>> If you look at a horse for a minute it could be the equivalent of training on 1 million images of a horse. If you trained a neural net with 1 million images of the same horse, it would learn to recognise that horse... but no other horse. Neural net datasets try to include as many variants of the target concept as possible in order to capture as many of the common features of instances of that concept as possible.…

> Also, a child can learn to recognise horses from a caricature of a horse

The way the chain of connection goes from the eye through the brain is like a tree, and when a child "sees" a horse, a number of those pathways are being activated, some of which are more general, and some of which are specific.

> which is a single image of a horse,

Although it may be a "single" image in the sense that is a single file on disk or a single printed image, a child is not seeing a single image. They are seeing a rushing river of horses, even if from just that 1 static image. Think of the 60hz refresh rate of your monitor. A child is seeing at least 60 "images per second", and likely many, many times more.

Re: “Less than one”-shot learning

#36

Post by ummonk »

breck wrote: post #34
ummonk wrote: post #27

Earlier quoted context omitted.

Not really. A child can see a single static picture of an unfamiliar animal and learn to recognize the animal from that picture alone.

No, they can’t. Flash an image of an animal for a microsecond in front of a child. They won’t “see” anything. One needs to “see” many, many versions of something before you “see” anything at all, nevermind learn something.

That isn’t seeing many versions of the animal. It is merely processing the one training sample over several seconds, because it takes a few hundred milliseconds for the human brain and visual system to process an image, and several more seconds for the brain to analyze and memorize key features of the sample.

Standard machine learning systems don’t work like this. No matter how much time they’re allowed to process a single sample image, they are unable to learn to classify it. There are of course machine learning systems that work more like humans (facial recognition is a notable example), but they’re the exception not the norm.

Re: “Less than one”-shot learning

#37

Post by heyitsguay »

ampdepolymerase wrote: post #6
alquemist wrote: post #2

The title is misleading. The core technique still uses 60,000 images from MNIST, but 'distills' them into 10 images that contain the information from the original 60,000. The 10 'distilled' images look nothing like digits. Learning a complex model from 10 (later reduced to 2) 'distilled' number arrays is an interesting research idea, but it has little to do with reducing the size of the input dataset. Arguably the he…

It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future neural networks on your smart car having "upgrade packs" or country specific data that can be used to fine tune the main network in a matter of minutes) A high-level form of patch-and-diff for networks. There is probably a ML Ops startup opportunity somewhere i…

I don't think this changes anything. Deployed networks typically use only inference from pretrained weights, and those weights are what get transferred for model "updates". You can have all your devices using weight array W0 for a neural net architecture, spend a million compute hours training that net on cutting edge systems to produce a much better weight array W1, then upgrade all the deployed devices by sending them W1 which will be the same size as W0.

Re: “Less than one”-shot learning

#38

Post by ampdepolymerase »

heyitsguay wrote: post #37
ampdepolymerase wrote: post #6

Earlier quoted context omitted.

It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future neural networks on your smart car having "upgrade packs" or country specific data that can be used to fine tune the main network in a matter of minutes) A high-level form of patch-and-diff for networks. There is probably a ML Ops startup opportunity somewhere i…

I don't think this changes anything. Deployed networks typically use only inference from pretrained weights, and those weights are what get transferred for model "updates". You can have all your devices using weight array W0 for a neural net architecture, spend a million compute hours training that net on cutting edge systems to produce a much better weight array W1, then upgrade all the deployed devices by sending t…

For aerospace applications where bandwidth is severely limited, a hundred megabyte NN is a lot.

Re: “Less than one”-shot learning

#39

Post by breck »

dmurray wrote: post #32
YeGoblynQueenne wrote: post #30

Earlier quoted context omitted.

>> If you look at a horse for a minute it could be the equivalent of training on 1 million images of a horse. If you trained a neural net with 1 million images of the same horse, it would learn to recognise that horse... but no other horse. Neural net datasets try to include as many variants of the target concept as possible in order to capture as many of the common features of instances of that concept as possible.…

Transfer learning is a thing though. If you train your network with thousands of pictures of cats, dogs, chickens, birds, goats but no horses, it's possible you can teach it to recognise a horse with a single additional image. Not sure exactly how good the state of the art is compared to human children on that task, but it can't be too far off.

Ah, the transfer learning is the more important point, dominating my point on augmentation.

I can show a 2 year old a single picture of a horse and have them learn it, but only after training them on 5 billion pictures of non-horses.

Re: “Less than one”-shot learning

#40

Post by breck »

ummonk wrote: post #36
breck wrote: post #34

Earlier quoted context omitted.

No, they can’t. Flash an image of an animal for a microsecond in front of a child. They won’t “see” anything. One needs to “see” many, many versions of something before you “see” anything at all, nevermind learn something.

That isn’t seeing many versions of the animal. It is merely processing the one training sample over several seconds, because it takes a few hundred milliseconds for the human brain and visual system to process an image, and several more seconds for the brain to analyze and memorize key features of the sample. Standard machine learning systems don’t work like this. No matter how much time they’re allowed to process a…

All machine learning models that I've trained have involved a data augmentation step. These are artificial augmentations though, so not as effective as more real data. Looking at the same horse live from 100 different perspectives might be better for learning what a horse is than training on 100 static images of different horses.
Post reply on HN