Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Less than one”-shot learning

technologyreview.com

Post reply on HN
Print view View original

21–30 of 46 posts

Re: “Less than one”-shot learning

#21

Post by bonoboTP »

codelord wrote: post #13

The title is click-bait. This has been known for several years[1], the technique has little practical value, and the assertion that you can learn from no data is completely false and misleading. The training data was compressed to a few examples. To the journalist: it's OK not to maximize for click-bait when you write an article. [1]: https://www.ttic.edu/dl/dark14.pdf

This is not a magazine article written by a journalist, but a press release written by MIT marketing people to hype up research from their university.

Re: “Less than one”-shot learning

#22

Post by bonoboTP »

echelon wrote: post #12

I've started to view Technology Review as a PR puff piece for MIT. They often overstate claims or leave out critical details. As an example, the media lab is still citing innovation with deep fakes, claiming entirely novel results people are shocked to see. They hype their own researchers even though there are kids on YouTube who that have been making similar content up to a year prior to Technology Review's publicat…

In this case it's clear since this is a publication of MIT and they hype up their own research of course.

But even when you're note reading a university publication/PR piece, you still see this effect. One big player like a famous lab at Stanford or MIT publishes an incremental paper that is a followup on a well known existing research direction, where several groups are working in parallel on very similar things, and then it's presented as if it was some breakthrough and the whole subfield was invented by them right now.

It's very, very hard for outsiders to recognize this and to really understand what the actual incremental step in a particular paper is. Necessarily, when explaining to laypeople, you can only scratch the surface and present the rought idea of a whole big research field, and it gets really murky what is part of the established, pre-existing research field and what is the novel contribution.

I'm sure I fall victim to this when reading outside my expertise as well, i.e. when reading about genetics stuff or quantum computing.

Re: “Less than one”-shot learning

#23

Post by breck »

> ...very different from human learning. A child often needs to see just a few examples of an object, or even only one, before being able to recognize it for life.

I see this a lot. It's completely wrong. I'm not trying to pick on the author here, I think 95%+ of people share this misunderstanding of deep learning.

If you see "only one" horse, say for even a second, you really are seeing a huge number of horses, from various angles, with various shades of lighting. The motions of the horse; the motions of your head (even if slight); the undulations of the light; are generating a much larger number of basically augmented training data. If you look at a horse for a minute it could be the equivalent of training on 1 million images of a horse. I'm not sure the exact OOM, but it's certainly orders of magnitude more than "one" horse.

(Relatedly: Some people say there is an experiment you can conduct at home to see the actual images your brain is training on).

Re: “Less than one”-shot learning

#24

Post by jerry80 »

codelord wrote: post #13

The title is click-bait. This has been known for several years[1], the technique has little practical value, and the assertion that you can learn from no data is completely false and misleading. The training data was compressed to a few examples. To the journalist: it's OK not to maximize for click-bait when you write an article. [1]: https://www.ttic.edu/dl/dark14.pdf

Why not just make one image where the pixel values are the trained network weights? Then you can say you can distill any training set into just one image.

Re: “Less than one”-shot learning

#25

Post by ummonk »

Am I correct that they don't use a test-train split for generating these distilled images? Until you test on new images outside of what is inputted to the distiller, it seems to be a way to just overfit specific images, probably by combining unique elements of each into a single composite image. There are plenty of classical signal processing ways to do this (including just building a composite patchwork quilt).

Re: “Less than one”-shot learning

#26

Post by ummonk »

nharada wrote: post #10

Direct link to paper: https://arxiv.org/pdf/2009.08449.pdf Interesting paper, although the headline is of course sensational. The crux of the paper is that by using "soft labels" (for example a probability distribution rather than one-hot), it's possible to create a decision boundary that encodes more classes than you have examples. In fact, only two examples can be used to encode any finite number of classes. This i…

Right it's just creating a 2D basis space and placing the various labels inside that space.

(And the previous paper was about compositing elements of multiple dataset images into a single training example - likely overfitting).

Re: “Less than one”-shot learning

#27

Post by ummonk »

breck wrote: post #23

> ...very different from human learning. A child often needs to see just a few examples of an object, or even only one, before being able to recognize it for life. I see this a lot. It's completely wrong. I'm not trying to pick on the author here, I think 95%+ of people share this misunderstanding of deep learning. If you see "only one" horse, say for even a second, you really are seeing a huge number of horses, from…

Not really. A child can see a single static picture of an unfamiliar animal and learn to recognize the animal from that picture alone.

Re: “Less than one”-shot learning

#28

Post by vonwoodson »

jerry80 wrote: post #24
codelord wrote: post #13

The title is click-bait. This has been known for several years[1], the technique has little practical value, and the assertion that you can learn from no data is completely false and misleading. The training data was compressed to a few examples. To the journalist: it's OK not to maximize for click-bait when you write an article. [1]: https://www.ttic.edu/dl/dark14.pdf

Why not just make one image where the pixel values are the trained network weights? Then you can say you can distill any training set into just one image.

You just nailed it. The whole thing feels like reinventing the wheel of ML.

Re: “Less than one”-shot learning

#29

Post by jsharf »

alquemist wrote: post #2

The title is misleading. The core technique still uses 60,000 images from MNIST, but 'distills' them into 10 images that contain the information from the original 60,000. The 10 'distilled' images look nothing like digits. Learning a complex model from 10 (later reduced to 2) 'distilled' number arrays is an interesting research idea, but it has little to do with reducing the size of the input dataset. Arguably the he…

To me, this seems similar to training a neural network on 10,000 images and then publishing the weights themselves as a distilled "image" (of course, it's not the weights themselves but a "distillation"). I feel like you could get something similar by taking the trained network and then applying deep-dream techniques for each classification label on a blank canvas.

Re: “Less than one”-shot learning

#30

Post by YeGoblynQueenne »

breck wrote: post #23

> ...very different from human learning. A child often needs to see just a few examples of an object, or even only one, before being able to recognize it for life. I see this a lot. It's completely wrong. I'm not trying to pick on the author here, I think 95%+ of people share this misunderstanding of deep learning. If you see "only one" horse, say for even a second, you really are seeing a huge number of horses, from…

>> If you look at a horse for a minute it could be the equivalent of training on 1 million images of a horse.

If you trained a neural net with 1 million images of the same horse, it would learn to recognise that horse... but no other horse. Neural net datasets try to include as many variants of the target concept as possible in order to capture as many of the common features of instances of that concept as possible. A single horse would not suffice, e.g. 1 million images of a white horse would teach a neural net that horses are only white, etc.

Also, a child can learn to recognise horses from a caricature of a horse- which is a single image of a horse, not 1 million images. Whatever human minds do when they learn to recognise objects, that's not training with big data in real time.

Post reply on HN