Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-supervised learning: The dark matter of intelligence

ai.facebook.com

Post reply on HN
Print view View original

21–30 of 76 posts

Re: Self-supervised learning: The dark matter of intelligence

#21

Post by visarga »

iujjkfjdkkdkf wrote: post #3
montenegrohugo wrote: post #2

One of the problems we have to be conscious about with self supervised learning is that the already existing problem of transparency and understandability gets worsened by orders of magnitude. If current billion parameter models are opaque, the result of SSL is, or will be, the equivalent of a black hole. How do you understand what the model has learned? Why has it learned that way? What things could have possible go…

I dont agree with this, either we know what the pretext task was, or we fine tune with labels, but in either case, it's no different to analyze than supervised models. I do agree that more parameters and a bigger training set makes the models more opaque (or at least more expensive to understand), but the self supervision is not the reason.

Or instead of analyzing the model we could analyze the training data for biases. Most bias comes from data.

Re: Self-supervised learning: The dark matter of intelligence

#22

Post by Xplune13 »

Not directly related to the SSL part, but I have a question about how the work on AI is going. - Why are we even trying to create an AGI when we have no idea how it will turn out? - AI in day to day functions can be useful and I'm all for that but what is the point of even trying to give the AI self-awareness?

Re: Self-supervised learning: The dark matter of intelligence

#23

Post by visarga »

just1morething wrote: post #19

If you cut some words out of a newspaper and a person writes words in those blanks then what has happened? If we cut out parts of a painting and a person paints in those blanks then what has happened? How is the person who wrote in the blanks or the person who painted in the blanks changed by writing or painting in the blanks? How are they changed when someone responds to what they've written or painted? How did the…

One could make a similarly long list of poorly understood concepts in psychology.

Re: Self-supervised learning: The dark matter of intelligence

#24

Post by visarga »

Reebz wrote: post #13

State of the art performance is being broken in multiple fields rapidly these days. However, AI explainability has a long way to go. Scaling opaqueness makes this problem worse. Fine tuning labels black-box style is a terrifying concept to most who are working in fields where great risk must be managed to avoid unintended and disparate impact. Facebook making an oopsies suggesting my friends face in a photo instead o…

High risk models should work with humans in the loop, not autonomous.

Re: Self-supervised learning: The dark matter of intelligence

#25

Post by drdeca »

Xplune13 wrote: post #22

Not directly related to the SSL part, but I have a question about how the work on AI is going. - Why are we even trying to create an AGI when we have no idea how it will turn out? - AI in day to day functions can be useful and I'm all for that but what is the point of even trying to give the AI self-awareness?

So it can handle the strategy of making sure no one else makes one which is more harmful. :P

But as a serious answer: of course, there is no major economic use of trying to make sure it has an internal subjective experience, but that’s not what people are aiming for when referring to AGI. The goal is that it be able to accomplish general tasks and goals. Like, what can you do with it? Everything you can do at all. That’s what people are aiming at. What tasks of reasoning and planning could a person do for you? An AGI would, basically by definition, be able to do those same kinds of things. (Of course, something could be an AGI but not as intelligent as a typical human, so long as it could still reason about all the same kinds of things. But at that point it would presumably just be a question of scaling things up.)

Re: Self-supervised learning: The dark matter of intelligence

#26

Post by fantod »

just1morething wrote: post #19

If you cut some words out of a newspaper and a person writes words in those blanks then what has happened? If we cut out parts of a painting and a person paints in those blanks then what has happened? How is the person who wrote in the blanks or the person who painted in the blanks changed by writing or painting in the blanks? How are they changed when someone responds to what they've written or painted? How did the…

If the person who filled in the blanks already had a good grasp of the language, they probably didn't learn much. However, fill-in-the-blank exercises can be a good language learning tool for children as well as adults learning a foreign language.

Re: Self-supervised learning: The dark matter of intelligence

#27

Post by ilaksh »

Self-supervised learning is very promising. But it is strange to me that so much computer vision research has no mention of the 3d structure of objects.

That's why I like things like this https://arxiv.org/abs/2102.12627 "How to represent part-whole hierarchies in a neural network" by Hinton.

Re: Self-supervised learning: The dark matter of intelligence

#28

Post by NotEvil »

gremlinsinc wrote: post #7

> How is it that humans can learn to drive a car in about 20 hours of practice with very little supervision, while fully autonomous driving still eludes our best AI systems trained with thousands of hours of data from human drivers? Wow, are these so-called ai scientists really that daft? Sure on paper we drive 20-40 hours "practicing" then hit the roads, but we've been back-seat driving and driving via video games o…

I think you miss the point, it's not strictly about "it take 20 hour for an child to drive a car" and more of it can do this without external help. From your example, It's about how the child knows to stop at red light when he was never told that you have to stop at red light.

Re: Self-supervised learning: The dark matter of intelligence

#29

Post by just1morething »

visarga wrote: post #23
just1morething wrote: post #19

If you cut some words out of a newspaper and a person writes words in those blanks then what has happened? If we cut out parts of a painting and a person paints in those blanks then what has happened? How is the person who wrote in the blanks or the person who painted in the blanks changed by writing or painting in the blanks? How are they changed when someone responds to what they've written or painted? How did the…

One could make a similarly long list of poorly understood concepts in psychology.

I've found that plucking words from paragraphs has, on occasion, brought them more firmly under my control-- most of them remain mysterious to me when they pop up in papers on programming or psychology.

Rather than list "ideas" or "concepts" I try to make a record of what appeared and the response I had to its appearance.

How are we to solve difficult problems if we stop mentioning them or their difficulty?

Re: Self-supervised learning: The dark matter of intelligence

#30

Post by just1morething »

fantod wrote: post #26
just1morething wrote: post #19

If you cut some words out of a newspaper and a person writes words in those blanks then what has happened? If we cut out parts of a painting and a person paints in those blanks then what has happened? How is the person who wrote in the blanks or the person who painted in the blanks changed by writing or painting in the blanks? How are they changed when someone responds to what they've written or painted? How did the…

If the person who filled in the blanks already had a good grasp of the language, they probably didn't learn much. However, fill-in-the-blank exercises can be a good language learning tool for children as well as adults learning a foreign language.

There is something special about that "lock and key" moment when you feel that satisfying "click" as your response opens the next opportunity.
Post reply on HN