Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-supervised learning: The dark matter of intelligence

ai.facebook.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Self-supervised learning: The dark matter of intelligence

#31

Post by asamiam »

Reebz wrote: post #13

State of the art performance is being broken in multiple fields rapidly these days. However, AI explainability has a long way to go. Scaling opaqueness makes this problem worse. Fine tuning labels black-box style is a terrifying concept to most who are working in fields where great risk must be managed to avoid unintended and disparate impact. Facebook making an oopsies suggesting my friends face in a photo instead o…

We want our models to be opaque in the same way we like to ask people why they made a certain decision or act in the way they do. What's interesting to me is if you want to know about a person you'll get better information by asking their closest friends than themselves.

Perhaps there's something in the black box nature of our own self understanding similar to these hyper complex function approximators.

Judges make harsher rulings when they're hungry. A regulator might decline a loan and justify it b/c they had a bad experience with someone that reminds them of the person they're dealing with. AI makes bad decisions in boundary conditions or when they're trained on biased data. I think it's good to strive for opacity in every case but in complex decision spaces I'm not sure if we'll ever get fully satisfactory explanations.

Which is a meandering way of making the non-point that models making judgments have made me re-evaluate what I trust and why, and I think in many cases I'd rather trust a black box if I know what's gone into it and what's come out.

Re: Self-supervised learning: The dark matter of intelligence

#32

Post by d_burfoot »

I wrote a book about this 10 years ago:

https://arxiv.org/abs/1104.5466

Annotation doesn't scale. You are not going to get to real intelligence by labelling billions of data points.

I expect that computer vision for autonomous vehicles will become good enough for practical use quite soon after the AV companies start doing large scale self-supervised learning (I don't like that term). Maybe Tesla and Waymo and others are doing that already.

Another great application area is medical imaging. Progress in this area has been seriously hampered by the limitations of the annotation paradigm. Once people start to do SSL, there should be rapid progress.

Re: Self-supervised learning: The dark matter of intelligence

#33

Post by vincnetas »

visarga wrote: post #20
montenegrohugo wrote: post #2

One of the problems we have to be conscious about with self supervised learning is that the already existing problem of transparency and understandability gets worsened by orders of magnitude. If current billion parameter models are opaque, the result of SSL is, or will be, the equivalent of a black hole. How do you understand what the model has learned? Why has it learned that way? What things could have possible go…

> How do you understand what the model has learned? How do you do that with humans? You watch their behavior, language and you test them.

Yes. Humans make mistakes, humans with computers make mistakes faster. Computers without humans make mistakes even faster.

Re: Self-supervised learning: The dark matter of intelligence

#34

Post by uoaei »

visarga wrote: post #20
montenegrohugo wrote: post #2

One of the problems we have to be conscious about with self supervised learning is that the already existing problem of transparency and understandability gets worsened by orders of magnitude. If current billion parameter models are opaque, the result of SSL is, or will be, the equivalent of a black hole. How do you understand what the model has learned? Why has it learned that way? What things could have possible go…

> How do you understand what the model has learned? How do you do that with humans? You watch their behavior, language and you test them.

Humans can explain their reasoning when you ask them. The best we have with NNs is a fancy version of first-order sensitivity analysis.

Re: Self-supervised learning: The dark matter of intelligence

#35

Post by knuthsat »

montenegrohugo wrote: post #2

One of the problems we have to be conscious about with self supervised learning is that the already existing problem of transparency and understandability gets worsened by orders of magnitude. If current billion parameter models are opaque, the result of SSL is, or will be, the equivalent of a black hole. How do you understand what the model has learned? Why has it learned that way? What things could have possible go…

Tools for exploring the models are already being developed.

Language models that are not given concepts of part-of-speech, dependency trees, grammars, develop representations of their own and people have found ways to inspect them.

Re: Self-supervised learning: The dark matter of intelligence

#36

Post by Schiphol »

knuthsat wrote: post #35
montenegrohugo wrote: post #2

One of the problems we have to be conscious about with self supervised learning is that the already existing problem of transparency and understandability gets worsened by orders of magnitude. If current billion parameter models are opaque, the result of SSL is, or will be, the equivalent of a black hole. How do you understand what the model has learned? Why has it learned that way? What things could have possible go…

Tools for exploring the models are already being developed. Language models that are not given concepts of part-of-speech, dependency trees, grammars, develop representations of their own and people have found ways to inspect them.

Could you please provide pointers to relevant papers? I'm very interested.

Re: Self-supervised learning: The dark matter of intelligence

#37

Post by ankfo04300 »

vladmk wrote: post #17

Does not anyone worry about this whole human level AI Pandora’s box trap? Why not work on nukes themselves, sounds safer to me

Don't worry. A sufficiently powerful AGI won't destroy the universe to satisfy its own goals; it'll just build a holodeck (by tampering with its sensors) so that it can gain +infinity pleasure right away.

Unlike software, which can be mathematically airtight, there's no reason to assume that physical devices built by humans are unexploitable in the face of the massive intelligence of the AGI. So the AGI will just ask itself which is easier: delicately performing surgery on its own sensors, or building tons of nukes.

Honestly, that people are even worried about AGI doomsday shows to me the power of narrative. Everybody has heard of SkyNet and the sorcerer's apprentice, nobody has heard of the AGI who outsmarted itself by creating digital porn for its objective function. Therefore, through the miracle of the availability heuristic, AGI doomsday it is.

Re: Self-supervised learning: The dark matter of intelligence

#38

Post by social_quotient »

just1morething wrote: post #30
fantod wrote: post #26

Earlier quoted context omitted.

If the person who filled in the blanks already had a good grasp of the language, they probably didn't learn much. However, fill-in-the-blank exercises can be a good language learning tool for children as well as adults learning a foreign language.

There is something special about that "lock and key" moment when you feel that satisfying "click" as your response opens the next opportunity.

Escape rooms for language models :)

(Context https://en.m.wikipedia.org/wiki/Escape_room )

Re: Self-supervised learning: The dark matter of intelligence

#39

Post by neoncontrails »

These are interesting epistemological questions. Is there a compelling reason to label a certain neural architecture as a "siamese network"? I'm guessing that this is a reference to conjoined twins, a biological phenomenon with no connection to the country previously known as Siam (now Thailand).

Re: Self-supervised learning: The dark matter of intelligence

#40

Post by scribu »

asamiam wrote: post #31
Reebz wrote: post #13

State of the art performance is being broken in multiple fields rapidly these days. However, AI explainability has a long way to go. Scaling opaqueness makes this problem worse. Fine tuning labels black-box style is a terrifying concept to most who are working in fields where great risk must be managed to avoid unintended and disparate impact. Facebook making an oopsies suggesting my friends face in a photo instead o…

We want our models to be opaque in the same way we like to ask people why they made a certain decision or act in the way they do. What's interesting to me is if you want to know about a person you'll get better information by asking their closest friends than themselves. Perhaps there's something in the black box nature of our own self understanding similar to these hyper complex function approximators. Judges make h…

> Judges make harsher rulings when they're hungry.

Actually, that finding is highly improbable: https://nautil.us/blog/impossibly-hungry-judges

Post reply on HN