Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

socraticmodels.github.io

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#21

Post by nl »

vagabund wrote: post #10
nynx wrote: post #9

Earlier quoted context omitted.

I’m certainly not discounting the results and I don’t see anything wrong with suggesting what I think would generally be a good path to look at in the future.

It's not wrong per se, and I'm obviously in no place to police the discussion, but it's only tangentially related to the post and often clouds out what would be a more pointed deliberation over this research. Maybe I'm expecting too much of HN, but I've seen these same two top level comments under myriad ML posts. Sorry for the meta-discussion that's gotten us further away from this really remarkable paper.

I agree.

It's completely speculative. There is no evidence at all that Spiking NNs really work better is any circumstances.

Speaking as someone who has worked in the ML field, it feels to me like advocates for them are caught up in the biological plausibility argument. That's an interesting branch of research, but has very little to do with how AI should be implemented using transistors. In some ways the "neural networks" name has done a great disservice because people keep getting caught in the trap of comparing them to how the human brain works.

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#22

Post by nl »

arjvik wrote: post #2

We've come to the consensus that large language models are just stochastic parrots... What makes us think that we can achieve a higher level of intelligence by putting them in conversation? I think the next step in NLP will be a drastic innovation on today's learning model.

> We've come to the consensus that large language models are just stochastic parrots

Anyone who thinks this REALLY doesn't know how language models work. A properly trained LM will only parrot something back because of lack of diversity in training data. This does happen in some cases (eg, GPL license or something) but those are pretty unique cases.

People on HN seem to think this a lot, but they are just wrong.

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#23

Post by axg11 »

At this point I'm comfortable in putting transformers as one of the top three developments in machine learning history. The way things are headed, they may turn out to be one of the most important "discoveries" ever made by humankind.

I'm extremely optimistic about how transformers can recursively speed up progress in multiple areas of science. Transformers are reaching a point where they can demonstrate reasoning abilities within the ballpark of what you might expect from a human. For certain qualities, they far exceed what any human is capable of. One of those areas being depth of knowledge. Transformers (e.g. RETRO) can incorporate a library of knowledge far larger than any human can. Soon we will improve and harness this ability to the point where it may be pointless to create a scientific hypothesis without first "consulting" a large language model that is able to process the entire library of scientific publications.

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#24

Post by visarga »

gjm11 wrote: post #17
arjvik wrote: post #2

We've come to the consensus that large language models are just stochastic parrots... What makes us think that we can achieve a higher level of intelligence by putting them in conversation? I think the next step in NLP will be a drastic innovation on today's learning model.

"Stochastic parrots" -- have you seen, e.g., the examples in the PaLM paper of how it does on "chained inference" tasks? I don't see how you can classify that as mere parroting.

"Stochastic parrots" is a disparaging term coined by SJW propaganda. As if the brain is not stochastic, or we don't parrot from cultural sources. Language models have been accused of bias and lack of explainability, but humans are biased too and can't really explain how we take decisions.

Overall this term says "limited to the intelligence of a parrot" which is false, models can solve math and coding problems, generate passable art, translate and speak in hundreds of languages and beat us at all board and card games. When was a parrot able to do that?

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#25

Post by riku_iki »

gjm11 wrote: post #17
arjvik wrote: post #2

We've come to the consensus that large language models are just stochastic parrots... What makes us think that we can achieve a higher level of intelligence by putting them in conversation? I think the next step in NLP will be a drastic innovation on today's learning model.

"Stochastic parrots" -- have you seen, e.g., the examples in the PaLM paper of how it does on "chained inference" tasks? I don't see how you can classify that as mere parroting.

> it does on "chained inference" tasks

To me, it is more proof of "stochastic parrot" behavior: model seen most of the available math information in internet, and even with significant computational power, can solve only 58% of elementary school level questions, and they were probably those with clear examples in training data, and can't generalize on those beyond.

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#26

Post by visarga »

axg11 wrote: post #23

At this point I'm comfortable in putting transformers as one of the top three developments in machine learning history. The way things are headed, they may turn out to be one of the most important "discoveries" ever made by humankind. I'm extremely optimistic about how transformers can recursively speed up progress in multiple areas of science. Transformers are reaching a point where they can demonstrate reasoning ab…

This paper shows we can combine models like lego bricks even without end-to-end training using language as intermediate representation. That means more flexibility in training the models, each on its own dataset, and more ways they can be combined in. By getting rid of fine-tuning the models may retain their robustness to distribution shifts.

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#27

Post by visarga »

nynx wrote: post #4

This is super impressive. Transformers have consistently done better than almost anyone thought. I still hold the opinion that we’re going to need to move to spiking neuron (SNN) models in the future to keep growing the networks. Spiking networks require lots of storage, but a lot, lot less compute. They also propagate additional information in the _timing_ of the spikes, not just the values. There are a lot of low-h…

> Spiking networks require lots of storage, but a lot, lot less compute.

One way or another we need a 1000x increase in efficiency to be able to run these models on edge hardware with full privacy and outside the control of the big corporations.

Funny that Gary Marcus is pleading on Twitter to get Dall-E 2 access in order to formulate his response. He isn't getting access yet. https://twitter.com/GaryMarcus/status/1513215530366234625

That kind of gate-keeping is possible because the costs of training and inferencing these models is too high today.

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#28

Post by chaxor »

nl wrote: post #22
arjvik wrote: post #2

We've come to the consensus that large language models are just stochastic parrots... What makes us think that we can achieve a higher level of intelligence by putting them in conversation? I think the next step in NLP will be a drastic innovation on today's learning model.

> We've come to the consensus that large language models are just stochastic parrots Anyone who thinks this REALLY doesn't know how language models work. A properly trained LM will only parrot something back because of lack of diversity in training data. This does happen in some cases (eg, GPL license or something) but those are pretty unique cases. People on HN seem to think this a lot, but they are just wrong.

It'sespecially true for ML in general on HN, but it's generally true for a lot of areas in the public - people often mistake skepticism for expertise or knowledge. I think the phenomenon is similar to the large crowd that cries "the sample is too small" any time statistics are brought up.

It's the first thing anyone learns, and it's easy to do.

It's really unfortunate, but that's why you see so many on HN that dismiss new technologies in ML (especially in NLP, since everyone can understand the output - that's less true in e.g. protein folding)

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#29

Post by prox »

visarga wrote: post #27
nynx wrote: post #4

This is super impressive. Transformers have consistently done better than almost anyone thought. I still hold the opinion that we’re going to need to move to spiking neuron (SNN) models in the future to keep growing the networks. Spiking networks require lots of storage, but a lot, lot less compute. They also propagate additional information in the _timing_ of the spikes, not just the values. There are a lot of low-h…

> Spiking networks require lots of storage, but a lot, lot less compute. One way or another we need a 1000x increase in efficiency to be able to run these models on edge hardware with full privacy and outside the control of the big corporations. Funny that Gary Marcus is pleading on Twitter to get Dall-E 2 access in order to formulate his response. He isn't getting access yet. https://twitter.com/GaryMarcus/status/15…

What’s the current problem with control here? Outside of the loop layman here.

Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language

#30

Post by teruakohatu »

prox wrote: post #29
visarga wrote: post #27

Earlier quoted context omitted.

> Spiking networks require lots of storage, but a lot, lot less compute. One way or another we need a 1000x increase in efficiency to be able to run these models on edge hardware with full privacy and outside the control of the big corporations. Funny that Gary Marcus is pleading on Twitter to get Dall-E 2 access in order to formulate his response. He isn't getting access yet. https://twitter.com/GaryMarcus/status/15…

What’s the current problem with control here? Outside of the loop layman here.

These transformer models are so huge, they require extremely expensive and specialist hardware beyond what enthusiasts and even many academica access to.

There is no chance in the near future consumers or Edge devices will be able to run these models locally, data is going to have to be fed back into the cloud.

Post reply on HN