Earlier quoted context omitted.
What’s the current problem with control here? Outside of the loop layman here.
These transformer models are so huge, they require extremely expensive and specialist hardware beyond what enthusiasts and even many academica access to. There is no chance in the near future consumers or Edge devices will be able to run these models locally, data is going to have to be fed back into the cloud.
Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
31–40 of 40 posts
Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
#32At this point I'm comfortable in putting transformers as one of the top three developments in machine learning history. The way things are headed, they may turn out to be one of the most important "discoveries" ever made by humankind. I'm extremely optimistic about how transformers can recursively speed up progress in multiple areas of science. Transformers are reaching a point where they can demonstrate reasoning ab…
This paper shows we can combine models like lego bricks even without end-to-end training using language as intermediate representation. That means more flexibility in training the models, each on its own dataset, and more ways they can be combined in. By getting rid of fine-tuning the models may retain their robustness to distribution shifts.
Gpt-3 type models are very good at selecting for arbitrary qualities from among a list of options. Generating a list of 10 potential answers, then running prompts on the candidates to select for quality, accuracy, style, and so forth resembles the cyclic formulation of ideas in humans. The process used to generate essays and articles - draft, edit, revise, simplify, repeat until satisfied - can be implemented trivially. Those processes will transfer to larger models, and things like RETRO reduce resources by orders of magnitude.
Cognitive architecture seems to be an accurate descriptor of the use of multiple models and the logic layers for many-shot, many model development.
It may not be human level with zero-shot output, but how many humans produce human-level output in their stream-of-consciousness output? The act of consideration, recursing over an idea and refining it, is achievable with these models in a way that humans can debug and tweak cycle to cycle.
Multipass "consideration" and revision methodologies can capture almost any meta-cognitive processes used by humans, whether it's Socratic method or the AP style guide or an arbitrary jumble of rules derived from 4chan posters.
Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
#33Earlier quoted context omitted.
What’s the current problem with control here? Outside of the loop layman here.
These transformer models are so huge, they require extremely expensive and specialist hardware beyond what enthusiasts and even many academica access to. There is no chance in the near future consumers or Edge devices will be able to run these models locally, data is going to have to be fed back into the cloud.
Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
#34Earlier quoted context omitted.
It's not wrong per se, and I'm obviously in no place to police the discussion, but it's only tangentially related to the post and often clouds out what would be a more pointed deliberation over this research. Maybe I'm expecting too much of HN, but I've seen these same two top level comments under myriad ML posts. Sorry for the meta-discussion that's gotten us further away from this really remarkable paper.
I agree. It's completely speculative. There is no evidence at all that Spiking NNs really work better is any circumstances. Speaking as someone who has worked in the ML field, it feels to me like advocates for them are caught up in the biological plausibility argument. That's an interesting branch of research, but has very little to do with how AI should be implemented using transistors. In some ways the "neural netw…
Transformers have a sequence context, but it constructs its own context dependent notion of orderliness with attention.
Persistent or recurrent activation states can extend the context window past the current tokenizing limitations. Better still would be dynamic construction where new knowledge can be carefully grafted into a network without training, and updates over the recurrent states feeding back into modifying learned structures.
Spiking networks might provide a clear architecture to achieve some of those goals, but it's really just recurrence shuffled around different stages of processing.
Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
#35Earlier quoted context omitted.
"Stochastic parrots" -- have you seen, e.g., the examples in the PaLM paper of how it does on "chained inference" tasks? I don't see how you can classify that as mere parroting.
"Stochastic parrots" is a disparaging term coined by SJW propaganda. As if the brain is not stochastic, or we don't parrot from cultural sources. Language models have been accused of bias and lack of explainability, but humans are biased too and can't really explain how we take decisions. Overall this term says "limited to the intelligence of a parrot" which is false, models can solve math and coding problems, genera…
Neural networks can do math, but a lookup and memorized value model is structurally a lot different than a calculator model. The difference between them is a matter of weights for any given architecture. Tokenizing properly for math would help, but doing bit level tokenizing would be best, because that would allow multimodal domains to integrate more readily (i.e. audio/video/text models could share learned features more easily than if you are using parsed or domain specific tokens.) It's a great time to be alive.
Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
#36Earlier quoted context omitted.
"Stochastic parrots" -- have you seen, e.g., the examples in the PaLM paper of how it does on "chained inference" tasks? I don't see how you can classify that as mere parroting.
> it does on "chained inference" tasks To me, it is more proof of "stochastic parrot" behavior: model seen most of the available math information in internet, and even with significant computational power, can solve only 58% of elementary school level questions, and they were probably those with clear examples in training data, and can't generalize on those beyond.
The process kinda goes like this -
Think of ten answers to this question: blah blah blah
From these ten answers, which are the best 3?
Of the three answers, which is the best?
Revise and edit the best answer to be simpler or more understandable.
Prompt engineering is a nascent field, and we haven't seen nuanced or sophisticated use of the tool yet. Most of the metrics reported in papers are barely better than a naive Turing test. It doesn't take much introspection to know that even humans endlessly iterate and revise their output, and the best extemporaneous speech doesn't match well curated and edited material. It shouldn't surprise us that similar editing and revision processes will benefit transformer output.
Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
#37Earlier quoted context omitted.
"Stochastic parrots" -- have you seen, e.g., the examples in the PaLM paper of how it does on "chained inference" tasks? I don't see how you can classify that as mere parroting.
"Stochastic parrots" is a disparaging term coined by SJW propaganda. As if the brain is not stochastic, or we don't parrot from cultural sources. Language models have been accused of bias and lack of explainability, but humans are biased too and can't really explain how we take decisions. Overall this term says "limited to the intelligence of a parrot" which is false, models can solve math and coding problems, genera…
Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
#38Earlier quoted context omitted.
These transformer models are so huge, they require extremely expensive and specialist hardware beyond what enthusiasts and even many academica access to. There is no chance in the near future consumers or Edge devices will be able to run these models locally, data is going to have to be fed back into the cloud.
Smaller models with better performance are beginning to arrive. Things like RETRO, better training data, longer training time, and scale optimization will have these models on phones and desktops doing crazy things in the near future.
Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
#39Earlier quoted context omitted.
> We've come to the consensus that large language models are just stochastic parrots Anyone who thinks this REALLY doesn't know how language models work. A properly trained LM will only parrot something back because of lack of diversity in training data. This does happen in some cases (eg, GPL license or something) but those are pretty unique cases. People on HN seem to think this a lot, but they are just wrong.
It's especially true for ML in general on HN, but it's generally true for a lot of areas in the public - people often mistake skepticism for expertise or knowledge. I think the phenomenon is similar to the large crowd that cries "the sample is too small" any time statistics are brought up. It's the first thing anyone learns, and it's easy to do. It's really unfortunate, but that's why you see so many on HN that dismi…
This is a pretty good insight.
> that's why you see so many on HN that dismiss new technologies in ML (especially in NLP, since everyone can understand the output
I think also in NLP people see output that is the same as some training data, so think it is copying it. It takes some a little bit more thought to think "ok if I asked 100 experts to try to write how to sort an array in Python" or their code is going to be very similar. This doesn't mean it is copied.
Re: Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language
#40Earlier quoted context omitted.
I agree. It's completely speculative. There is no evidence at all that Spiking NNs really work better is any circumstances. Speaking as someone who has worked in the ML field, it feels to me like advocates for them are caught up in the biological plausibility argument. That's an interesting branch of research, but has very little to do with how AI should be implemented using transistors. In some ways the "neural netw…
Spiking comes with persistence baked in, so anything done with them has an implicit sequence and temporal context. Like LSTM, it automatically means the architecture is going to handle some problems better than a naive perceptron. Transformers have a sequence context, but it constructs its own context dependent notion of orderliness with attention. Persistent or recurrent activation states can extend the context wind…
Interesting. I hadn't really thought about this. Although I wonder if there is a more direct way of achieving this.