Um, e^{ipi} + 1 is zero, not 2.
Don't read only the examples, but also the text between them ;)
Gopher – A 280B parameter language model
41–50 of 127 posts
Re: Gopher – A 280B parameter language model
#42The number of parameters could be a vanity metric--like saying my CPU is 1000W (is that fast or inefficient?). From the first (of three) linked papers in the article. > Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language mo…
I remember some talk in which the researcher quantitatively said that parameter count is the only important metric for transformers it seems. It doesn't matter if the model is taller or thicker or have more number of attention heads.
Re: Gopher – A 280B parameter language model
#43Earlier quoted context omitted.
Nothing like a little anthropomorphism to completely distort otherwise good faith interpretations of bot behavior.
How is the impression of playfulness not a good faith interpretation? You of course know that the model is not capable of thought or reasoning - only the appearance of them as needed to match its training corpus. A training corpus of completely human generated data. As such, how could anything it does, be anything but anthropomorphic? Now, if this model were trained exclusively on a corpus of mathematical proofs stri…
> You of course know that the model is not capable of thought or reasoning
Do we know? It's the reverse Chinese room problem. :p
Re: Gopher – A 280B parameter language model
#44The number of parameters could be a vanity metric--like saying my CPU is 1000W (is that fast or inefficient?). From the first (of three) linked papers in the article. > Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language mo…
The number of parameters is absolutely not a vanity metric, but it is inefficient, and efficiency is a related-but-different area of research (e.g. distillation) that's more of a business problem than a research problem.
Re: Gopher – A 280B parameter language model
#45Earlier quoted context omitted.
The problem seems to be that these models provide fairly accurate information at many occasions and occasionally complete blunders. Humans provide less accurate information most of the time but with a certain amount of self-reflection/meta-cognition, and they will usually recognize total blunders or display reasonable uncertainty about them. There are only very few applications where it would make sense to take the r…
Accuracy is improving rapidly though. I agree that the current accuracy levels are not high enough to be relied upon. > Humans ... they will usually recognize total blunder I question this assumption. I don't believe this is true, even for subject matter experts. I've worked with radiology data where experts with 10+ years of experience make blunders that disagree with a consensus panel of radiologists.
Radiology is one of those fields were a lot of it comes from experience and intuition mostly because of how complex the human body is. Compare this to a physicist; pretty sure you wont get as much disagreements there.
Re: Gopher – A 280B parameter language model
#46Is this model as closed as GPT-3, or has it been open-sourced?
Re: Gopher – A 280B parameter language model
#47why DeepMind's papers all have logos and copyright statements on them and are hosted not on arxiv.org ? This looks so weird.
Do they publish their latex source anywhere?
Re: Gopher – A 280B parameter language model
#48[deleted]
Re: Gopher – A 280B parameter language model
#49Earlier quoted context omitted.
The problem seems to be that these models provide fairly accurate information at many occasions and occasionally complete blunders. Humans provide less accurate information most of the time but with a certain amount of self-reflection/meta-cognition, and they will usually recognize total blunders or display reasonable uncertainty about them. There are only very few applications where it would make sense to take the r…
Accuracy is improving rapidly though. I agree that the current accuracy levels are not high enough to be relied upon. > Humans ... they will usually recognize total blunder I question this assumption. I don't believe this is true, even for subject matter experts. I've worked with radiology data where experts with 10+ years of experience make blunders that disagree with a consensus panel of radiologists.
I realize I should have been more precise. I agree that there are many areas in which AI can and already has excelled humans and less often makes grave mistakes than humans. I specifically had natural language processing with a focus on "intelligent" conversation in mind. The issues in that area might have less to do with the pattern recognition ability and more with the lack of appropriate meta-cognition, introspection, and self-doubt. Maybe having several AIs internally berate which answer is best before uttering it would already do the trick, though.
Re: Gopher – A 280B parameter language model
#50Earlier quoted context omitted.
How is the impression of playfulness not a good faith interpretation? You of course know that the model is not capable of thought or reasoning - only the appearance of them as needed to match its training corpus. A training corpus of completely human generated data. As such, how could anything it does, be anything but anthropomorphic? Now, if this model were trained exclusively on a corpus of mathematical proofs stri…
> You of course know that the model is not capable of thought or reasoning Do we know? It's the reverse Chinese room problem. :p
A good point - I'm taking it as given that reasoning of any depth is more of an iterative process, with one thought advancing as a meta-cognitively guided feedback to the next until a conclusion is reached. One prompt->completion cycle from a language model wouldn't necessarily meet that definition, but I bet it could be a component in a system that tries to do so.
I aspire one day to find the free weekends and adequate hubris to build a benchtop implementation of Julian Jayne's Bicameral Mind with 1+N GPT-3 or GPT-neo instances prompting each other iteratively to see where the train of semantics wanders. (as I'm sure others have already)