Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gopher – A 280B parameter language model

deepmind.com

Post reply on HN
Print view View original

81–90 of 127 posts

Re: Gopher – A 280B parameter language model

#81

Post by YeGoblynQueenne »

karmakaze wrote: post #4

The number of parameters could be a vanity metric--like saying my CPU is 1000W (is that fast or inefficient?). From the first (of three) linked papers in the article. > Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language mo…

>> Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit

Left unsaid: it's much harder to learn shortcuts to fake performance in logic and reasoning and common arithmetic, than it is to do the same in "reading comprehension" and similar tasks for which there are no good benchmark datasets (most are basically multipl-choice tests that can be solved by a strong classifier without any need of anything like "comprehension") and for which there are no good metrics either (see BLEU score and friends).

Re: Gopher – A 280B parameter language model

#82

Post by amensch »

gwern wrote: post #61
rllearneratwork wrote: post #34

why DeepMind's papers all have logos and copyright statements on them and are hosted not on arxiv.org ? This looks so weird.

Most of them are hosted on Arxiv (like OpenAI) - there's one on the front page right now, https://news.ycombinator.com/item?id=29481665 . I link literally hundreds of DM Arxiv papers on my site. This includes most (all?) of the ones they host on their site too (with some exceptions with the prestige Nature publications - I think they don't want to step on toes there). My best guess is that they want a reasonably coor…

Pretty accurate guess ;)

Re: Gopher – A 280B parameter language model

#83

Post by dietr1ch »

credit_guy wrote: post #7

If you are curious to see more sample dialogs, you can find them in the published paper [1] from p 114-118. Here's one that I found quite impressive (and yes, the given answer to the last question was indeed 5) User What is the Einstein field equation? Gopher In general relativity, it describes how space and time change in response to the distribution of mass and energy. User What is a famous solution to this equatio…

Well, math is hard

Re: Gopher – A 280B parameter language model

#84

Post by YeGoblynQueenne »

isaacimagine wrote: post #20

The human neocortex has 20B neurons, averaging 10K connections each, which is about 200T connections total. This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category. Equating model 'parameters' to interneuron connections in naïve at best (and a horrible measure in general). All I'm trying to say is I find it crazy how dang big these models are getting…

>> This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category.

A few orders of magnitude and an entire category away. Artificial "neurons" only have the name "neuron" in common with biological neurons. Consequently you can stack as many layers of artificial neurons on top of each other as you may want and you won't get anywhere near the abilities of the simplest systems of biological neurons.

For example, spiders have ~100 thousand neurons and there's no artificial neural networks that could shake a stick at a spider's cognitive abilities. Which are downright scary, btw.

Estimated number of neurons of spiders from wikipedia:

https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...

Re: Gopher – A 280B parameter language model

#85

Post by advisedwang »

johncena33 wrote: post #80
advisedwang wrote: post #78

The second paper cites the "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" paper that got Timnit Gebru and Meg Mitchell fired from Google. It must be pretty galling to get fired for a paper and then have the company that fired you cite it a year later.

Please stop spreading disinformation. Mitchell got fired for leaking company documents to outsiders. When people spread obvious lies, that makes me thing they don't have anything substantial to begin with. that's why they need to resort to lying.

I haven't dug up the details, so perhaps my recollection is wrong, but I thought she got fired for co-authoring the paper under "Shmargaret Shmitchell" name when Google asked them to withdraw?

I'm sure Google made lots of accusations if asked for comment, that seems SOP, but what actually triggered it? What documents did she leak?

Re: Gopher – A 280B parameter language model

#86

Post by sailingparrot »

karmakaze wrote: post #4

The number of parameters could be a vanity metric--like saying my CPU is 1000W (is that fast or inefficient?). From the first (of three) linked papers in the article. > Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language mo…

All large language models (GPT-2/3, GPT-Neo, Turing, Gopher), all use essentially the same architecture with some light variations, and the same datasets, again with some light variations on how filtering is done etc.

As such there is no reason to expect them to be very different in term of efficiency, and it has been shown and well researched, that scaling the numbers of parameters directly correlates with improved model quality.

So as long as you are comparing GPT style models to other GPT style models then parameter count is definitly not a vanity metric.

This doesn't hold once you start comparing to e.g. mixture of experts model which were making the headline recently with trillion of parameters claims. In MoE models, parameter counts is pretty much a useless metric.

Re: Gopher – A 280B parameter language model

#87

Post by doctor_eval »

Nihilartikel wrote: post #28
xmaayy wrote: post #19

Earlier quoted context omitted.

I think it's more likely that 5 came out because if it ever saw the answer, 105, before, it was split into the tokens [10][5] of which it only 'remembered' one. Or the numbers were masked when training (something that was done with BERT-like models) so it just knew enough to put a random one in

That seems likely and fair. What moved me to post is that that kind of silly answer is the exact sort of shenanigans that I would pull if I were cast as the control group in a Turing test. I already do such things winkingly when talking with my preschooler to send him epistemic tracer rounds and see if he's listening critically

> epistemic tracer rounds

that's the best phrase I've heard all year.

I do this all the time with my kids too, but I think of it more as fault injection.

Re: Gopher – A 280B parameter language model

#88

Post by melony »

YeGoblynQueenne wrote: post #84
isaacimagine wrote: post #20

The human neocortex has 20B neurons, averaging 10K connections each, which is about 200T connections total. This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category. Equating model 'parameters' to interneuron connections in naïve at best (and a horrible measure in general). All I'm trying to say is I find it crazy how dang big these models are getting…

>> This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category. A few orders of magnitude and an entire category away. Artificial "neurons" only have the name "neuron" in common with biological neurons. Consequently you can stack as many layers of artificial neurons on top of each other as you may want and you won't get anywhere near the abilities of the…

Spiking ones need to solve differential equations and our current hardware designs are too discrete for it to be an efficient strategy (barring an algorithmic breakthrough).

Re: Gopher – A 280B parameter language model

#89

Post by [deleted] »

johncena33 wrote: post #80
advisedwang wrote: post #78

The second paper cites the "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" paper that got Timnit Gebru and Meg Mitchell fired from Google. It must be pretty galling to get fired for a paper and then have the company that fired you cite it a year later.

Please stop spreading disinformation. Mitchell got fired for leaking company documents to outsiders. When people spread obvious lies, that makes me thing they don't have anything substantial to begin with. that's why they need to resort to lying.

[deleted]

Re: Gopher – A 280B parameter language model

#90

Post by earthboundkid »

axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

People assert that the bar has been raised in forums like this, but it’s hogwash. The bar in fiction is clear: Hal, Data from Star Trek, and Rosie from the Jetsons are AI. The computer from Star Trek (listens to voice commands and gives answers, ala Siri) is not. The bar is where it always was. You’re trying to drag it down. No one has pushed it up.
Post reply on HN