Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gopher – A 280B parameter language model

deepmind.com

Post reply on HN
Print view View original

91–100 of 127 posts

Re: Gopher – A 280B parameter language model

#91

Post by GrantZvolsky »

SavantIdiot wrote: post #10

Next to "Human Expert", I'd like to see it compared to "Average American" or "Average College Grad". That might be more of a realistic notion of how close this model is to everyday US citizenry rather than experts. Sure I'd love to see a radiology assistant, too.

The paper does present an "Average American" comparison when it refers to Amazon Turk. The Turk workers performed significantly better on middle-school reading comprehension (RACE-m) while Gopher performed slightly better on high-school reading comprehension tasks (RACE-h).

Re: Gopher – A 280B parameter language model

#92

Post by ShamelessC »

YetAnotherNick wrote: post #42
karmakaze wrote: post #4

The number of parameters could be a vanity metric--like saying my CPU is 1000W (is that fast or inefficient?). From the first (of three) linked papers in the article. > Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language mo…

I remember some talk in which the researcher quantitatively said that parameter count is the only important metric for transformers it seems. It doesn't matter if the model is taller or thicker or have more number of attention heads.

That's correct. The original paper is "Scaling Laws for Transformers".

Re: Gopher – A 280B parameter language model

#94

Post by lapinot »

coolspot wrote: post #93
Siira wrote: post #46

Is this model as closed as GPT-3, or has it been open-sourced?

Language models are too dangerous to be accesible by mere mortals. Among OpenAI (and MS), Google and Nvidia, none of them have released their pretrained large language models.

> Language models are too dangerous to be accesible by mere mortals.

Just like strong cryptography right? This is bullshit. Either it's plain dangerous and you shouldn't be doing it or it's morally fine and it should be open. Their final usecase is pushing ads or extorting data from unsuspecting people's interactions anyway so who are they kidding with their ethics..

I mean they'd be foolish giving it away: it's a big investment and a rare good, highly strategic. Let's not wrap that kind of action in some savior stance. They are just making money and it's ugly.

Re: Gopher – A 280B parameter language model

#95

Post by muzani »

Nihilartikel wrote: post #8
credit_guy wrote: post #7

If you are curious to see more sample dialogs, you can find them in the published paper [1] from p 114-118. Here's one that I found quite impressive (and yes, the given answer to the last question was indeed 5) User What is the Einstein field equation? Gopher In general relativity, it describes how space and time change in response to the distribution of mass and energy. User What is a famous solution to this equatio…

One has to wonder if the final response is the first glimmer of an artificial sense of humor. Failing at simple arithmetic after nailing some advanced physics answers has the air of playful bathos.

GPT-3 is very capable of humor, even better than humans are. It's not really that it finds it funny, but more that it's mimicking a conversational pattern. Sarcasm, silliness, snarkiness, it's all there. Gopher seems to also be trained on books and the internet, so it wouldn't be so surprising.

Re: Gopher – A 280B parameter language model

#96

Post by random3 »

axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

> how many Americans know what a Schwarzschild metric is?

Reminds me of Feynman's anecdote about the bird and knowing the name of something https://www.youtube.com/watch?v=px_4TxC2mXU

Re: Gopher – A 280B parameter language model

#97

Post by gh0std3v »

muzani wrote: post #95
Nihilartikel wrote: post #8

Earlier quoted context omitted.

One has to wonder if the final response is the first glimmer of an artificial sense of humor. Failing at simple arithmetic after nailing some advanced physics answers has the air of playful bathos.

GPT-3 is very capable of humor, even better than humans are. It's not really that it finds it funny, but more that it's mimicking a conversational pattern. Sarcasm, silliness, snarkiness, it's all there. Gopher seems to also be trained on books and the internet, so it wouldn't be so surprising.

While I want to believe this is true, I think the above response was more a lack of Gopher's ability to analyze mathematical equations properly than an attempt to display a sense of humour. Many NLP approaches work by creating word embeddings, which don't always help the model to understand "first-order logic" language mixed with spoken language.

Though who knows, maybe it does have a sense of humour.

Re: Gopher – A 280B parameter language model

#98

Post by nimih »

marginalia_nu wrote: post #3
quaintdev wrote: post #2

Gohper and language in same sentence I thought this was related to GoLang. They should have chosen something else to name that language model.

To be fair, the name Gopher in tech predates GoLang by quite a few years.

To be fair to the GP poster, they may have been making a joke[1].

[1]https://github.com/golang/go/issues/9#issue-51275878

Re: Gopher – A 280B parameter language model

#99

Post by runnerup »

lapinot wrote: post #94
coolspot wrote: post #93

Earlier quoted context omitted.

Language models are too dangerous to be accesible by mere mortals. Among OpenAI (and MS), Google and Nvidia, none of them have released their pretrained large language models.

> Language models are too dangerous to be accesible by mere mortals. Just like strong cryptography right? This is bullshit. Either it's plain dangerous and you shouldn't be doing it or it's morally fine and it should be open. Their final usecase is pushing ads or extorting data from unsuspecting people's interactions anyway so who are they kidding with their ethics.. I mean they'd be foolish giving it away: it's a bi…

> Their final usecase is pushing ads or extorting data from unsuspecting people's interactions anyway so who are they kidding with their ethics..

The final usecase is actually brainwashing. There could come a future where every single comment you read online is not a human but rather a multi-faceted AI tailored specifically for you. Your current worldview could be measured and the AI could steer you closer towards a "target" worldview.

The comments presented to you could also be moderately coordinated with comments presented to other humans in your real-life circles to establish "serendipity" where you and your friends/family/coworkers evolve similar ideas/worldviews together to reinforce your new beliefs. And even catalyze coordinated human efforts towards common goals.

Doing this could lead most people to believe almost anything. Theoretically this could also be coupled with VR chatrooms to enhance salience.

Re: Gopher – A 280B parameter language model

#100

Post by gattilorenz »

imachine1980_ wrote: post #55
axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

>Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval I agree in the same way than 70 % of people have less 100 iq, we depend in specialist when I need to know if I have epilepsi I need a person/thing who work is be up to date, and have the less bias as possible and agregator models are quite usually miss in understand what is crital info, try to program only whit gith…

> 70 % of people have less 100 iq

Being pedantic here, but isn't that more like ~50%, by definition? Or did I misunderstand how IQ works?

Post reply on HN