Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gopher – A 280B parameter language model

deepmind.com

Post reply on HN
Print view View original

111–120 of 127 posts

Re: Gopher – A 280B parameter language model

#111

Post by muzani »

gh0std3v wrote: post #97
muzani wrote: post #95

Earlier quoted context omitted.

GPT-3 is very capable of humor, even better than humans are. It's not really that it finds it funny, but more that it's mimicking a conversational pattern. Sarcasm, silliness, snarkiness, it's all there. Gopher seems to also be trained on books and the internet, so it wouldn't be so surprising.

While I want to believe this is true, I think the above response was more a lack of Gopher's ability to analyze mathematical equations properly than an attempt to display a sense of humour. Many NLP approaches work by creating word embeddings, which don't always help the model to understand "first-order logic" language mixed with spoken language. Though who knows, maybe it does have a sense of humour.

In full context, it's a conversation. "What does this mean, and what does that mean, what is also related to that?" If you ask a human impressively difficult questions then an absurdly trivial one, a human will probably respond with sarcasm. I'd expect Gopher to as well. It might get the answer right if you were asking it a series of arithmetic questions.

According to the link, Gopher is far better at math than GPT-3, and GPT-3 can solve "15 x 7", so I'd assume that Gopher would be able to as well.

Re: Gopher – A 280B parameter language model

#112

Post by vagab0nd »

freediver wrote: post #73

This model like its predecessor still lacks the ability to infer within constraints of physical world [1]. It does not have any notion of what a physical world is, and still is, at the end of the day, a statistical model. Attempting to do recreate the entirety of the physical world by describing it in text is an impressive feat, but a futile attempt. It is prone to inference errors that the model itself is not aware…

> This model like its predecessor still lacks the ability to infer within constraints of physical world [1]. It does not have any notion of what a physical world is, and still is, at the end of the day, a statistical model.

How did you reach this conclusion? Is there evidence or was it inferred from the model structure?

Re: Gopher – A 280B parameter language model

#113

Post by leobg »

freediver wrote: post #73

This model like its predecessor still lacks the ability to infer within constraints of physical world [1]. It does not have any notion of what a physical world is, and still is, at the end of the day, a statistical model. Attempting to do recreate the entirety of the physical world by describing it in text is an impressive feat, but a futile attempt. It is prone to inference errors that the model itself is not aware…

Exactly! Same problem with GPT-3. That’s why most of the commercial applications people have found for it so far are in marketing. Even if you tell it explicitly not to make up stuff and only chose from applicable answers, it will confabule. And you never know when it does.

Re: Gopher – A 280B parameter language model

#114

Post by lostmsu »

tralarpa wrote: post #66
isaacimagine wrote: post #20

The human neocortex has 20B neurons, averaging 10K connections each, which is about 200T connections total. This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category. Equating model 'parameters' to interneuron connections in naïve at best (and a horrible measure in general). All I'm trying to say is I find it crazy how dang big these models are getting…

> This model is only a few orders of magnitude away from that I think you can add several orders of magnitude to that since nerve cells are more like microcontrollers (with memory, adaptation etc.) than simple nodes. I remember a scientific article that made a big impression on me: when a dragon fly sees a prey, only 8 neurons (connected to the eyes and the wings) are responsible for keeping it oriented toward the ta…

For all we know these microcontrollers may be that complicated only because they try to emulate discrete logic, and you actually need hundreds of them to make a single unit reliable enough for that purpose.

Re: Gopher – A 280B parameter language model

#115

Post by ramraj07 »

xmaayy wrote: post #19
Nihilartikel wrote: post #8

Earlier quoted context omitted.

One has to wonder if the final response is the first glimmer of an artificial sense of humor. Failing at simple arithmetic after nailing some advanced physics answers has the air of playful bathos.

I think it's more likely that 5 came out because if it ever saw the answer, 105, before, it was split into the tokens [10][5] of which it only 'remembered' one. Or the numbers were masked when training (something that was done with BERT-like models) so it just knew enough to put a random one in

Some of the funniest jokes I tell are retellings of jokes from obscure comedians of the past. How is this AI any different from me?

Re: Gopher – A 280B parameter language model

#116

Post by cscurmudgeon »

axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

> Schwarzschild metric

No, memorizing trivia is not it.

Can GPT-3 prove theorems about Schwarzschild metric?

Re: Gopher – A 280B parameter language model

#117

Post by mellosouls »

ramraj07 wrote: post #115
xmaayy wrote: post #19

Earlier quoted context omitted.

I think it's more likely that 5 came out because if it ever saw the answer, 105, before, it was split into the tokens [10][5] of which it only 'remembered' one. Or the numbers were masked when training (something that was done with BERT-like models) so it just knew enough to put a random one in

Some of the funniest jokes I tell are retellings of jokes from obscure comedians of the past. How is this AI any different from me?

You enjoy the joke.

Re: Gopher – A 280B parameter language model

#118

Post by sdenton4 »

cscurmudgeon wrote: post #116
axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

> Schwarzschild metric No, memorizing trivia is not it. Can GPT-3 prove theorems about Schwarzschild metric?

Can an average human prove theorems?

Re: Gopher – A 280B parameter language model

#119

Post by visarga »

mellosouls wrote: post #117
ramraj07 wrote: post #115

Earlier quoted context omitted.

Some of the funniest jokes I tell are retellings of jokes from obscure comedians of the past. How is this AI any different from me?

You enjoy the joke.

AI enjoys "low error rate". They have to, otherwise they would not exist.

AIs that don't lower the error rate are abandoned, AIs that score well are replicated and improved. It's evolution at work, but they have to enjoy (optimise for) lower error rates in order to even exist.

Re: Gopher – A 280B parameter language model

#120

Post by [deleted] »

the8472 wrote: post #79
bee_rider wrote: post #60

Earlier quoted context omitted.

I can definitely see your argument, but there are some downsides to deploying ASAP: * Long term, the reputational damage to self driving cars might be significant if one is released that is only slightly better than a drunk driver. If this hinders uptake later, when self-driving cars are completely superior to normal humans, we've possibly produced a net negative. * While a self driving car should be safer than a dru…

This is mostly worrying about hard to predict second-order effects. Exactly the same problem that caused various health organizations (including WHO and FDA) send misleading messages around corona and vaccinations. > Issues around who is responsible when a self driving car which is known to be worse than a typical human is deployed, fails, and someone is hurt. Even if insurance rates would be initially(!) higher than…

[deleted]
Post reply on HN