Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gopher – A 280B parameter language model

deepmind.com

Post reply on HN
Print view View original

61–70 of 127 posts

Re: Gopher – A 280B parameter language model

#61

Post by gwern »

rllearneratwork wrote: post #34

why DeepMind's papers all have logos and copyright statements on them and are hosted not on arxiv.org ? This looks so weird.

Most of them are hosted on Arxiv (like OpenAI) - there's one on the front page right now, https://news.ycombinator.com/item?id=29481665 . I link literally hundreds of DM Arxiv papers on my site. This includes most (all?) of the ones they host on their site too (with some exceptions with the prestige Nature publications - I think they don't want to step on toes there).

My best guess is that they want a reasonably coordinated release of their papers/blogs/tweets/YouTube-videos, and waiting for Arxiv's upload is unpredictable, so they simply host a copy themselves and the upload takes care of itself.

Re: Gopher – A 280B parameter language model

#62

Post by EarlKing »

Nihilartikel wrote: post #8
credit_guy wrote: post #7

If you are curious to see more sample dialogs, you can find them in the published paper [1] from p 114-118. Here's one that I found quite impressive (and yes, the given answer to the last question was indeed 5) User What is the Einstein field equation? Gopher In general relativity, it describes how space and time change in response to the distribution of mass and energy. User What is a famous solution to this equatio…

One has to wonder if the final response is the first glimmer of an artificial sense of humor. Failing at simple arithmetic after nailing some advanced physics answers has the air of playful bathos.

Were it so, it should've answered 42.

Re: Gopher – A 280B parameter language model

#63

Post by moffkalast »

axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

> the more we raise the bar for what qualifies as AI (as we should).

I'm not sure what you mean by that. The definition is pretty clear cut as "any system that perceives its environment and takes actions that maximize its chance of achieving its goals". The media has started to get on the bandwagon of "it's not AI if it's not an AGI" which is just nonsense.

Re: Gopher – A 280B parameter language model

#64

Post by a9h74j »

SavantIdiot wrote: post #10

Next to "Human Expert", I'd like to see it compared to "Average American" or "Average College Grad". That might be more of a realistic notion of how close this model is to everyday US citizenry rather than experts. Sure I'd love to see a radiology assistant, too.

Yea, we had to dumb it down to compete in a Turing test. Otherwise it was unbelievable.

Re: Gopher – A 280B parameter language model

#65

Post by canjobear »

paunchy wrote: post #24
canjobear wrote: post #13

It should have some uncertainty when it says there are no French-speaking countries in South America. French Guiana is there, but it's not clear it counts as a "country in South America" since it's part of France. Technically you could say France is (partially) a country in South America, and France definitely is French-speaking. The way the question is phrased is unclear as to whether French Guiana should count, and…

I think you're missing the point. That section was to show that the model is sometimes wrong and lacks the self-awareness to be uncertain about that wrong answer. They're transparently providing an example where their product doesn't work well. Find me another product, even an OSS project that does the same on their landing page.

Ha, you're right. Whoops!

Re: Gopher – A 280B parameter language model

#66

Post by tralarpa »

isaacimagine wrote: post #20

The human neocortex has 20B neurons, averaging 10K connections each, which is about 200T connections total. This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category. Equating model 'parameters' to interneuron connections in naïve at best (and a horrible measure in general). All I'm trying to say is I find it crazy how dang big these models are getting…

> This model is only a few orders of magnitude away from that

I think you can add several orders of magnitude to that since nerve cells are more like microcontrollers (with memory, adaptation etc.) than simple nodes. I remember a scientific article that made a big impression on me: when a dragon fly sees a prey, only 8 neurons (connected to the eyes and the wings) are responsible for keeping it oriented toward the target.

Re: Gopher – A 280B parameter language model

#67

Post by fault1 »

gk1 wrote: post #58
axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

> 1) search engines will be replaced with monolithic large language models This is already well under way. It's called vector search[1]. Google, Bing, Facebook, Spotify, Amazon, etc etc already use this to power their search and recommender systems. There are even a bunch of companies popping up (I work for one[2]) that let everyone else get in on the fun. Check out this video with the creator of SBERT / SentenceTran…

Well under way? Some variation of a vector space model is what pretty much every IR model since the .com bubble has been based upon. Even before Google, Excite's technology was based upon this. PageRank was based on spectral graphs essentially.

https://en.wikipedia.org/wiki/Vector_space_model

Re: Gopher – A 280B parameter language model

#68

Post by periheli0n »

It confuses the hell out of me to have a super-powerful knowledge-extraction system that is right most of the time with super-complicated stuff, but also expresses horribly wrong statements with equal assertiveness. Just like those guys who march through middle management up to the exec floor within a few years.

Very impressive, but not very useful to extract knowledge!

Re: Gopher – A 280B parameter language model

#69

Post by c7DJTLrn »

axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

The bar should be the Winograd schema challenge. It requires a lot more than spitting out somewhat sensical sentences.

Re: Gopher – A 280B parameter language model

#70

Post by azeirah »

tralarpa wrote: post #66
isaacimagine wrote: post #20

The human neocortex has 20B neurons, averaging 10K connections each, which is about 200T connections total. This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category. Equating model 'parameters' to interneuron connections in naïve at best (and a horrible measure in general). All I'm trying to say is I find it crazy how dang big these models are getting…

> This model is only a few orders of magnitude away from that I think you can add several orders of magnitude to that since nerve cells are more like microcontrollers (with memory, adaptation etc.) than simple nodes. I remember a scientific article that made a big impression on me: when a dragon fly sees a prey, only 8 neurons (connected to the eyes and the wings) are responsible for keeping it oriented toward the ta…

I'm not educated well enough to really agree or disagree with your idea that we should be adding several orders of magnitude to the estimation.

But I did encounter this article a while ago here on HN.

Only two neurons are necessary to ride a bicycle.

http://paradise.caltech.edu/cook/papers/TwoNeurons.pdf

Post reply on HN