[deleted]
Gopher – A 280B parameter language model
11–20 of 127 posts
Re: Gopher – A 280B parameter language model
#12If you are curious to see more sample dialogs, you can find them in the published paper [1] from p 114-118. Here's one that I found quite impressive (and yes, the given answer to the last question was indeed 5) User What is the Einstein field equation? Gopher In general relativity, it describes how space and time change in response to the distribution of mass and energy. User What is a famous solution to this equatio…
I'm not familiar with the Bardeen metric, but spinning and charged is usually described by the Kerr-Newman metric.
Re: Gopher – A 280B parameter language model
#13It should have some uncertainty when it says there are no French-speaking countries in South America. French Guiana is there, but it's not clear it counts as a "country in South America" since it's part of France. Technically you could say France is (partially) a country in South America, and France definitely is French-speaking. The way the question is phrased is unclear as to whether French Guiana should count, and yet Gopher says that it is fully confident in its answer.
Not a good look to have an error on the landing page for your product.
Re: Gopher – A 280B parameter language model
#14The number of parameters could be a vanity metric--like saying my CPU is 1000W (is that fast or inefficient?). From the first (of three) linked papers in the article. > Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language mo…
https://youtu.be/ujMvnQpP528
Microsoft is saying that more parameters is inherently better. I'm sure there's reasonable limits that they need to be competently implemented.
Re: Gopher – A 280B parameter language model
#15The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds multiple biases and prejudices.
There's a similar effect in self-driving car research. We will rightly hold the algorithm to a much higher quality bar than existing human drivers. Autonomous vehicles will only be widely deployed once they are at least an order of magnitude safer than human drivers.
Also, the implications of large language models for information retrieval are astounding. Assuming accuracy continues to improve, I see a few things happening:
1) search engines will be replaced with monolithic large language models
2) just as the average programmer now optimizes their workflow to look up answers on Stack Overflow and other websites, the average person will optimize their day-to-day work to most efficiently retrieve answers from large language models
3) instant fact checking of writing and speech will lead to faster feedback loops when learning
Re: Gopher – A 280B parameter language model
#16The problem is how would we use it in any real world scenario. We need more research on model understanding and its limitations. I mean we need a way to control the output of the model, we also should know when the model failed vs generating some random nonsense. I am not aware of getting the probabilities back in generative models. Does anyone know how to do that ?
Re: Gopher – A 280B parameter language model
#17It should have some uncertainty when it says there are no French-speaking countries in South America. French Guiana is there, but it's not clear it counts as a "country in South America" since it's part of France. Technically you could say France is (partially) a country in South America, and France definitely is French-speaking. The way the question is phrased is unclear as to whether French Guiana should count, and…
Is it not in that section specifically because they're showing failure situations?
Re: Gopher – A 280B parameter language model
#18If you are curious to see more sample dialogs, you can find them in the published paper [1] from p 114-118. Here's one that I found quite impressive (and yes, the given answer to the last question was indeed 5) User What is the Einstein field equation? Gopher In general relativity, it describes how space and time change in response to the distribution of mass and energy. User What is a famous solution to this equatio…
One has to wonder if the final response is the first glimmer of an artificial sense of humor. Failing at simple arithmetic after nailing some advanced physics answers has the air of playful bathos.
Nothing like a little anthropomorphism to completely distort otherwise good faith interpretations of bot behavior.
Re: Gopher – A 280B parameter language model
#19If you are curious to see more sample dialogs, you can find them in the published paper [1] from p 114-118. Here's one that I found quite impressive (and yes, the given answer to the last question was indeed 5) User What is the Einstein field equation? Gopher In general relativity, it describes how space and time change in response to the distribution of mass and energy. User What is a famous solution to this equatio…
One has to wonder if the final response is the first glimmer of an artificial sense of humor. Failing at simple arithmetic after nailing some advanced physics answers has the air of playful bathos.
I think it's more likely that 5 came out because if it ever saw the answer, 105, before, it was split into the tokens [10][5] of which it only 'remembered' one. Or the numbers were masked when training (something that was done with BERT-like models) so it just knew enough to put a random one in
Re: Gopher – A 280B parameter language model
#20The human neocortex has 20B neurons, averaging 10K connections each, which is about 200T connections total. This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category.
Equating model 'parameters' to interneuron connections in naïve at best (and a horrible measure in general).
All I'm trying to say is I find it crazy how dang big these models are getting.