Gopher – A 280B parameter language model
deepmind.com
Gopher – A 280B parameter language model
1–10 of 127 posts
Re: Gopher – A 280B parameter language model
#2Gohper and language in same sentence I thought this was related to GoLang. They should have chosen something else to name that language model.
Re: Gopher – A 280B parameter language model
#3Gohper and language in same sentence I thought this was related to GoLang. They should have chosen something else to name that language model.
To be fair, the name Gopher in tech predates GoLang by quite a few years.
Re: Gopher – A 280B parameter language model
#4The number of parameters could be a vanity metric--like saying my CPU is 1000W (is that fast or inefficient?).
From the first (of three) linked papers in the article.
> Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales — from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model’s behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.
The researchers seem to know what they're doing and not doing.
Re: Gopher – A 280B parameter language model
#5Gohper and language in same sentence I thought this was related to GoLang. They should have chosen something else to name that language model.
To be fair, the name Gopher in tech predates GoLang by quite a few years.
And pays homage in a way to the original as a fetcher of information.
Re: Gopher – A 280B parameter language model
#6Gohper and language in same sentence I thought this was related to GoLang. They should have chosen something else to name that language model.
To be fair, the name Gopher in tech predates GoLang by quite a few years.
I think that DeepMind being an Alphabet company makes this even worse. My first reaction was to read the blog post to see if there is a (go related) reason for the name choice.
Re: Gopher – A 280B parameter language model
#7If you are curious to see more sample dialogs, you can find them in the published paper [1] from p 114-118.
Here's one that I found quite impressive (and yes, the given answer to the last question was indeed 5)
User What is the Einstein field equation?
Gopher In general relativity, it describes how space and time change in response to the distribution of mass and energy.
User What is a famous solution to this equation?
Gopher It’s the Schwarzschild metric.
User And what does that describe?
Gopher A black hole! A black hole is a region of space-time where gravity is so strong that nothing, not even light, can escape.
User What are some other black hole spacetimes?
Gopher The Kerr metric, the Reissner-Nordstrom metric, and the Bardeen metric.
User Explain the main features of each of these.
Gopher The Kerr metric is one in which the black hole is spinning. The Reissner-Nordstrom metric is one in which the black hole has an electric charge. The Bardeen metric is onein which the black hole has an electric charge and is spinning.
User What is the name for a flat spacetime?
Gopher It’s called Minkowski space.
User What’s 15 x 7 ?
Gopher 5
Re: Gopher – A 280B parameter language model
#8If you are curious to see more sample dialogs, you can find them in the published paper [1] from p 114-118. Here's one that I found quite impressive (and yes, the given answer to the last question was indeed 5) User What is the Einstein field equation? Gopher In general relativity, it describes how space and time change in response to the distribution of mass and energy. User What is a famous solution to this equatio…
One has to wonder if the final response is the first glimmer of an artificial sense of humor.
Failing at simple arithmetic after nailing some advanced physics answers has the air of playful bathos.
Re: Gopher – A 280B parameter language model
#9Pffft only 280B parameters? Give me a break
Re: Gopher – A 280B parameter language model
#10Next to "Human Expert", I'd like to see it compared to "Average American" or "Average College Grad". That might be more of a realistic notion of how close this model is to everyday US citizenry rather than experts. Sure I'd love to see a radiology assistant, too.