Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gopher – A 280B parameter language model

deepmind.com

Post reply on HN
Print view View original

101–110 of 127 posts

Re: Gopher – A 280B parameter language model

#101

Post by akie »

periheli0n wrote: post #68

It confuses the hell out of me to have a super-powerful knowledge-extraction system that is right most of the time with super-complicated stuff, but also expresses horribly wrong statements with equal assertiveness. Just like those guys who march through middle management up to the exec floor within a few years. Very impressive, but not very useful to extract knowledge!

> Just like those guys who march through middle management up to the exec floor within a few years.

If they can do it, so can you.

If both the response to the corona virus, and the presidency of Trump have shown us ONE thing, it's that those who are higher up in the pyramid are NOT necessarily more clever or better informed than you are.

Re: Gopher – A 280B parameter language model

#102

Post by aix1 »

gattilorenz wrote: post #100
imachine1980_ wrote: post #55

Earlier quoted context omitted.

>Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval I agree in the same way than 70 % of people have less 100 iq, we depend in specialist when I need to know if I have epilepsi I need a person/thing who work is be up to date, and have the less bias as possible and agregator models are quite usually miss in understand what is crital info, try to program only whit gith…

> 70 % of people have less 100 iq Being pedantic here, but isn't that more like ~50%, by definition? Or did I misunderstand how IQ works?

You're exactly right: by construction, IQ scores are normally distributed with a mean of 100 and a standard deviation of 15.

Re: Gopher – A 280B parameter language model

#103

Post by nerdponx »

akie wrote: post #101
periheli0n wrote: post #68

It confuses the hell out of me to have a super-powerful knowledge-extraction system that is right most of the time with super-complicated stuff, but also expresses horribly wrong statements with equal assertiveness. Just like those guys who march through middle management up to the exec floor within a few years. Very impressive, but not very useful to extract knowledge!

> Just like those guys who march through middle management up to the exec floor within a few years. If they can do it, so can you. If both the response to the corona virus, and the presidency of Trump have shown us ONE thing, it's that those who are higher up in the pyramid are NOT necessarily more clever or better informed than you are.

Their "handlers" are absolutely more clever than most of us.

Re: Gopher – A 280B parameter language model

#104

Post by aix1 »

rdedev wrote: post #45
axg11 wrote: post #29

Earlier quoted context omitted.

Accuracy is improving rapidly though. I agree that the current accuracy levels are not high enough to be relied upon. > Humans ... they will usually recognize total blunder I question this assumption. I don't believe this is true, even for subject matter experts. I've worked with radiology data where experts with 10+ years of experience make blunders that disagree with a consensus panel of radiologists.

Radiology is one of those fields were a lot of it comes from experience and intuition mostly because of how complex the human body is. Compare this to a physicist; pretty sure you wont get as much disagreements there.

Your comment reminded me of this fascinating work:

Levenson RM, Krupinski EA, Navarro VM, Wasserman EA (2015) Pigeons (Columba livia) as Trainable Observers of Pathology and Radiology Breast Cancer Images. PLoS ONE 10(11): e0141357. https://doi.org/10.1371/journal.pone.0141357

https://journals.plos.org/plosone/article?id=10.1371/journal...

Re: Gopher – A 280B parameter language model

#105

Post by aix1 »

ska wrote: post #74
13415 wrote: post #49

Earlier quoted context omitted.

I realize I should have been more precise. I agree that there are many areas in which AI can and already has excelled humans and less often makes grave mistakes than humans. I specifically had natural language processing with a focus on "intelligent" conversation in mind. The issues in that area might have less to do with the pattern recognition ability and more with the lack of appropriate meta-cognition, introspect…

> AI can and already has excelled humans and less often makes grave mistakes than humans. Radiology, to continue the example, isn't one of them. We've been doing ML/AI in radiology data since the 90s, and results have been, and remain, decidedly mixed.

A couple of points:

1. It's easy to forget how recent many of modern ML methods for computer vision are. (E.g. U-net only goes back to 2015!)

2. It's not totally clear to be what you mean by "mixed results" (have we solved every problem in radiology? probably not). However, it is clear that there certainly have been some successes. Here's one example:

https://www.nature.com/articles/s41586-019-1799-6.epdf?autho...

Re: Gopher – A 280B parameter language model

#106

Post by geoffreyirving »

gwern wrote: post #61
rllearneratwork wrote: post #34

why DeepMind's papers all have logos and copyright statements on them and are hosted not on arxiv.org ? This looks so weird.

Most of them are hosted on Arxiv (like OpenAI) - there's one on the front page right now, https://news.ycombinator.com/item?id=29481665 . I link literally hundreds of DM Arxiv papers on my site. This includes most (all?) of the ones they host on their site too (with some exceptions with the prestige Nature publications - I think they don't want to step on toes there). My best guess is that they want a reasonably coor…

https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04359

Gopher soon. :)

Re: Gopher – A 280B parameter language model

#107

Post by ska »

aix1 wrote: post #105
ska wrote: post #74

Earlier quoted context omitted.

> AI can and already has excelled humans and less often makes grave mistakes than humans. Radiology, to continue the example, isn't one of them. We've been doing ML/AI in radiology data since the 90s, and results have been, and remain, decidedly mixed.

A couple of points: 1. It's easy to forget how recent many of modern ML methods for computer vision are. (E.g. U-net only goes back to 2015!) 2. It's not totally clear to be what you mean by "mixed results" (have we solved every problem in radiology? probably not). However, it is clear that there certainly have been some successes. Here's one example: https://www.nature.com/articles/s41586-019-1799-6.epdf?autho...

That paper is a good example actually. The first clinically approved, commercially available breast screening CADe system using NN models was available in the 90s. It too was aimed at the 2nd reader problem. At the time there was a lot of concern radiology circles that algorithms were going to take over. That pretty quickly died down as people worked with CADe and CAD systems.

Breast is one of the obvious targets, as data availability is pretty good. So over 25 years there have been incremental improvements, sure, but no real eye opening jumps, The move to deep models has helped a bit but nothing revolutionary. You still find very influential radiologists who aren’t convinced it’s worth the time, yet. I think all of them expect it to be a growing part of the workflows over time but that’s about it. Personally I think the impact will both be much bigger than the pessimistic radiologists think, and much further off than the optimistic ML think (for both non technical and technical reasons).

I suspect the broader availability of good digital data has had far more impact than the modeling updates. Don’t get me wrong, I appreciate the tools and modeling developed over the last decade - but I think the big wins are far more about data, and secondarily compute availability than about models.

Re: Gopher – A 280B parameter language model

#108

Post by gwern »

Nihilartikel wrote: post #8
credit_guy wrote: post #7

If you are curious to see more sample dialogs, you can find them in the published paper [1] from p 114-118. Here's one that I found quite impressive (and yes, the given answer to the last question was indeed 5) User What is the Einstein field equation? Gopher In general relativity, it describes how space and time change in response to the distribution of mass and energy. User What is a famous solution to this equatio…

One has to wonder if the final response is the first glimmer of an artificial sense of humor. Failing at simple arithmetic after nailing some advanced physics answers has the air of playful bathos.

One definitely has to wonder. We know that GPT-3 solves 1/2 digit arithmetic pretty much perfectly†; people criticized this very hotly as "it's just memorizing", but regardless, whether it learned or memorized arithmetic, that should apply even more to Gopher, which performs so much better and is larger. How can GPT-3 solve similar arithmetic near-perfectly and Gopher then be unable to...? Are we going to argue that "15 x 7" never appears in Internet scrapes and that's why Gopher couldn't memorize it?

I would want to ask it "15 x 7" outside of a dialogue or with examples, or look at the logprobs, or check whether "15 * 7" works (could there be something screwed up in the tokenization or data preprocessing where the 'x' breaks it? I've seen weirder artifacts from BPEs...). GPT-3 does not always 'cooperate' in prompting or dialogues or read your mind in guessing what it 'should' say, and there's no reason to expect Gopher to be any different. The space before the question mark also bothers me. Putting spaces before punctuation in Internet culture is used in a lot of unserious ways, wouldn't you agree 〜

I definitely would not hastily jump to the conclusion, based on one dialogue, "ah yes, despite its incredible performance across a wide variety of benchmarks surpassing GPT-3 by considerable margins and being expected to do better on arithmetic than GPT-3, well, I guess Gopher just can't multiply 1-digit numbers or even guess the magnitude or first digit of the result! What a pity!"

† quickly checking, GPT-3 can solve '15 x 7 =105'.

Re: Gopher – A 280B parameter language model

#109

Post by ska »

aix1 wrote: post #105
ska wrote: post #74

Earlier quoted context omitted.

> AI can and already has excelled humans and less often makes grave mistakes than humans. Radiology, to continue the example, isn't one of them. We've been doing ML/AI in radiology data since the 90s, and results have been, and remain, decidedly mixed.

A couple of points: 1. It's easy to forget how recent many of modern ML methods for computer vision are. (E.g. U-net only goes back to 2015!) 2. It's not totally clear to be what you mean by "mixed results" (have we solved every problem in radiology? probably not). However, it is clear that there certainly have been some successes. Here's one example: https://www.nature.com/articles/s41586-019-1799-6.epdf?autho...

> It's not totally clear to be what you mean by "mixed results" (have we solved every problem in radiology? probably not)

To expand a bit as maybe not clear from my other reply (can't edit). Not only have we not solved every problem in radiology, we haven't really knocked a single one out of the park.

By mixed results, I mean that the practical, i.e. clinical impact of these approaches has been pretty small, and this is likely to continue to be true for foreseeable future To be fair, there are lots of non-technical and cultural issues behind this - not just failure to generalize.

Re: Gopher – A 280B parameter language model

#110

Post by freemint »

YeGoblynQueenne wrote: post #84
isaacimagine wrote: post #20

The human neocortex has 20B neurons, averaging 10K connections each, which is about 200T connections total. This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category. Equating model 'parameters' to interneuron connections in naïve at best (and a horrible measure in general). All I'm trying to say is I find it crazy how dang big these models are getting…

>> This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category. A few orders of magnitude and an entire category away. Artificial "neurons" only have the name "neuron" in common with biological neurons. Consequently you can stack as many layers of artificial neurons on top of each other as you may want and you won't get anywhere near the abilities of the…

I think AI is actuator constrained compared to a spider which might be the larger problem.
Post reply on HN