Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gopher – A 280B parameter language model

deepmind.com

Post reply on HN
Print view View original

71–80 of 127 posts

Re: Gopher – A 280B parameter language model

#71

Post by fault1 »

axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

> closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should).

This is true, and in some regard, how it has always been.

https://en.wikipedia.org/wiki/AI_effect

Already, outside of obvious marketing, a lot of uses of the current wave of AI/deep learning have stopped becoming magical.

Re: Gopher – A 280B parameter language model

#72

Post by freediver »

gk1 wrote: post #58
axg11 wrote: post #15

The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…

> 1) search engines will be replaced with monolithic large language models This is already well under way. It's called vector search[1]. Google, Bing, Facebook, Spotify, Amazon, etc etc already use this to power their search and recommender systems. There are even a bunch of companies popping up (I work for one[2]) that let everyone else get in on the fun. Check out this video with the creator of SBERT / SentenceTran…

IMO vector search is pretty much a solved problem with simple to use, open source libraries like Faiss offering incredible performance out of the box for most commercial use cases.

A much harder problem is creating accurate vectors to begin with. Even the most advanced language models today create word/sentence embeddings that leave a lot to be desired. Not to mention this is slow and GPU intensive.

Creating an end-to-end solution for embedding/searching/ranking (of which vector search is just one component, the other one should be some kind of keyword based search to increase precision) is what would be very valuable to offer as a service.

Re: Gopher – A 280B parameter language model

#73

Post by freediver »

This model like its predecessor still lacks the ability to infer within constraints of physical world [1]. It does not have any notion of what a physical world is, and still is, at the end of the day, a statistical model.

Attempting to do recreate the entirety of the physical world by describing it in text is an impressive feat, but a futile attempt.

It is prone to inference errors that the model itself is not aware of. There are less and less of such errors, but the very inability to tell that what you are outputting is gibberish is one of the biggest problems of these models. In other words, if the model could simply say 'I do not know' or 'I am not sure' for every occasion when it is not 100% sure in something (like a human can), this would drastically improve the usefulness.

[1] Spent a lot of time pondering on this topic https://arxiv.org/pdf/1906.01873.pdf

Re: Gopher – A 280B parameter language model

#74

Post by ska »

13415 wrote: post #49
axg11 wrote: post #29

Earlier quoted context omitted.

Accuracy is improving rapidly though. I agree that the current accuracy levels are not high enough to be relied upon. > Humans ... they will usually recognize total blunder I question this assumption. I don't believe this is true, even for subject matter experts. I've worked with radiology data where experts with 10+ years of experience make blunders that disagree with a consensus panel of radiologists.

I realize I should have been more precise. I agree that there are many areas in which AI can and already has excelled humans and less often makes grave mistakes than humans. I specifically had natural language processing with a focus on "intelligent" conversation in mind. The issues in that area might have less to do with the pattern recognition ability and more with the lack of appropriate meta-cognition, introspect…

> AI can and already has excelled humans and less often makes grave mistakes than humans.

Radiology, to continue the example, isn't one of them. We've been doing ML/AI in radiology data since the 90s, and results have been, and remain, decidedly mixed.

Re: Gopher – A 280B parameter language model

#75

Post by buddhistdude »

freediver wrote: post #73

This model like its predecessor still lacks the ability to infer within constraints of physical world [1]. It does not have any notion of what a physical world is, and still is, at the end of the day, a statistical model. Attempting to do recreate the entirety of the physical world by describing it in text is an impressive feat, but a futile attempt. It is prone to inference errors that the model itself is not aware…

Your mom’s a statistical model

Re: Gopher – A 280B parameter language model

#76

Post by selfhoster11 »

halfdan wrote: post #32
selfhoster11 wrote: post #21

Can we please stop calling new tech "Gopher"? It's a name that belongs to a network protocol, not to a programming language or an AI model.

The animal called Gopher would like to have a word with you...

That too.

Re: Gopher – A 280B parameter language model

#77

Post by selfhoster11 »

ferdowsi wrote: post #25
selfhoster11 wrote: post #21

Can we please stop calling new tech "Gopher"? It's a name that belongs to a network protocol, not to a programming language or an AI model.

No. The network protocol lost cultural rights to the name by its lack of success.

1. Arguably, the HTTP + HTML web failed. It was designed as a pemissionless hypertext document language, yet more and more of its usage gets rid of every single one of these properties - non-hyperlinkable applications that beg for permission from large monopolies to post something.

2. Success is not the only metric of worth. Nuclear power has "failed" despite being the only reliable solution to meeting the base load of a power grid vs fossil fuels.

Re: Gopher – A 280B parameter language model

#78

Post by advisedwang »

The second paper cites the "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" paper that got Timnit Gebru and Meg Mitchell fired from Google. It must be pretty galling to get fired for a paper and then have the company that fired you cite it a year later.

Re: Gopher – A 280B parameter language model

#79

Post by the8472 »

bee_rider wrote: post #60
the8472 wrote: post #54

Earlier quoted context omitted.

> We will rightly hold the algorithm to a much higher quality bar than existing human drivers. Rightly? Delaying a life-saving measure when it's the #9 cause of death? They should be deployed selectively as soon as they can make the most risk-prone situations safer than the cohort of involved human drivers. E.g. getting drunk people home. We don't have to wait until they surpass the average driver because the average…

I can definitely see your argument, but there are some downsides to deploying ASAP: * Long term, the reputational damage to self driving cars might be significant if one is released that is only slightly better than a drunk driver. If this hinders uptake later, when self-driving cars are completely superior to normal humans, we've possibly produced a net negative. * While a self driving car should be safer than a dru…

This is mostly worrying about hard to predict second-order effects. Exactly the same problem that caused various health organizations (including WHO and FDA) send misleading messages around corona and vaccinations.

> Issues around who is responsible when a self driving car which is known to be worse than a typical human is deployed, fails, and someone is hurt.

Even if insurance rates would be initially(!) higher than for a regular taxi driver I would expect it to end up cheaper overall and save lives.

Also note that "worse-than-the-old-average" may still end up improving the average because the magnitude by which they're worse is lower. Also, there's nothing magical about "worse-than-average" because for all distributions (except the dirac distribution) there is always some fraction worse than the average, so we always have to expect some fraction of drivers to be worse than average and there's no fundamental reason why they must be human. In fact, if the fleet of all cars in the world consisted of very safe, very consistently behaving autonomous cars, except for a single vehicle driven by a very conservative human driver who only slowly circles in a fenced private community and never hits anyone then statistically the entire autonomous car fleet may be "worse than average" due to the single outlier.

Re: Gopher – A 280B parameter language model

#80

Post by johncena33 »

advisedwang wrote: post #78

The second paper cites the "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" paper that got Timnit Gebru and Meg Mitchell fired from Google. It must be pretty galling to get fired for a paper and then have the company that fired you cite it a year later.

Please stop spreading disinformation. Mitchell got fired for leaking company documents to outsiders. When people spread obvious lies, that makes me thing they don't have anything substantial to begin with. that's why they need to resort to lying.
Post reply on HN