Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is GPT-3? written in layman's terms

tinkeredthinking.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: What is GPT-3? written in layman's terms

#51

Post by logifail »

WA wrote: post #41
thimkerbell wrote: post #36

I have seen comments on Reddit that smelled as though something like this was used to create them. I don't see how, in future, we are going to be able to meet up and discuss constructively with strangers online, if this thing can be used to emulate people.

Or imagine the amount of useless blog posts for SEO spam. We already suffer from information overload. This will add a lot more.

> We already suffer from information overload

I would suggest this isn't quite right, we actually suffer from content overload, not information overload.

The quantity of useful information in a lot of the content we are offered is depressingly small.

Re: What is GPT-3? written in layman's terms

#52

Post by ypcx »

travbrack wrote: post #20
codingslave wrote: post #11

A comment from r/machinelearning: "It seems obvious from the demos that GPT-3 is capable of reasoning. But not consistently. It would be critical, imo, to see if we can identify a pattern of activity in it associated with the lucid responses vs activity when it prodcues nonsense. If/when we have such a apattern we would need to find a way to enforce it to happen in every interaction" And people agree: "Dunno why you…

I think the downvotes are from people who don't want to believe that AGI is possible, or that we're taking baby steps towards it.

It's not a matter of belief at this time. The evidence is right in front of our eyes. But it takes intelligence to recognize intelligence. Contextual extension by abstract inference is the basic (and hardest to achieve) building block of AGI, and we have achieved it. The rest is about utilizing this same power for the querying (priming) part of the reasoning loop, within the contexts we are interested in.

Part of me is glad we're not exactly there yet, because the thought of this running autonomously in a thinking cycle is downright scary. What will you find when you sit behind the console in the morning? It took us months to start understanding this in its current one-shot mode.

I don't care about the ideological downvotes, but we will do better if we start taking this very seriously. It's no longer theoretical that this (and machine learning as such) will have unprecedented (and impossible to predict) impact on everything we know, and the timeline is now measured in months instead of years or decades.

Re: What is GPT-3? written in layman's terms

#53

Post by T-A »

solidasparagus wrote: post #17

I have some issues with how this article describes GPT and its impact. The scenarios that are described as using GPT-3 remains the stuff of science fiction as far as I know. They would require major technological breakthroughs in areas of autonomous agents, combining language models with 'concept-based' models, and more mundane things like labelling (can you imagine how difficult it would take to accurately label all…

Neither scenario at the end of the article (therapist, legal analysis) is particularly futuristic.

From 2019: https://abilitynet.org.uk/news-blogs/eliza-ellie-evolution-a...

From 2018: https://www.techspot.com/news/77189-machine-learning-algorit...

Re: What is GPT-3? written in layman's terms

#54

Post by dddbbb »

Trasmatta wrote: post #28
pontifier wrote: post #26

Earlier quoted context omitted.

I wonder if that's one reason they can't open it up. Perhaps they are ensuring that responses that come from the AI are sufficiently different from anything in the training corpus, so it can't be queried for sensitive data or large chunks of copyrighted material.

It's basically already open through AI Dungeon. I've had all sorts of conversations so far with copyrighted characters.

That's still coming via the API. They may be blocking any large chunks of text from being reproduced, rather than individual characters (which would be harder to police).

Re: What is GPT-3? written in layman's terms

#55

Post by TomMarius »

SaltyLemonZest wrote: post #23
codingslave wrote: post #11

A comment from r/machinelearning: "It seems obvious from the demos that GPT-3 is capable of reasoning. But not consistently. It would be critical, imo, to see if we can identify a pattern of activity in it associated with the lucid responses vs activity when it prodcues nonsense. If/when we have such a apattern we would need to find a way to enforce it to happen in every interaction" And people agree: "Dunno why you…

I'm not convinced that "capable of reasoning, but not consistently" is a meaningful claim. The examples seem to primarily consist of people spending hours trying things, until eventually GPT-3 outputs a chunk of reasoning they could personally do in seconds. Does that mean that GPT-3 is doing the reasoning, or does it mean that GPT-3 is an English-based lookup table and they managed to find a clever sequence of searc…

Have you seen the database prompt?

https://www.gwern.net/GPT-3#the-database-prompt

Re: What is GPT-3? written in layman's terms

#56

Post by YeGoblynQueenne »

solidasparagus wrote: post #17

I have some issues with how this article describes GPT and its impact. The scenarios that are described as using GPT-3 remains the stuff of science fiction as far as I know. They would require major technological breakthroughs in areas of autonomous agents, combining language models with 'concept-based' models, and more mundane things like labelling (can you imagine how difficult it would take to accurately label all…

>> Please stop telling people that neural nets are related to brain neurons. They have essentially no relationship other than the name and it just fosters this fear of Terminator and obscures the real issues that need to be thought about.

Indeed, this is supported by the opinions of the foremost experts in deep learning and neural networks:

IEEE Spectrum: We read about Deep Learning in the news a lot these days. What’s your least favorite definition of the term that you see in these stories?

Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver.

https://spectrum.ieee.org/automaton/artificial-intelligence/...

Re: What is GPT-3? written in layman's terms

#57

Post by YeGoblynQueenne »

bozzcl wrote: post #8

> GPT-3 continuation: A confused voice came from inside. When I opened the door, the person that looked back at me was Hayama Hayato. Why was Hayama, who I only shared memories of me playing soccer with, in my room at this hour of the night? That question immediately flew out from my mouth. [I knew that name felt familiar]( https://oregairu.fandom.com/wiki/Hayato_Hayama ). Does that mean GPT-3 was trained on an arbit…

>> [I knew that name felt familiar](https://oregairu.fandom.com/wiki/Hayato_Hayama). Does that mean GPT-3 was trained on an arbitrary, huge database of text? I wonder how copyright applies here.

It could have lifted the name (and some additional context in the generated sentence) from the fandom wiki you link to, or something similar. It probably wasn't trained with the text of light novels; even though you can er find some of those online, they are generally scans and GPT-3 is trained on text, as far as I can tell.

In any case if it was lifted from sources about Oregairu and not the light novel itself, then it'd most likely be considered fair use. I mean, there's a wikipedia article that describes the characters (including Hayato Hayama) and all.

P.S. I haven't read that one. Is it any good?

Re: What is GPT-3? written in layman's terms

#58

Post by solidasparagus »

T-A wrote: post #53
solidasparagus wrote: post #17

I have some issues with how this article describes GPT and its impact. The scenarios that are described as using GPT-3 remains the stuff of science fiction as far as I know. They would require major technological breakthroughs in areas of autonomous agents, combining language models with 'concept-based' models, and more mundane things like labelling (can you imagine how difficult it would take to accurately label all…

Neither scenario at the end of the article (therapist, legal analysis) is particularly futuristic. From 2019: https://abilitynet.org.uk/news-blogs/eliza-ellie-evolution-a... From 2018: https://www.techspot.com/news/77189-machine-learning-algorit...

Neither of those are particularly compelling IMO.

The first is a digital information gatherer that is explicitly not playing the role of a therapist (likely for the reasons I mentioned above around risk/cost of failure, difficulty in evaluating what constitutes 'good advice', and probably legal barriers). There is a world of difference between a chatbot that does information retrieval and an autonomous agent that provides therapy.

The second is also vastly different from the "summarize legislation and detect 'nefarious' clauses" scenario in the article. They are identifying errors in standard NDAs, which is a (comparatively) well-defined, straightforward supervised learning task where the data has a pretty consistent shape unlike congressional legislation and what 'nefarious' means. (I have to make some assumptions here since as far as I can tell there isn't a technical paper on the work).

These sound similar to the tasks in the article, but once you get into the details of implementing and deploying them, I don't think they're very similar. You would still need to solve the problems that have plagued self-driving cars for years: high cost of failure, unpredictable failure modes, a long-tail distribution of data that prevents realistic-to-collect data sets from generalizing well enough, almost no progress in AI for autonomous agents (RL is promising, but it hasn't really made it out of the lab yet AFAICT).

Re: What is GPT-3? written in layman's terms

#59

Post by YeGoblynQueenne »

ypcx wrote: post #52
travbrack wrote: post #20

Earlier quoted context omitted.

I think the downvotes are from people who don't want to believe that AGI is possible, or that we're taking baby steps towards it.

It's not a matter of belief at this time. The evidence is right in front of our eyes. But it takes intelligence to recognize intelligence. Contextual extension by abstract inference is the basic (and hardest to achieve) building block of AGI, and we have achieved it. The rest is about utilizing this same power for the querying (priming) part of the reasoning loop, within the contexts we are interested in. Part of me…

>> Contextual extension by abstract inference is the basic (and hardest to achieve) building block of AGI, and we have achieved it.

What is "contextual extension by abstract inference" and why do you say it's "the basic building block of AGI"? Can you point to an authoritative source for the two parts of the statement (i.e. a source that defines "contextual extension by abstract inference" and a source that asserts this is "the basic building block of AGI")?

Re: What is GPT-3? written in layman's terms

#60

Post by gverrilla »

logifail wrote: post #51
WA wrote: post #41

Earlier quoted context omitted.

Or imagine the amount of useless blog posts for SEO spam. We already suffer from information overload. This will add a lot more.

> We already suffer from information overload I would suggest this isn't quite right, we actually suffer from content overload, not information overload. The quantity of useful information in a lot of the content we are offered is depressingly small.

I would suggest this isn't quite right, we suffer from advertisement overload and "content" is actually a "hype" word - itselft a product of advertisement (propaganda).
Post reply on HN