Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's GPT-3 may be the biggest thing since Bitcoin

maraoz.com

Post reply on HN
Print view View original

541–550 of 554 posts

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#541

Post by MauranKilom »

abernard1 wrote: post #518
MauranKilom wrote: post #434

Earlier quoted context omitted.

Throw some novel text prompt/task at it and see what happens. If it was just "glorified lookup tables" then the result should be consistently garbage. Note in particular that "like a function that can do the same computation with fewer inputs" maps very well to GPT-3 - it can complete many interesting tasks by just having a few samples provided to it, instead of having to fine-tune it with more training.

> Note in particular that "like a function that can do the same computation with fewer inputs" maps very well to GPT-3 - it can complete many interesting tasks by just having a few samples provided to it, instead of having to fine-tune it with more training. The reason it doesn't need more training is because it's already trained itself with millions of lifetimes of human data and encoded that in the parameters! Huma…

> The reason it doesn't need more training is because it's already trained itself with millions of lifetimes of human data and encoded that in the parameters!

You know what else is trained by the experiences of thousands of individual (and billions of collective) human lifetimes of data? And several trillions of non-human ones?

> Humans aren't born trained with data.

That's either very wrong or about to evolve into a no true scotsman regarding what counts as data.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#542

Post by abernard1 »

fsflover wrote: post #536
ianhorn wrote: post #497

Earlier quoted context omitted.

GPT-3 was trained on half a trillion words (common crawl, webtext, two book corpuses, and wikipedia, IIRC). At about 100 words per minute, that's almost ten thousand years of continuous speech. By my estimate it's probably a few thousand times what people actually hear in a lifetime. We don't experience nearly the volume of language that it did.

You forgot that we also absorb a much larger set of data through other senses.

Absolutely 100% agree.

Why then, the continued obsession with building single-media models?

Is focusing on the Turing test and language proficiency bringing us further away from the goals of legitimate intelligence?

I would argue "yes", which was my original comment. At no point in us trying to replicate what an adult sounds like have we actually demonstrated anything remotely like the IQ of a small child. And there's this big gap where it's implied by some that this process goes 1) sound like an adult -> 2) think like an adult, which seems to be missing the boat imo. (There's logically this intermediate step where we have this adult-sounding monster AI child.)

If we could constrain the vocabulary to that a child might be exposed to, the correlative trickery of these models would be more obvious. The (exceptionally good) quality of these curve fits wouldn't trick us with vocabulary and syntax that looks like something we'd say. The dumb things would sound dumb, and the smart things would sound smart. And maybe, probably even, that would require us fusing in all sorts of other experiential models to make that happen.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#543

Post by ekianjo »

graham_paul wrote: post #433
ekianjo wrote: post #97

Earlier quoted context omitted.

When most of your population is borderline starving the number of top engineers can be counted on very few hands in NK.

Still enough to build a nuclear arsenal. So that's probably enough to build AI talent

Their nuclear arsenal is built with Chinese and Russian brains, not NK's native technology.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#544

Post by ianhorn »

abernard1 wrote: post #542
fsflover wrote: post #536

Earlier quoted context omitted.

You forgot that we also absorb a much larger set of data through other senses.

Absolutely 100% agree. Why then, the continued obsession with building single-media models? Is focusing on the Turing test and language proficiency bringing us further away from the goals of legitimate intelligence? I would argue "yes", which was my original comment. At no point in us trying to replicate what an adult sounds like have we actually demonstrated anything remotely like the IQ of a small child. And there'…

> Why then, the continued obsession with building single-media models?

I think it's literally just working with available data. With some back of the envelope math, GPT-3's training corpus is thousands of lifetimes of language heard. All else equal, I'm sure the ML community would almost unanimously agree that thousands of lifetimes of other data with many modes of interaction and different media would be better. It would take forever to do and would cost insane amounts of money. But some kinds of labels are relatively cheap, and some data don't need labels at all, like this internet text corpus. I think that explains the obsession with single-media models. There's a lot more work to do and this is, believe it or not, still the low hanging fruit.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#545

Post by LdSGSgvupDV »

pjc50 wrote: post #370
thewarrior wrote: post #133

Earlier quoted context omitted.

I've started working on a version of GPT-2 which generates English text. The purpose of this is to improve its ability to predict the next character in a text, by having it learn 'grammatical rules' for English. It already works well for predicting the next character when it has seen only a small amount of text, but becomes less accurate as the amount of training text increases. I have managed to improve this by havi…

> Plot twist: This comment was generated by GPT-3 prompted with some of the comments in this thread. This kills the forum. Seriously, once this is weaponised, discussion of politics on the internet with strangers becomes completely pointless instead of just mostly pointless. You could potentially convince a human; you can't convince a neural net that isn't in learning mode.

Could you potentially convince a human for politics issue? It is extremely hard to convince stranger in the forum when there are some priors in mind.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#546

Post by abernard1 »

ianhorn wrote: post #544
abernard1 wrote: post #542

Earlier quoted context omitted.

Absolutely 100% agree. Why then, the continued obsession with building single-media models? Is focusing on the Turing test and language proficiency bringing us further away from the goals of legitimate intelligence? I would argue "yes", which was my original comment. At no point in us trying to replicate what an adult sounds like have we actually demonstrated anything remotely like the IQ of a small child. And there'…

> Why then, the continued obsession with building single-media models? I think it's literally just working with available data. With some back of the envelope math, GPT-3's training corpus is thousands of lifetimes of language heard. All else equal, I'm sure the ML community would almost unanimously agree that thousands of lifetimes of other data with many modes of interaction and different media would be better. It…

> thousands of lifetimes of other data with many modes of interaction and different media would be better.

But why not just 1 lifetime of different kinds of data? Heck, why not an environment of 3 years of multi-media data that a child would experience? That wouldn't cost insane amounts of money (or probably anything even close to what we've spent on deep learning as a species).

A corpus limited to the experiences of a single agent would create a very compelling case for intelligence if at the end of that training there was something that sounded and acted smart. It couldn't "jump the gun" as it were, by a lookup of some very intelligent statement that was made somewhere else. It would imply the agent was creatively generating new models as opposed to finding pre-existing ones. It'd even be generous to plain-ol'-AI as well as deep learning, because it would allow both causal models to explain learned explicit knowledge (symbolic), or interesting tacit behavior (empirical ML).

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#547

Post by ianhorn »

abernard1 wrote: post #546
ianhorn wrote: post #544

Earlier quoted context omitted.

> Why then, the continued obsession with building single-media models? I think it's literally just working with available data. With some back of the envelope math, GPT-3's training corpus is thousands of lifetimes of language heard. All else equal, I'm sure the ML community would almost unanimously agree that thousands of lifetimes of other data with many modes of interaction and different media would be better. It…

> thousands of lifetimes of other data with many modes of interaction and different media would be better. But why not just 1 lifetime of different kinds of data? Heck, why not an environment of 3 years of multi-media data that a child would experience? That wouldn't cost insane amounts of money (or probably anything even close to what we've spent on deep learning as a species). A corpus limited to the experiences of…

> But why not just 1 lifetime of different kinds of data? Heck, why not an environment of 3 years of multi-media data that a child would experience? That wouldn't cost insane amounts of money (or probably anything even close to what we've spent on deep learning as a species).

How would you imagine creating such an environment in a way that allows you to train models quickly?

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#548

Post by classified »

meredydd wrote: post #28

I am deeply enjoying this comment thread - it's a bit of a Barium Meal [0] for determining how many people read (a) the headline, (b) the first paragraph, or (c) the whole thing before jumping straight into the compose box. Having read to the bottom, the quality of text generation there absolutely blew me away. GPT-2 texts have a somewhat disconnected quality - "it only makes sense if you're not really paying attenti…

Eliza could do this better. Or, just use a Markov chain that has read enough corporate PR bullshit. It's just sad how many people use this "AI" meme to fulfill their need to worship something.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#549

Post by natch »

tgv wrote: post #280
natch wrote: post #254

Earlier quoted context omitted.

It’s actually about AI which is distinct from intelligence. >6 orders of magnitude less input That is utterly mistaken. We have the input of millions of generations of evolution which have shaped our brains and given us a lot of instinctive knowledge that we do not need to learn from environmental input that happens during our lifetime. Instead it was learned over the course of billions of years, during the lifetimes…

With that style of argumentation, you can say that NNs have even more input than humans: they also have all of the technical development of the last 50,000 years built into them.

Not really. Out evolution and existence in our current form rely on many things that have happened in the entire universe up to this point. But I’m not saying each of our brains and bodies encode all that information. We just benefit from it with an intricate physical structure that would have been difficult to create any other way.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#550

Post by tgv »

natch wrote: post #549
tgv wrote: post #280

Earlier quoted context omitted.

With that style of argumentation, you can say that NNs have even more input than humans: they also have all of the technical development of the last 50,000 years built into them.

Not really. Out evolution and existence in our current form rely on many things that have happened in the entire universe up to this point. But I’m not saying each of our brains and bodies encode all that information. We just benefit from it with an intricate physical structure that would have been difficult to create any other way.

And the same goes for GPT-3 and the resources it needs.
Post reply on HN