Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's GPT-3 may be the biggest thing since Bitcoin

maraoz.com

Post reply on HN
Print view View original

251–260 of 554 posts

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#251

Post by natch »

abernard1 wrote: post #187
natch wrote: post #172

Earlier quoted context omitted.

This kind of facile moving the goalposts is imho a cheap shot and (not imho, fact) is a recurring phenomenon as we make incremental progress toward AI. Progress is often made with steps that would have been astonishing a few years ago. And every time the bar is raised higher. Rightly so, but characterizing this as doing the wrong thing is missing the point of what we, and the system, are learning. Yes it's not intell…

> This kind of facile moving the goalposts is imho a cheap shot and (not imho, fact) is a recurring phenomenon as we make incremental progress toward AI. I think you missed my point. I think we're going in the wrong direction for AI entirely, and these "advances" are fundamentally misguided. OpenAI is explicitly about "intelligence," and so we should question if this is in fact that. It's clear that humans have funda…

But we’re not done yet. Give it time. We can go in plenty of directions. Just because you don’t think the current direction is right, that doesn’t rule out other directions happening. And who’s to say this stuff won’t end up being somehow useful? There’s a great talk on how trying to make progress toward an objective in evolutionary algorithms is not a good way to get there.

https://www.youtube.com/watch?v=dXQPL9GooyI

Of course evolutionary algorithms are just one direction as well. But that doesn’t mean that nothing else is happening.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#252

Post by foobarbecue »

p1esk wrote: post #185
cs702 wrote: post #147

Earlier quoted context omitted.

The thing that kills me is that to the vast majority of human beings the nonsensical technobabble above is probably indistinguishable from real, honest, logically consistent technobabble.[a] Soon enough, someone will replicate the Sokal hoax[b] with GPT-3 or another state-of-the-art language-generation model. It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the s…

If he didn't put in the last line ("plot twist ...") I'm pretty sure no one here on HN would have guessed it. In fact, while reading that comment I started to wonder why no one has tried to use GPT to generate text one character at a time. Or if someone has, what are the advantages and disadvantages over the BPE approach.

Your second paragraph is GPT-3?

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#254

Post by natch »

abernard1 wrote: post #187
natch wrote: post #172

Earlier quoted context omitted.

This kind of facile moving the goalposts is imho a cheap shot and (not imho, fact) is a recurring phenomenon as we make incremental progress toward AI. Progress is often made with steps that would have been astonishing a few years ago. And every time the bar is raised higher. Rightly so, but characterizing this as doing the wrong thing is missing the point of what we, and the system, are learning. Yes it's not intell…

> This kind of facile moving the goalposts is imho a cheap shot and (not imho, fact) is a recurring phenomenon as we make incremental progress toward AI. I think you missed my point. I think we're going in the wrong direction for AI entirely, and these "advances" are fundamentally misguided. OpenAI is explicitly about "intelligence," and so we should question if this is in fact that. It's clear that humans have funda…

It’s actually about AI which is distinct from intelligence.

>6 orders of magnitude less input

That is utterly mistaken.

We have the input of millions of generations of evolution which have shaped our brains and given us a lot of instinctive knowledge that we do not need to learn from environmental input that happens during our lifetime.

Instead it was learned over the course of billions of years, during the lifetimes of other organisms that preceded us.

Our brain structure was developed and tuned by all these inputs to have some built in pretrained models. That’s what instincts are. Billions of years in the making. Millions, at the very least, if you want to restrict it to recent primates, although doing so is nonsensical.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#255

Post by wadkar »

Serious question: what does (will?) GPT-3 (or GPT-2/BERT) produce if provided with the STOP token (end of sentence/period).

I want to know if there is any interesting investigation happening on the creativity by these models in a sort of “tabula rasa” spirit (if it makes sense)

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#256

Post by highfrequency »

A lot of people are saying that GPT-3 is a huge step up from GPT-2. This is confusing to me looking at the results in the paper as someone not too familiar with GPT-2.

The tasks where GPT-3 scores much higher than GPT-2 are the ones most removed from broad language comprehension / general intelligence: arithmetic and unscrambling tasks (as in: which character does not belong in the string "app$le"). On the other tasks it does 5-15% better. This seems like a modest improvement for a model that is thirteen times larger than its predecessor.

I don't mean to minimize the accomplishment--training models at such a massive scale is difficult. But genuinely wondering if I am missing something here--are people's reactions more from a qualitative judgement about the paragraphs produced by GPT-3 vs. GPT-2? If so, a non-cherry-picked side-to-side comparison would be great to see. If this is a big leap, why doesn't it show up the quantitative results?

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#257

Post by O_H_E »

nullc wrote: post #157
Dangeranger wrote: post #131

Earlier quoted context omitted.

Can you elaborate on how you are having GPT2 contribute to your comments? What is your process?

I write a prompt, often copying text from the articles or other comments, and have it generate a lot of completions. I skim over the completions and grab interesting parts. For example, complaining about "quoted text" in my above comment was GPT2's suggestion (and also an actual issue with GPT2 which it was exhibiting by producing that text). Actually, all the text above from "There are a few" and beyond were written…

.....MAN, I am not sure how I feel about this. Don't take this the wrong way, it definitely looks like you are just using technology to produce art/craft in your own way.

I will definitely look into this the next time I am writing cover letters — that I dread.

Tools/sites/docker-images that improve your workflow?

Damn, kudos again.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#258

Post by O_H_E »

3wolf wrote: post #44

It sounds like you have an output that has been generated by a recurrent neural network trained on a large corpus. The more I read about neural networks the more clear it becomes that there are some classes of them that just write some good stuff. The work of Andrej Karpathy on RNNs for poetry and other things comes to mind. Is your GPT-3 actually even a neural network, though? More research is needed! ^written in re…

Worthy to note that as another comment mentioned, GPT3 was likely trained on pre-GPT2 data.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#259

Post by selestify »

nullc wrote: post #138
javajosh wrote: post #130

Earlier quoted context omitted.

There are a few concrete things the (fake) article says, primarily about what software to try (OpenAI's GPT-3), and where to try it (bitcointalk forum). Personally, I actually resent being mislead like this, at least on that second point, even with the full disclosure. The output is very high quality, but it is making at least one falsifiable assertion (no test was ever done in that forum).

If you've ever spent much time with a toddler you might have noticed that they spout a lot of fantasy. Learning to not make up untrue claims takes years of additional training for humans.

I’ve never spent much time with toddlers. What do they make up things about? Their own actions, other’s actions, claims about the environment?
Post reply on HN