Live data from Hacker News

OpenAI's GPT-3 may be the biggest thing since Bitcoin

maraoz.com

321–330 of 554 posts

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#321
post #238
post #233

Earlier quoted context omitted.

How many attempts did it take or did you just choose the first one? I have to admit, this is passing my turing test...

Really? I got about halfway through and realized that the comment had no point . If you tried to summarize what it was arguing, beyond the first sentence, I don't think you could make a coherent summary. Maybe the real lesson is we don't expect human-written comments on discussion fora to be particularly coherent....

I would not have imagined it was automatically written. Rambling and there's little connection between the first part and the latter, but absolutely something that might appear on a random internet forum.

I am genuinely awed.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#322
post #9

I published a response today to the sudden hype urging people to temper their expectations for GPT-3 a bit: https://minimaxir.com/2020/07/gpt3-expectations/ GPT-3 is objectively a step forward in the field of AI text-generation, but the current hype on VC Twitter misrepresents the model's current capabilities. GPT-3 isn't magic.

One of the biggest issues is with cherry-picking. Generative ML results benefit greatly from humans sampling the best results. They are capable of producing astonishing results but don’t do this consistently this has a huge impact on any effort to productize. For example I’ve seen quite a few examples of text->design, text->code, with GPT-3 you could build a demo in a day, but the product will probably be useless if…

Yes, it is more like happy path testing.

However I like spirit of optimism and first looks at encouraging and very promising results.

Exciting times!

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#323

It is ironic as those who now post GPT-3 generated content, is essentially biasing the next version of GPT's web training data

I wonder how much data it would take to actual cause noticeable (or triggerable) behaviors in the model. Like I've noticed certain models have been trained off of my university's course captures by their very odd and specific vocabulary/capitalization of specific terms but surely if you're scanning the entire web you'd need a lot more to sway it

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#325

Oh man. I don’t know where to begin. I’ll just say that this analysis is predicated on someone who thinks Bitcoin has proven anything in the real world. We’re way past BTC being a subculture experiment. Everyone knows about it. And still nobody (in the statistical sense) uses it.

a lot of people use it a lot.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#326
post #138

Earlier quoted context omitted.

If you've ever spent much time with a toddler you might have noticed that they spout a lot of fantasy. Learning to not make up untrue claims takes years of additional training for humans.

I’ve never spent much time with toddlers. What do they make up things about? Their own actions, other’s actions, claims about the environment?

My six year old will just continue as long as he has people's attention, and if that means he has to make things up, so be it. Freely stealing phrases from other recent conversations.

So this morning he heard about an animal, it was kind of a lion. But with bat's ears, it lives in Africa. It looks like it's a rock, but it's actually not, it's rock shaped but has tiny legs. And it's gray and hard. Its face... It doesn't really have a face. It lives up in trees where it eats bamboo and apples. It has these huge fangs like sabertooth tigers, you know?

It's glorious.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#327
post #138

Earlier quoted context omitted.

If you've ever spent much time with a toddler you might have noticed that they spout a lot of fantasy. Learning to not make up untrue claims takes years of additional training for humans.

I’ve never spent much time with toddlers. What do they make up things about? Their own actions, other’s actions, claims about the environment?

All of those, in my experience.

My smallest kid has a habit of telling stories about himself that actually come from whatever he heard recently, e.g. "once I was Godzilla..", or claims about things in reality that come from stories or misunderstandings all mixed up "did you know, there are three pigs, but they are not pigs, they are wolves and a hunter came and killed them but they weren't wolves they were dragons..."

It's actually very GPT-3-ish now that I think of it.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#328
post #65
post #53

Earlier quoted context omitted.

GPT-3 is a neat party trick. But the things that'll be done with web archives* in the next 20y will make it look like the PDP-8. ~love, a web archivist * GPT-3 is trained on one

No pressure: feel free to ignore me, please. Would you mind elaborating? I'm interested in what you have to say (and, of course, feel free to say it privately if you prefer). I would like to even hear your dreams, wild speculations, or gut feelings about the matter.

Sure, what do you want to know?

I currently work on synbio × web archival.

Some of us are cooking up futuretech aimed at storing all of IA (archive.org) in a shoebox. Others are working on putting archival tools in more normal web users' hands, and making those tools do things that people tend to value more in the short-term, like help them understand what they're researching, rather than merely stash pages.

My ambitions for web archives are outsized compared to other archivists, but I'm fine with that. I'm looking beyond web archives as we currently understand them toward web archives as something else that doesn't quite exist yet: everyday artefacts, colocated and integrated with other web technology to an extent that they serve in essential sensemaking, workflow, and maybe security roles.

Right now, some obvious, pressing priorities are (a) preserving vastly more content and (b) doing more with the archives themselves.

A: The overwhelming majority of born-digital content is lost within a far narrower time-slice than would admit preservation at current rates, and data growth is accelerating beyond the reach of conventional storage media. So, for me, the world's current largest x is never the true object of my desire. I'm after a way to hold the world that is and the world to come.

Ideally, that world to come is one where lifelong data stewardship of everything from your own genome to your digital footprint is ubiquitously available and loss of information has been largely rendered optional.

This, of course, requires magic storage density that simply defies fundamental limitations of conventional storage media. I'm strongly confident that we're getting early glimpses of the first real Magic contenders. All lie outside, or on the far periphery of, the evolutionary tree that got us the storage media we have today. For instance, I'm running an art exhibition that involves encoding all the works on DNA.

B: Distributed archival that comes almost as naturally as browsing is well within reach, and with that comes some very new potential for distributed computation on archives. One hand washes the other.

One important thing to realize here is that, in many cases, you can name a very small handful of individuals as the reason why current archival resources exist. GPT-3 is cracking the surface by training on data produced by one guy named Sebastian, for instance.

…i'm sorta tired and have to respond to something about every twitter snapshot since June being broken, though, so I'll pick this back up later.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#329
post #284

Earlier quoted context omitted.

> Even on re-reading more closely, it doesn't feel like the world's best writing, but I don't notice major loss of coherence until the last couple of paragraphs. I guessed it was fake before getting to the end, not from the content, but from the fact that all the sentences are roughly the same length and follow the same basic grammatical patterns. Real people purposely mix up their sentence structure in order to keep…

Besides predictable sentence structure, GPT-3 writes like George R. R. Martin: Interesting premises, solid setup but then it devolves into rambling tangents and never quite delivers the concluding action that ties everything together. Lots of examples I've seen have phrases like "see table below". Of course there's no table and it's hard to imagine how there could be. But GPT is trained on internet content and the in…

I am really curious how the model would be if you would train it with a decent amount of really good literature. Kazuo Ishiguro et al. instead of Reddit.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#330
post #147

Earlier quoted context omitted.

The thing that kills me is that to the vast majority of human beings the nonsensical technobabble above is probably indistinguishable from real, honest, logically consistent technobabble.[a] Soon enough, someone will replicate the Sokal hoax[b] with GPT-3 or another state-of-the-art language-generation model. It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the s…

It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the social sciences. And then, it'll be all over for us. We won't have any more funding and our jobs will disappear. I can already hear the protests: "But we're not just scientists! We're also philosophers!" Well, yes and no. Philosophers are supposed to think about things philosophically, but they don't actually d…

So it looks like we're about 2 years away from the 'Her' relationship model.
Post reply on HN