Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's GPT-3 may be the biggest thing since Bitcoin

maraoz.com

Post reply on HN
Print view View original

431–440 of 554 posts

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#431

Post by amelius »

pjc50 wrote: post #424
amelius wrote: post #393

Earlier quoted context omitted.

Perhaps it will destroy anonymity. Because the only way to be sure a human wrote something is if you somehow know who the comment came from. We might end up with reputation based conversations.

Of course, the human can simply lend their name to the robot. And as previously discussed, ending anonymity entrenches the existing power structure.

> Of course, the human can simply lend their name to the robot.

That could have consequences for their reputation, though.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#432

Post by graham_paul »

mcilai wrote: post #30
graham_paul wrote: post #8

Earlier quoted context omitted.

> Given the latest developments, I would put the odds at fifty-fifty within the next decade. thing is technological development is not linear, you can't predict future development based on the last n decades. You can't assume an AI winter is not coming because it most likely is

> because it most likely is "Ark estimates that Deep Learning has created $1 trillion in market value so far. " -- we might be headed towards a warm winter (from https://www.nextbigfuture.com/2020/01/ark-invests-big-five-t... )

Value is there until it isn't. A lot of the value here comes from its potential rather than its current use

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#433

Post by graham_paul »

ekianjo wrote: post #97
Rebelgecko wrote: post #57

Earlier quoted context omitted.

How much of a choice do they have? I can't imagine many of the top engineers in North Korea are allowed to leave

When most of your population is borderline starving the number of top engineers can be counted on very few hands in NK.

Still enough to build a nuclear arsenal. So that's probably enough to build AI talent

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#434

Post by MauranKilom »

abernard1 wrote: post #278
natch wrote: post #254

Earlier quoted context omitted.

It’s actually about AI which is distinct from intelligence. >6 orders of magnitude less input That is utterly mistaken. We have the input of millions of generations of evolution which have shaped our brains and given us a lot of instinctive knowledge that we do not need to learn from environmental input that happens during our lifetime. Instead it was learned over the course of billions of years, during the lifetimes…

>>6 orders of magnitude less input >That is utterly mistaken. I did say of data of the "same sort". What's absolutely crazy is somehow we think of our DNA base pairs as somehow more important than the physical context that DNA ends up in (society, humans, talking, etc.) We have the ability to be intelligent and make thoughts with 1 millionth the amount of textual data as this OpenAI GPT-3 study. Maybe... just maybe..…

Throw some novel text prompt/task at it and see what happens. If it was just "glorified lookup tables" then the result should be consistently garbage.

Note in particular that "like a function that can do the same computation with fewer inputs" maps very well to GPT-3 - it can complete many interesting tasks by just having a few samples provided to it, instead of having to fine-tune it with more training.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#435

Post by Joeri »

abernard1 wrote: post #278
natch wrote: post #254

Earlier quoted context omitted.

It’s actually about AI which is distinct from intelligence. >6 orders of magnitude less input That is utterly mistaken. We have the input of millions of generations of evolution which have shaped our brains and given us a lot of instinctive knowledge that we do not need to learn from environmental input that happens during our lifetime. Instead it was learned over the course of billions of years, during the lifetimes…

>>6 orders of magnitude less input >That is utterly mistaken. I did say of data of the "same sort". What's absolutely crazy is somehow we think of our DNA base pairs as somehow more important than the physical context that DNA ends up in (society, humans, talking, etc.) We have the ability to be intelligent and make thoughts with 1 millionth the amount of textual data as this OpenAI GPT-3 study. Maybe... just maybe..…

Humans are exposed to much more input than just the things they read. Think of every thing you've ever seen and how much data that represents. All of that is your training data. Is it more or less than GPT-3?

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#436

Post by saberience »

kevmo314 wrote: post #204
abernard1 wrote: post #196

Earlier quoted context omitted.

> This post was generated using GPT-3. [;)] Your post was generated using GPT-3 and 175 billion parameters of pre-existing human writing, contextualized, distilled, and cross-referenced with terminology we've agreed on for centuries. It's a parrot, and I remain unimpressed. Take the learned knowledge of GPT-3 (because it must be so smart right?) and have it actually do something. Buy stocks, make chemical formulas, b…

> That post was generated using GPT-3 and 175 billion parameters of pre-existing human writing, contextualized, distilled, and cross-referenced with terminology we've agreed on for centuries. So basically like DNA?

DNA is nothing like the training of GPT. DNA does not encord a massive amount of statistics of words and language and how concepts, words, etc, relate to one another.

All DNA does it encode for how to grow, build, and mantain a human body. That human body has the potential to learn a language and communicate, but if you put a baby human inside an empty room and drop in food, it will never learn language and never communicate. DNA isn't magic and comparing "millions of years of evolution" of DNA is nothing like the Petabytes of data that GPT-3 needs to operate.

Again DNA has no knowledge embedded in it, it has no words or data embedded. Data in the sense that we imagine Wikipedia stored in JSON files on a hard disk. DNA stores an algorithm for growth of a human, that's it.

The GPT-3 model is probably > 700GB in size. That is, for GPT to be able to generate text it needs an absolutely massive "memory" of existing text which it can recite verbatim. In contrast, young human children can generate more novel insights with many orders of magnitude less data in "memory" and less training time.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#438

Post by MauranKilom »

simonsarris wrote: post #80

I could tell this was GPT-3 because it reads like SEO-style sentence structure :/ > I imagine that similar results can be obtained by republishing GPT-3’s outputs to other message boards, blogs, and social media. I actually wrote a bit about this scenario and how it could explode comment sections to stonewall topics, which I'm calling Commentdämmerung: https://simonsarris.com/commentdammerung Still, I think a lot of…

Every time an article talks about training/finetuning/improving GPT-3, you know it's GPT-3 talking about itself. There's no option to do those things for GPT-3 outside of OpenAI.

More broadly, you can be very sure that a text about GPT-3 originated from GPT-3 if it makes more sense when you replace "GPT-3" by "GPT-2" everywhere. GPT-3 naturally has no knowledge of itself because all related text was written after its training. The closest is GPT-2 (for which there is apparently plenty of corpus to sample from), so that's the content that GPT-3 writes about when forced to talk about itself.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#439

Post by saberience »

nullc wrote: post #390
O_H_E wrote: post #257

Earlier quoted context omitted.

.....MAN, I am not sure how I feel about this. Don't take this the wrong way, it definitely looks like you are just using technology to produce art/craft in your own way. I will definitely look into this the next time I am writing cover letters — that I dread. Tools/sites/docker-images that improve your workflow? Damn, kudos again.

I just tried GPT2 at cover letters and I can't figure out a prompt that gets it to consistently write good ones, instead I can mostly get it to write bad ones (as in the sort of stuff you might see unsuccessful applicants submit). My example problem was to get it to apply to Y Combinator. It managed to misspell the organization which isn't something GPT2 normally does. Here are some examples: https://0bin.net/paste/R…

Maybe this is a dumb question, because you seem like someone who thinks in a radically different way to me. But why would you try and use GPT-2 to help you write better English versus actually learn to become a better writer yourself?

I say this as someone who reads extensively and is considered a "good" writer and have won prizes before for my writing. I became a better writer through reading more books (especially classic literature) and writing more. Cribbing off a computer algorithm and copying and pasting seems like it would just be hurting your own growth, as well as being lazy to boot.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#440

Post by GreenHeuristics »

greyface- wrote: post #110
jcahill wrote: post #53

Earlier quoted context omitted.

GPT-3 is a neat party trick. But the things that'll be done with web archives* in the next 20y will make it look like the PDP-8. ~love, a web archivist * GPT-3 is trained on one

This is an interesting thought. GPT-3 used 45TB of raw CommonCrawl data (which was filtered down to 570GB prior to training). The Internet Archive has 48PB of raw data.

That 48PB is mostly just old video game roms and isos though
Post reply on HN