Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI releases larger GPT-2 model

openai.com

Post reply on HN
Print view View original

31–40 of 87 posts

Re: OpenAI releases larger GPT-2 model

#31

Post by qlm »

jchw wrote: post #5
minimaxir wrote: post #2

Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…

>(e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2 ) Wow, thats great. “The Bullshit Bubble” “Fuck you, Bootstrap” “We should give up on America” - they’re practically comedy, yet very believable too.

My favourite is "The secret Nazi medical charity giving out freezers".

Very reminiscent of headlines from The Day Today: https://youtu.be/wdEcO8_2Kl8

Re: OpenAI releases larger GPT-2 model

#32

Post by Felz »

p1esk wrote: post #22
Felz wrote: post #18

Oh wow, I was literally just making a toy Discord bot for GPT-2. Guess I'll update it with the bigger model. EDIT: Done! It takes about 3x longer than it did before to generate a response, so if you try it be very patient. Also, I made this in three hours so I wouldn't be surprised if it goes up in flames at some point. https://github.com/ScottPeterJohnson/gpt2-discord

How long does it take, and which hardware do you use?

About 30 seconds for a smaller response on an EC2 instance. Mind you this is without a GPU because I couldn't figure out how to set one up.

Re: OpenAI releases larger GPT-2 model

#33

Post by ben_w »

lopmotr wrote: post #25
burtonator wrote: post #10

Earlier quoted context omitted.

This thing really does scare me and fascinate me at the same time. There's so many nefarious things you could do with this... Generate fake news. spam google. etc. One valuable use could be to generate comedy and parody. You could also make it to sabotage others too. You could set it lose on nazi forums and have them argue with bots constantly.

I think people are over-worried about fake material. We've been able to say lies since we became human. We've been able to print lies since printing. We can already edit photos. People routinely make fake screenshots of messages to defame others. Newspapers sometimes print fake stories based on lies given by their sources or which they fail to vet. But provenance still exists to prove things, not how convincing it lo…

If evidence alone was enough, there wouldn’t be any debate about antivaxxers, climate change, the Iraq WMDs, the 9/11 and moon landing conspiracies, Holocaust denial, the correct way to bring peace to the Israel/Palestine conflict, the safety of being white in a Muslim community or being black around an American cop, Brexit, any of the Assange drama, or if Snowden was acting appropriately by leaking what he leaked. And those are just the disagreements I’ve witnessed.

Just because lies and propaganda have always been around, it doesn’t mean they were never a problem.

Re: OpenAI releases larger GPT-2 model

#34

Post by bazza451 »

minimaxir wrote: post #2

Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…

This is really cool.

Just to pick your brain (I’m an ML noob and it triggered an idea in my head). I was wondering if GPT-2 could be used to generate small paraphrases for an input sentence for search suggestions? e.g “I’m going to the mall today” -> “I will be going to mall today”.

Re: OpenAI releases larger GPT-2 model

#35

Post by septor »

Here’s a summary: we are aware of the fact that this model will harm society and we are releasing it anyway. We are fiddling around with the way it’s released in an attempt to absolve ourselves of blame while simultaneously collecting the profit in the form of a juicy acquisition.

The net result of these advanced forms of signal processing will be negative. Nobody has come forward to prove that they will benefit society on the whole or even that they are safe. But anyone who raises concern is shouted down and called names like “alarmist” and “Luddite.”

These companies are playing with fire, and the whole world stands to be burned. Wake the fuck up.

Re: OpenAI releases larger GPT-2 model

#36

Post by antpls »

applecrazy wrote: post #7
minimaxir wrote: post #2

Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…

One example from your dataset that gave me a laugh: Hiring technical debt (or "unsortable overtime") How do you hire technical debt?? Edit: another one- “In 2009, Africa power creation was switched on for the Google Earth Darth Vader Imperial Warplane Propaganda” I guess Google is diversifying :P

I guess it went through that path :

-Hiring -> Software Developer

-Software Developer -> Code

-Code -> Technical Debt

You only have to hire a Software Developer :)

Re: OpenAI releases larger GPT-2 model

#37

Post by indalo »

leod wrote: post #30
minimaxir wrote: post #2

Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…

I haven't heard of gradient checkpointing yet, thank you for the link! Do you know how it compares to gradient accumulation? The latter basically reduces the batch size, but takes the sum of multiple gradients before actually performing an update, thereby having the same effect as the original batch size. The generated titles are great! You can put them into hncynic ( https://github.com/leod/hncynic ) to get closer t…

wow! I just had a blast putting titles into that. the results are amazing. kudos!

Re: OpenAI releases larger GPT-2 model

#38

Post by feanaro »

septor wrote: post #35

Here’s a summary: we are aware of the fact that this model will harm society and we are releasing it anyway. We are fiddling around with the way it’s released in an attempt to absolve ourselves of blame while simultaneously collecting the profit in the form of a juicy acquisition. The net result of these advanced forms of signal processing will be negative. Nobody has come forward to prove that they will benefit soci…

Someone is going to invent this model sooner or later, simply because it is possible. There is not much sense in trying to stop it. We just have to adapt.

Re: OpenAI releases larger GPT-2 model

#39

Post by opdahl »

Felz wrote: post #18

Oh wow, I was literally just making a toy Discord bot for GPT-2. Guess I'll update it with the bigger model. EDIT: Done! It takes about 3x longer than it did before to generate a response, so if you try it be very patient. Also, I made this in three hours so I wouldn't be surprised if it goes up in flames at some point. https://github.com/ScottPeterJohnson/gpt2-discord

I tried adding it but I get the error message "This bot is private. Only the bot owner can add it."

Re: OpenAI releases larger GPT-2 model

#40

Post by septor »

feanaro wrote: post #38
septor wrote: post #35

Here’s a summary: we are aware of the fact that this model will harm society and we are releasing it anyway. We are fiddling around with the way it’s released in an attempt to absolve ourselves of blame while simultaneously collecting the profit in the form of a juicy acquisition. The net result of these advanced forms of signal processing will be negative. Nobody has come forward to prove that they will benefit soci…

Someone is going to invent this model sooner or later, simply because it is possible. There is not much sense in trying to stop it. We just have to adapt.

That’s not correct. What you are saying is that there is no plausible organized effort that could stop or slow the creation of signal processing models that will have pronounced negative impacts. The error is on two levels: you are using too much analogy with other technologies. And you are writing off the possibility of stopping ai when it’s still not clear that it can’t be stopped.

This isn’t something that can be built and tested in isolation like other things we are familiar with. Training these models is not an exact science. Nothing about ai is an exact science. Progress only comes with trial and error. And each trial requires huge compute resources; at least for the most capable and dangerous models. It can’t be done in your basement. Not without significant effort and drawing attention to yourself. Could we sense whenever someone was trying to do it? Could we form a global coalition to stop every attempt? That brings us to the next thing.

What you are doing is the following: we are both in a car that is about to roll off a cliff. I propose that we try pressing the brakes. You respond by saying that, geez it looks like we probably wouldn’t stop in time — we are going awfully fast and it probably wouldn’t work to press the brakes so why even try? Let’s just brace our heads and hope the impact doesn’t kill us.

Obviously the better thing to do is to try and press the brakes. Even if you aren’t sure if you can stop in time.

Post reply on HN