Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Releases Largest GPT-2 Text Generation Model

openai.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#61

Post by moultano »

490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

The notation in the code will be very familiar to anyone comfortable with the underlying research and math. The "conceptual" documentation is in the literature.

What you're asking for is the rough equivalent of asking a C programmer to name their loop variables "index" instead of "i." Everyone familiar with the concepts of c programming knows what "i" means in the context of a for loop. Similarly, everyone familiar with transformers knows what "gelu" and "attn" mean.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#62

Post by rm_-rf_slash »

AwesomeLemon wrote: post #37
rm_-rf_slash wrote: post #9

Earlier quoted context omitted.

There are too many edge cases to throw out the baby with the bathwater though. Reddit has a lot of really great niche communities that work hard to avoid toxicity (although to be fair I don’t go to anything news/politics related there, just memes and machine learning for me). Facebook, likewise, is a trash company with a trash product, but has completely replaced the town notice boards and meeting halls in rural comm…

Why can't reddit replace facebook as a notice board?

Fantastic question. The simple answer is that most people who are involved in community governance skew older, and older people primarily use Facebook as their social media fix, so getting updates about the goings-on in their communities or taking part in discussions about them is a natural fit between FarmVille updates and photos of grandkids.

Reddit works sort of all right for small and medium sized cities, but for most of small town rural America, people just use Facebook, not reddit, so it would likely be extremely hard to move people en masse out of Facebook for community organization. The network effects are simply far too massive to overcome no matter how good the other social network could be made.

Chicken and egg problem for people who work with chickens and eggs.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#63

Post by falcor84 »

gdb wrote: post #55
garmaine wrote: post #50

Earlier quoted context omitted.

I’m very sorry to see someone who obviously cares so much to be defending this code. This does not follow best practices, and using complexity of the underlying algorithm is just an excuse. Complex code can be beautiful and well documented. Writing a complex method in 174 lines is not elegant nor beautiful. Writing a well documented file that can take an engineer in a different specialty and bring them up to speed in…

We also have code like that. For example, that's the explicit goal of the Spinning Up repo: https://github.com/openai/spinningup/blob/master/spinup/algo... In practice, it's much harder to use that code, and we tend not to consume code like that internally. There's a real tradeoff!

ddpg() takes 17 parameters and is over 200 lines long. I'm very far from being a domain expert, but having worked in other complex domains, I'm pretty confident this can be redesigned such that it's both more maintainable and more pleasant to use.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#64

Post by m463 »

490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

Mirrors my thoughts regarding all math textbooks and published papers.

I remember reading a famous scientist (newton maybe) published a really accessible book on a subject, which was read by lots of lay persons and opened him up to lots of unwanted public attention.

So publishing in a more inscrutable way might be a way of assuring peer-to-peer communication.

Either that, or it's a labor of love where cleaning things up would detract from the forward momentum.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#65

Post by jachiam »

high_derivative wrote: post #58
gdb wrote: post #48

Earlier quoted context omitted.

(I work at OpenAI. Before that, I worked at Stripe. I've spent most of my software career thinking about how to build effective engineering cultures.) I think this code is actually well-written and maintainable. This is proven in practice because we've adopted it many places in OpenAI, and I've personally found it very easy to adapt to other use-cases (certainly much more so than the from-scratch Transformer implemen…

Ok, since you took the time to respond, I just want to be constructive as well: So I don't have a big problem with some of the function definitions which can be compact, as the other comment points out. The reason I don't like this code is that it does not comment anything on the critical bits. I don't necessarily care about whether you call the input to your matmul 'x' or 'tensor' or 'input' (although consistency is…

Hello! Spinning Up author here.

Very reasonable point that it is not clearly explained why you need to store logp_pi in the buffer. But the reason is that it would require additional code complexity to calculate it on the fly later. The likelihood ratio requires the denominator to be on the _old_ policy, so if you wanted to compute it on the fly, you would need to have a second policy in the computation graph to preserve the old policy while you change the current policy. You could not simply do a stop_gradient on the current policy and get the same results.

My personal feeling is that tutorial-style explanations like this don't fit nicely into code comment flow. As a result, most tutorial-style descriptions went into material on the Spinning Up website rather than into the code. It isn't 100% comprehensive, certainly, but RL has an enormous surface area (there are tons and tons of little details that teaching material could dive into) and I feel pretty good about what we were able to cover. :)

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#66

Post by dragonwriter »

endtime wrote: post #21
buboard wrote: post #13

Earlier quoted context omitted.

I bet anyone who bothers to read them twice realizes it's nonsensical

Anti-Foos already think that real primary Foo sources are nonsense (and they aren't the target audience anyway), and confirmation bias will take care of the pro-Foos.

If it's not moving opinion, it's not effective propaganda. Something that people will agree with only to the extent it restates what they already believe isn't working as propaganda.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#67

Post by high_derivative »

jachiam wrote: post #65
high_derivative wrote: post #58

Earlier quoted context omitted.

Ok, since you took the time to respond, I just want to be constructive as well: So I don't have a big problem with some of the function definitions which can be compact, as the other comment points out. The reason I don't like this code is that it does not comment anything on the critical bits. I don't necessarily care about whether you call the input to your matmul 'x' or 'tensor' or 'input' (although consistency is…

Hello! Spinning Up author here. Very reasonable point that it is not clearly explained why you need to store logp_pi in the buffer. But the reason is that it would require additional code complexity to calculate it on the fly later. The likelihood ratio requires the denominator to be on the _old_ policy, so if you wanted to compute it on the fly, you would need to have a second policy in the computation graph to pres…

Thank you for responding. Well, my point is that in particular the gradient on the likelihood ratio is what trips people up. They ask questions like 'why is this ratio not always 1' or similar. This is why I would say explaining what is going where here is critical, i.e. that we save the prior logp_pi (even though we could recompute it) to treat it as a constant value when computing the ratio/the gradient. That would be, from my perspective, the key pedagogical moment of a PPO tutorial. However his is purely subjective and I agree that one can feel differently about where to put explanations.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#70

Post by jchw »

josecyc wrote: post #68
hint23 wrote: post #33

You can try it at: http://textsynth.org

Thanks! any plans to make it faster? Like talk-to-transformer? Seems much more slower

Looks like it was created by Fabrice Bellard of tcc, ffmpeg, qemu, etc. fame. Does he read Hacker News comments?
Post reply on HN