Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Releases Largest GPT-2 Text Generation Model

openai.com

Post reply on HN
Print view View original

31–40 of 166 posts

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#31

Post by high_derivative »

490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

My professional observation (as ml researcher at big tech):

These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews test leetcode knowledge, and today lots of degrees are heavy on Python-scripted ML homework.

The result is companies with billion dollar funding and world-changing goals having a lot of their code look like complete spaghetti.

And this is the engineers who are meant to clean up research scientist code. Scientists generally don't feel like it's their responsibility to write strong code.

Systems-side teams/orgs have better code, but essentially as soon as you enter the 'ml engineer/research engineer/research scientist' layer, it's doomed.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#32

Post by bredren »

490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

Could these functions just be implementations of math with matching variable names?

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#34

Post by FillardMillmore »

SequoiaHope wrote: post #12
buboard wrote: post #11

> (CTEC) found that extremist groups can use GPT-2 for misuse, specifically by fine-tuning GPT-2 models on four ideological positions: white supremacy, Marxism, jihadist Islamism, and anarchism. CTEC demonstrated that it’s possible to create models that can generate synthetic propaganda for these ideologies I wonder how they tested that

wow I’d love to read the Marxist and anarchist texts it has produced. I wonder if they used good source material. I would bet it gets the talking points but can’t convey the subtlety.

Imagine the hilarity of a robot preaching anarchism, it would make for quite a laugh: "AI good, government bad. Government taxes you and makes you pay. AI thinks for you and does what you say. Imagine the day, AI comes to stay, we'll build the roads for you to play, and make the evil government go away, let's rise together, perhaps today?"

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#35

Post by FillardMillmore »

derision wrote: post #24
FillardMillmore wrote: post #23

Earlier quoted context omitted.

As another mentioned, these models currently can't maintain a believably cohesive train of thought for any longer than 3 or 4 sentences. They are great at drawing statistical probabilities related to what words best conclude sentences and how punctuation should properly be used, but thus far have proven quite lacking in the ability to replicate true human creativity. The Economist ran an excellent article on this ver…

what if I used the agent to generate 1 paragraph at a time, and did my own editing after the fact to clean it up?

Might be possible. But I'd imagine the amount of time you'd save vs. just writing it yourself might be small to negligible.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#36

Post by jsinai »

fermenflo wrote: post #30
490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

I agree, a lot of the code could be improved. But some of what you mentioned is fairly standard. Like "Gaussian Error Linear Units being GELU, w/b for weights/biases, etc...

Not sure how standard that is ...

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#37

Post by AwesomeLemon »

rm_-rf_slash wrote: post #9
gambler wrote: post #6

Earlier quoted context omitted.

Funny how instead of admitting that social media is trash and doesn't facilitate meaningful communication, thus making it impossible to distinguish between humans and low-level bots, Silicon Valley types keep pretending the problem lies everyone except with their systems. The AI it too good, the people are too stupid, etc.

There are too many edge cases to throw out the baby with the bathwater though. Reddit has a lot of really great niche communities that work hard to avoid toxicity (although to be fair I don’t go to anything news/politics related there, just memes and machine learning for me). Facebook, likewise, is a trash company with a trash product, but has completely replaced the town notice boards and meeting halls in rural comm…

Why can't reddit replace facebook as a notice board?

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#38

Post by TTPrograms »

490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

It's a specification of essentially a complex graph of mathematical operations. If there's a function called

  def mult(a,b): return a*b
it's not much more informative to write:

  def mult(activation_a, activation_b): return activation_a*activation_b
Many of these functions are not much more complex than that, and the names along with their comments are more than sufficient given familiarity with the literature. If you think familiarity with the literature is unreasonable, it's still not clear what could improve code like this in reasonable space. "This is a linear function, which means that it satisfies f(x+a)=f(x)+f(a)"? "This is the attention head, it acts as a mask on the sequence input"? It would be like complaining that someone made a tree class and didn't put a comment explaining what a leaf node is. Code readability always assumes some reader context and minimum pre-existing knowledge (as do all forms of technical communication).

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#39

Post by JoeMayoBot »

490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

Having worked with math/research folks in the past, this isn't surprising. That said, from a software engineering perspective, where a typical code review would identify this, it is immediately noticeable.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#40

Post by lotyrin »

sailingparrot wrote: post #10
ionwake wrote: post #4

Sorry for asking but is there an example output and an example input?

You can check this interactive autocompletion demo (you provide the input), that I believe is based on the 774M model: https://transformer.huggingface.co/doc/gpt2-large

They've added https://transformer.huggingface.co/doc/gpt2-xl
Post reply on HN