Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Releases Largest GPT-2 Text Generation Model

openai.com

Post reply on HN
Print view View original

41–50 of 166 posts

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#41

Post by gerash »

high_derivative wrote: post #31
490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

My professional observation (as ml researcher at big tech): These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews test leetcode knowledge, and today lots of degrees are heavy on Python-scripted ML homework. The result is companies with billion dollar funding and world-changing goals having a lot of their code look like complete spaghetti. And this is the engineers who are…

I don't know why you excluded PhDs but their code aren't any better necessarily. Hiring straight from school doesn't necessarily mean bad S/W engineering skills. But generally the research code is either made hastily during exploration or by people not having enough Software Engineering background.

Many research scientist I've seen at large tech companies don't have good CS background and write code that's either unreadable, unscalable, unmaintainable, or buggy at times. They often have good ideas but don't do much software design.

Having been both at infrastructure teams and research teams, there are certain individuals in research orgs joining straight from school who think they are responsible with coming with a new and sexy thing and other engineers are responsible to run in production. It's like computing eigenvectors in Matlab or Python over a toy dataset and thinking you've done the bulk of the work of computing PageRank in production and should receive all the credit for a full search engine.

That attitude is a red flag to me.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#42

Post by steve_musk »

jsinai wrote: post #36
fermenflo wrote: post #30

Earlier quoted context omitted.

I agree, a lot of the code could be improved. But some of what you mentioned is fairly standard. Like "Gaussian Error Linear Units being GELU, w/b for weights/biases, etc...

Not sure how standard that is ...

It’s very standard ML abbreviations.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#44

Post by high_derivative »

gerash wrote: post #41
high_derivative wrote: post #31

Earlier quoted context omitted.

My professional observation (as ml researcher at big tech): These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews test leetcode knowledge, and today lots of degrees are heavy on Python-scripted ML homework. The result is companies with billion dollar funding and world-changing goals having a lot of their code look like complete spaghetti. And this is the engineers who are…

I don't know why you excluded PhDs but their code aren't any better necessarily. Hiring straight from school doesn't necessarily mean bad S/W engineering skills. But generally the research code is either made hastily during exploration or by people not having enough Software Engineering background. Many research scientist I've seen at large tech companies don't have good CS background and write code that's either unr…

I don't see how what you are saying is contradicting me. The point is that the scientists generally don't view it as their job, whereas the ml engineers/research engineers typically have it as their job to do the software architecture/engineering side.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#45

Post by vitorbaptistaa »

hint23 wrote: post #33

You can try it at: http://textsynth.org

Pretty impressive! This is what I got (my initial text is in italic):

One day I woke up suddenly after a bad dream. I was sitting on my bed alone. As soon as I turned over I found myself lying down in a strange bed. It had no bed, no mattress and no cover for me. I was just lying on that thick carpet. I looked around to discover nothing but a white bed. There was no window, no furniture. No door. The only clue was a single white mirror that stood beside my bed. And there was a single candle lying on my bedside table. I was not worried or scared. I was fascinated.

I could see the floor.

I could see the walls.

And I could see everything that was around me.

I could hear the wind.

I could see everything from the windows, from the door that was always slightly open. I could even see the light.

I was in a dream world and I knew that I was in a dream. But what I wanted to know was what was happening in this world.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#46

Post by codingslave »

high_derivative wrote: post #31
490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

My professional observation (as ml researcher at big tech): These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews test leetcode knowledge, and today lots of degrees are heavy on Python-scripted ML homework. The result is companies with billion dollar funding and world-changing goals having a lot of their code look like complete spaghetti. And this is the engineers who are…

Yeah I'm waiting for the backlash when companies realize this

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#47

Post by vstuart »

hint23 wrote: post #33

You can try it at: http://textsynth.org

This is pretty good. I seeded GPT-2 (1.5B) with the first TWO sentences from this biomedical passage [source: https://www.ncbi.nlm.nih.gov/pubmed/19254780 ; I show the remaining original sentences, for comparison to GPT-2].

Yeast cells lacking the mitochondrial NADH kinase encoded by POS5 display increased sensitivity to hydrogen peroxide, a slow-growth phenotype, reduced mitochondrial function and increased levels of mitochondrial protein oxidation and mtDNA mutations. Here we examined gene expression in pos5Delta cells, comparing these data to those from cells containing deletions of superoxide dismutase-encoding genes SOD1 or SOD2.

Surprisingly, stress-response genes were down-regulated in pos5Delta, sod1Delta and sod2Delta cells, implying that cells infer stress levels from mitochondrial activity rather than sensing reactive oxygen species directly. Additionally, pos5Delta, but not sod1 or sod2, cells displayed an anaerobic expression profile, indicating a defect in oxygen sensing that is specific to pos5, and is not a general stress-response. Finally, the pos5Delta expression profile is quite similar to the hap1Delta expression profile previously reported, which may indicate a shared mechanism.

Here is the generated output (following those first two sentences; I needed to do this in Opera; Firefox did not work, possibly due to add-ons and settings).

The results show that POS5Delta cells exhibit decreased expression of two of the genes, SOD1 and SOD2, but increased expression of several other genes, including aldehyde dehydrogenase subunits, NADH dehydrogenase subunits 2 (ODU2), cytochrome c oxidase isoforms (OXO3), and cytochrome b oxidase/OXO1, and increased expression of aldehyde dehydrogenase subunits 1 (ODU1) and 1a2. The results also show that loss of two genes, SOA and SOA1, and two genes, SOA and SOA2, leads to decreased expression of aldehyde dehydrogenase subunits 1, 1a2 and 1a3 and increased expression of ODO1 and ODO1a1. The results show that loss of two genes, ODO2 and ODO1b1, leads to decreased expression of ODO2 and increased expression of O

The output is grammatically / syntactically correct (but o/w nonsensical: e.g. there are no genes ODU1, ODO2 ...).

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#48

Post by gdb »

high_derivative wrote: post #31
490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

My professional observation (as ml researcher at big tech): These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews test leetcode knowledge, and today lots of degrees are heavy on Python-scripted ML homework. The result is companies with billion dollar funding and world-changing goals having a lot of their code look like complete spaghetti. And this is the engineers who are…

(I work at OpenAI. Before that, I worked at Stripe. I've spent most of my software career thinking about how to build effective engineering cultures.)

I think this code is actually well-written and maintainable. This is proven in practice because we've adopted it many places in OpenAI, and I've personally found it very easy to adapt to other use-cases (certainly much more so than the from-scratch Transformer implementations I've written!).

As https://news.ycombinator.com/item?id=21456605 points out, the complexity of the code arises from the complexity of the underlying algorithm. Complexity due to software engineering concerns, like Tensorflow scopes, are elegantly handled. [edited for clarity:] Writing a Transformer in 174 lines of code requires a lot of deep thinking about the right underlying abstractions.

> but essentially as soon as you enter the 'ml engineer/research engineer/research scientist' layer, it's doomed.

We actually don't do this! Our only official technical title is "member of technical staff". (People sometimes choose to self-identify as an engineer or researcher, so you might see that on LinkedIn, but we don't have a distinction internally.) Everyone is responsible for their own code, and people care quite a bit about writing code that others can build on.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#49

Post by zitterbewegung »

sillysaurusx wrote: post #18
epoch_100 wrote: post #2

Paper: https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf Code: https://github.com/openai/gpt-2

If anyone wants to fine-tune the 1.5B model, I ported the gpt-2 code to TPUs. You can fine-tune it in Colab. Snapshots are 5.8GB. notebook: https://twitter.com/theshawwn/status/1191800180192010246 code: https://github.com/shawwn/gpt-2 It's a fork of nshepperd's gpt-2 codebase ( https://github.com/nshepperd/gpt-2 ) which lets you fine-tune 117M and 345M on GPUs. For a tutorial on how to fine-tune GPT-2, see http://gwe…

Cool this is awesome !

I’m going to try to retrain this with a twitter dataset called sentiment140 ( I have already processed it with gpt2 345M).

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#50

Post by garmaine »

gdb wrote: post #48
high_derivative wrote: post #31

Earlier quoted context omitted.

My professional observation (as ml researcher at big tech): These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews test leetcode knowledge, and today lots of degrees are heavy on Python-scripted ML homework. The result is companies with billion dollar funding and world-changing goals having a lot of their code look like complete spaghetti. And this is the engineers who are…

(I work at OpenAI. Before that, I worked at Stripe. I've spent most of my software career thinking about how to build effective engineering cultures.) I think this code is actually well-written and maintainable. This is proven in practice because we've adopted it many places in OpenAI, and I've personally found it very easy to adapt to other use-cases (certainly much more so than the from-scratch Transformer implemen…

I’m very sorry to see someone who obviously cares so much to be defending this code. This does not follow best practices, and using complexity of the underlying algorithm is just an excuse. Complex code can be beautiful and well documented.

Writing a complex method in 174 lines is not elegant nor beautiful. Writing a well documented file that can take an engineer in a different specialty and bring them up to speed in 1,000 lines is.

Post reply on HN