Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Demo of an OpenAI language model applied to code generation [video]

twitter.com

Post reply on HN
Print view View original

81–90 of 158 posts

Re: Demo of an OpenAI language model applied to code generation [video]

#81

Post by IdiocyInAction »

bglazer wrote: post #79
IdiocyInAction wrote: post #46

How does this do compared to other models? Is this a totally cutting edge result? On the surface, it seems quite impressive, but sans an environment to try it out with, I cannot be entirely sure. Still, this does make me question whether I chose a safe career, haha. The thing is, I'd really need to see a live demo to see how good this is. Making mistakes is actually kind of a big issue; as most people know, debugging…

I've read a fair number of papers on neural program synthesis lately. To me, these seemed to be obviously cherry picked examples, so you can't really evaluate the whole system based on them. However, this is fairly impressive for a couple reasons. First, the system constructs programs from natural language descriptions, rather than examples of input-output pairs or a formal specification, which are the most common se…

> Fine tuning a vanilla language model on a giant corpus of code feels like a dead end for the field, long-term. It seems obvious to me that humans are doing something more than just statistical pattern recognition and generation when we write and reason about code.

Yeah, this is the main reason why I would be interested in more examples. But, if this thing was trained on all of GitHub, I could imagine that it come up with decent-looking code for a lot of examples; a beefy, smarter Google with some rudimentary contextual understanding, if you will. Still, the presence of any mistakes is a no-go and I'd be really interested how it reacts to more realistic, specific requirements.

But yeah, I'd figure a model for code generation would have to have some kind of knowledge of syntax and semantics, rather than doing pure statistical pattern matching, to be of any real use. It would not only have to generate, but also to debug its code (I wonder whether you could do that purely with statistical pattern recognition). I might be wrong, of course, but I would be surprised if that is enough to write complex code.

Re: Demo of an OpenAI language model applied to code generation [video]

#82

Post by MauranKilom »

bglazer wrote: post #79
IdiocyInAction wrote: post #46

How does this do compared to other models? Is this a totally cutting edge result? On the surface, it seems quite impressive, but sans an environment to try it out with, I cannot be entirely sure. Still, this does make me question whether I chose a safe career, haha. The thing is, I'd really need to see a live demo to see how good this is. Making mistakes is actually kind of a big issue; as most people know, debugging…

I've read a fair number of papers on neural program synthesis lately. To me, these seemed to be obviously cherry picked examples, so you can't really evaluate the whole system based on them. However, this is fairly impressive for a couple reasons. First, the system constructs programs from natural language descriptions, rather than examples of input-output pairs or a formal specification, which are the most common se…

I am also hedging my hopes of this working on "more realistic" scenarios. It does produce code that looks natural to us, but i expect it to show clear "seams" where its understanding of something isn't deep enough.

But maybe this is just a question of how much compute (and network size/"depth") you invest. On a certain level we're also just some recurrent LSTM :)

Re: Demo of an OpenAI language model applied to code generation [video]

#83

Post by joshuak »

netsec_burn wrote: post #27
grensley wrote: post #20

Wow, this has the ability to be a total gamechanger. You have to be really observant about the bugs though, I would have totally missed the one with the price discount without executing it.

By lowering the barrier of entry of programming further, I wonder if we'll see more bugs (like the price discount) as a result of this?

I notice that the bug was in the user's failure to communicate the intent of the scalar. Presumably with regular use users would learn to be more clear and/or anticipate the likely fixes to ambiguous labels.

Also, since it would be used to build tests as well, I'd expect such misunderstandings to be pretty obvious. I would be willing to bet you'd see a net reduction in bugs, and a substantial reduction in typo related bugs.

But if you mean by lowered barrier of entry you mean the population of programmers would be less competent, yes bugs in the design might increase, however being able to more quickly get to the point of evaluating a design is a great way to learn better design.

Re: Demo of an OpenAI language model applied to code generation [video]

#84

Post by YeGoblynQueenne »

So that's basically program synthesis from natural language (ish) specifications (i.e. the comments).

I can see this being a useful tool [1]. However, I don't expect any ability for innovation. At best this is like having an exceptionally smart autocomplete function that can look up code snippets on SO for you (provided those code snippets are no longer than one line).

That's not to say that it can't write new code, that nobody has quite written before in the same way. But in order for a tool like this to be useful it must stick as close as possible to what is expected- or it will slow development down rather than helping it. Which means it can only do what has already been done before.

For instance- don't expect this to come up with a new sorting algorithm, out of the blue, or to be able to write good code to solve a certain problem when the majority of code solving that problem on github happens to be pretty bad.

In other words: everyone can relax. This will not take your job. Or mine.

____________

[1] I apologise to the people who know me and who will now be falling off their chairs. OK down there?

Re: Demo of an OpenAI language model applied to code generation [video]

#85

Post by MauranKilom »

IdiocyInAction wrote: post #81
bglazer wrote: post #79

Earlier quoted context omitted.

I've read a fair number of papers on neural program synthesis lately. To me, these seemed to be obviously cherry picked examples, so you can't really evaluate the whole system based on them. However, this is fairly impressive for a couple reasons. First, the system constructs programs from natural language descriptions, rather than examples of input-output pairs or a formal specification, which are the most common se…

> Fine tuning a vanilla language model on a giant corpus of code feels like a dead end for the field, long-term. It seems obvious to me that humans are doing something more than just statistical pattern recognition and generation when we write and reason about code. Yeah, this is the main reason why I would be interested in more examples. But, if this thing was trained on all of GitHub, I could imagine that it come u…

Five years ago we were already here: https://karpathy.github.io/2015/05/21/rnn-effectiveness/

Calling the field "statistical pattern matching" might be underselling it a bit, even if technically accurate on some level. I mean, syntax/semantics are clearly not the problem, those are the easiest to learn (see the paper above). If anything, I'm scared of it writing syntactically correct nonsense (or even worse, subtly-flawed-but-correct-looking code).

Re: Demo of an OpenAI language model applied to code generation [video]

#86

Post by YeGoblynQueenne »

bglazer wrote: post #79
IdiocyInAction wrote: post #46

How does this do compared to other models? Is this a totally cutting edge result? On the surface, it seems quite impressive, but sans an environment to try it out with, I cannot be entirely sure. Still, this does make me question whether I chose a safe career, haha. The thing is, I'd really need to see a live demo to see how good this is. Making mistakes is actually kind of a big issue; as most people know, debugging…

I've read a fair number of papers on neural program synthesis lately. To me, these seemed to be obviously cherry picked examples, so you can't really evaluate the whole system based on them. However, this is fairly impressive for a couple reasons. First, the system constructs programs from natural language descriptions, rather than examples of input-output pairs or a formal specification, which are the most common se…

>> Edit: Also, everyone should go try the FlashFill feature in Microsoft excel. As far as I know, it's the only example of program synthesis shipped in a consumer facing production system, and it works shockingly well.

And it's not a giant language model trained on a gigantic dataset. Rather, if memory serves, it's a buch of task-specific DSLs and rules, all hand-written from scratch.

Re: Demo of an OpenAI language model applied to code generation [video]

#87

Post by neatze »

bufferoverflow wrote: post #72
netsec_burn wrote: post #27

Earlier quoted context omitted.

By lowering the barrier of entry of programming further, I wonder if we'll see more bugs (like the price discount) as a result of this?

You still need a programmer to find the bugs. I think it's actually harder to spot and fix a bug, than to write a simple method that involves one.

Programming skills are directly correlated to programmer ability to debug, I will go as far stating; programming is not about writing code, but in ability to find bugs and figure out how to resolve them.

Re: Demo of an OpenAI language model applied to code generation [video]

#89

Post by joshuak »

YeGoblynQueenne wrote: post #84

So that's basically program synthesis from natural language (ish) specifications (i.e. the comments). I can see this being a useful tool [1]. However, I don't expect any ability for innovation. At best this is like having an exceptionally smart autocomplete function that can look up code snippets on SO for you (provided those code snippets are no longer than one line). That's not to say that it can't write new code,…

I agree completely with your expectation of the abilities of such a system.

However, I think very little programming labor is employed in the construction of new algorithms or even most business logic, even a casual stroll through github reveals a staggering amount of reimplementation.

I think the promise here is the ability to code in a more conceptual way with less fiddling with the finicky details.

Re: Demo of an OpenAI language model applied to code generation [video]

#90

Post by pharke »

YeGoblynQueenne wrote: post #84

So that's basically program synthesis from natural language (ish) specifications (i.e. the comments). I can see this being a useful tool [1]. However, I don't expect any ability for innovation. At best this is like having an exceptionally smart autocomplete function that can look up code snippets on SO for you (provided those code snippets are no longer than one line). That's not to say that it can't write new code,…

Yeah I'm thinking it would be more useful to have a really well indexed library of functions accessible by search.
Post reply on HN