Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Demo of an OpenAI language model applied to code generation [video]

twitter.com

Post reply on HN
Print view View original

151–158 of 158 posts

Re: Demo of an OpenAI language model applied to code generation [video]

#151

Post by gradys »

I worked on project very much like this last summer, a transformer language model applied to code completion.

You'd be surprised how easy it is to get a model that performs as well as what you see in the video. And it's even easier now that people have built great libraries for fine-tuning generative language models.

I encourage you to try it yourself! There are many interesting extensions for people to explore:

- Use bi-directional context (vanilla GPT-2 only sees backward context)

- Integrate with semantic analysis tools.

- Experiment with different context representations. You condition the model on an arbitrary sequence of N tokens. It's not necessarily the case that you should spend that whole budget on the N tokens that came immediately before. What about including the imports at the top of the file? What about the docstrings for functions that were just used? What about the filepath of the current file?

Don't look at something like this as though watching your job be automated away. Look at it as a tool that you can master and use to move up the stack.

Re: Demo of an OpenAI language model applied to code generation [video]

#152

Post by parksy »

BaronSamedi wrote: post #139
parksy wrote: post #133

I have thought about this before but I can see that logical errors are introduced which must be manually tested and reviewed anyway, so what if a more reliable approach could be achieved by training these data sets on test cases alongside passing code? This way developers just write unit tests or functional tests, and the AI generates code and retrains itself until the code passes for all tests. This could happen sil…

Instead of Test Driven Development, Test Only Development? I like that idea. This reminds me of an article I read a while ago on co-evolutionary training in genetic programming: one algorithm evolving to do something, with another evolving to break it.

Yeah that's a good way of putting it. Also has a catchy name, "TOD".

Ultimately as well we don't care what the code looks like, if it passes all tests then it "works". You probably don't even need to generate the code in a high level language, if people aren't ever going to really read it.

You'd probably need tests designed to ensure the code is executes quickly enough and automatically generate edge case test data so you don't end up with a blog where you can only post articles with the titles in the exact test data heh.

The future seems interesting for us developer types anyway. If a product designer could express their requirements in plain language developers would only really need to be around for cases where the models failed and more training data was needed to improve them.

Re: Demo of an OpenAI language model applied to code generation [video]

#153

Post by arielroth »

gradys wrote: post #151

I worked on project very much like this last summer, a transformer language model applied to code completion. You'd be surprised how easy it is to get a model that performs as well as what you see in the video. And it's even easier now that people have built great libraries for fine-tuning generative language models. I encourage you to try it yourself! There are many interesting extensions for people to explore: - Us…

Did you explore all of these things? What were your results?

Re: Demo of an OpenAI language model applied to code generation [video]

#154

Post by sguthery »

bauerm97 wrote: post #120
chii wrote: post #118

Earlier quoted context omitted.

> Similar to being a high level engineer that designs/defines/codes the broad strokes of something and then lets the lower level minions handle details and then the impl. turns out to have a bunch of details wrong that you didn't catch initially. And you wonder why there's so many bugs in software these days! I think the AI model is helpful, but the specification being ambiguous or under-specified is the problem, and…

I'm sure in the early days of compilers (I wasn't around back then, so I'm just assuming) they could also be fairly unreliable. Maybe they translated something in a completely idiotic way, lots of bugs, etc... But over time they improved and improved to the point that 99% of programmers never worry about anything except high level abstractions. This could be signalling the beginning of another such paradigm shift to…

Would that that were the case (compilers getting better). See http://embed.cs.utah.edu/csmith/ C compilers for the PDP-11 were pretty good if only because they were both simple, single threaded, and C was essentially a verbose version of the PDP-11 instruction set. Languages got more abstract and instruction sets (including their models of execution) got more complex. Optimizing the compilation of an abstraction with a faulty understanding of either the abstraction or the instruction set (or both) begets bugs you can't see as reported in the reference above. OB: I personally would like to see these folks point their AI code machine at netlib including the Collected Algorithms of the ACM. Generating numerical methods code is, in my experience, not the same as generating much of what is found on Github.

Re: Demo of an OpenAI language model applied to code generation [video]

#156

Post by OOPMan »

DJHenk wrote: post #134
YeGoblynQueenne wrote: post #84

So that's basically program synthesis from natural language (ish) specifications (i.e. the comments). I can see this being a useful tool [1]. However, I don't expect any ability for innovation. At best this is like having an exceptionally smart autocomplete function that can look up code snippets on SO for you (provided those code snippets are no longer than one line). That's not to say that it can't write new code,…

> In other words: everyone can relax. This will not take your job. Or mine Of course not. This technology converts writing code into bug hunting in pre-written code. Finding bugs in code that you did not write is way harder than writing the code yourself. So if anything, this makes programming harder, not easier, and we will need more programmers, not less.

Oh dear.

And then the model trains itself on the buggy code written and poorly debugged by these extra coders and then so on and so forth.

Codepocalypse.

Kill it with fire!

Re: Demo of an OpenAI language model applied to code generation [video]

#158

Post by imranq »

Where this would be most useful is automated testing suites just by specifying what you are testing for. A product manager looking to test portions of a system that absolutely need to work can specify code comments and generate 1000s of tests this way.

This is a gamechanger for ensuring the reliability of software. Many more people can be involved in the software development process, and inject their domain knowledge into it.

Are there any plans to open source the model? I would love to play around with it.

Post reply on HN