Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Demo of an OpenAI language model applied to code generation [video]

twitter.com

Post reply on HN
Print view View original

111–120 of 158 posts

Re: Demo of an OpenAI language model applied to code generation [video]

#111

Post by bo1024 »

IdiocyInAction wrote: post #46

How does this do compared to other models? Is this a totally cutting edge result? On the surface, it seems quite impressive, but sans an environment to try it out with, I cannot be entirely sure. Still, this does make me question whether I chose a safe career, haha. The thing is, I'd really need to see a live demo to see how good this is. Making mistakes is actually kind of a big issue; as most people know, debugging…

Ha. You hit the nail on the head. There is no rigorous way to measure AI-generated anything. (to my knowledge) So every demo is "ooh look at this" and actual performance is not scientifically evaluated, because we don't know how. This includes images, text, etc.

Re: Demo of an OpenAI language model applied to code generation [video]

#114

Post by justinmeiners »

raghavgoyal14 wrote: post #107
um_ya wrote: post #106

Earlier quoted context omitted.

It would be nice to have an AI that could write unit tests, or look over your code and understand and explain where you might have bugs.

> look over your code and understand and explain where you might have bugs. This would certainly be interesting. I'm not aware of active research going on in this area (any pointers would be helpful!). This would require an agent to have thorough understanding of the logic you're trying to implement, and locate the piece of code where it silently fails. For this you'd again need a training dataset where the input is…

> I'm not aware of active research going on in this area (any pointers would be helpful!).

Look at the static analysis tool in clang. Xcode uses it well.

Re: Demo of an OpenAI language model applied to code generation [video]

#115

Post by nmfisher »

monkeydust wrote: post #62

As a product person wondering how much more productive this will make my engineers? In the surface looks impressive.

The samples are pretty impressive, and that's coming from someone who's intimately familiar with the internals of these these kind of models.

From bitter experience, though, I also know how unreliable these models can be. It's possible (indeed, likely) they generated 20 samples, threw away 19 that were garbage and just showed you the one that looked nice.

If - and that's a big if - it reliably generates good-quality code, then it would probably be a nice productivity boost (~25%, particularly when it comes to tests).

Based on what I've seen over the past few years, though, I'm skeptical.

Re: Demo of an OpenAI language model applied to code generation [video]

#116

Post by westurner »

YeGoblynQueenne wrote: post #84

So that's basically program synthesis from natural language (ish) specifications (i.e. the comments). I can see this being a useful tool [1]. However, I don't expect any ability for innovation. At best this is like having an exceptionally smart autocomplete function that can look up code snippets on SO for you (provided those code snippets are no longer than one line). That's not to say that it can't write new code,…

> At best this is like having an exceptionally smart autocomplete function that can look up code snippets on SO for you (provided those code snippets are no longer than one line).

Yeah, all it could do for you is autocomplete around what it thinks the specification might be at that point in time.

> But what if Andy gets another dinosaur, a mean one? -- Toy Story (1995)

Re: Demo of an OpenAI language model applied to code generation [video]

#118

Post by chii »

Swizec wrote: post #98
joshuak wrote: post #89

Earlier quoted context omitted.

I agree completely with your expectation of the abilities of such a system. However, I think very little programming labor is employed in the construction of new algorithms or even most business logic, even a casual stroll through github reveals a staggering amount of reimplementation. I think the promise here is the ability to code in a more conceptual way with less fiddling with the finicky details.

> I think the promise here is the ability to code in a more conceptual way with less fiddling with the finicky details. This is basically how product managers code. Or former engineers turned engineering managers. Or even team leads. Hell, maybe like an architect? You come up with a rough sketch, design the system, think through a couple edge cases, tell the computer what you need, and the computer figures out the de…

> Similar to being a high level engineer that designs/defines/codes the broad strokes of something and then lets the lower level minions handle details

and then the impl. turns out to have a bunch of details wrong that you didn't catch initially. And you wonder why there's so many bugs in software these days!

I think the AI model is helpful, but the specification being ambiguous or under-specified is the problem, and the effort to sort that out is hard. I'm not sure an AI can help in that aspect, and that's where most of the value of programming comes from.

Re: Demo of an OpenAI language model applied to code generation [video]

#120

Post by bauerm97 »

chii wrote: post #118
Swizec wrote: post #98

Earlier quoted context omitted.

> I think the promise here is the ability to code in a more conceptual way with less fiddling with the finicky details. This is basically how product managers code. Or former engineers turned engineering managers. Or even team leads. Hell, maybe like an architect? You come up with a rough sketch, design the system, think through a couple edge cases, tell the computer what you need, and the computer figures out the de…

> Similar to being a high level engineer that designs/defines/codes the broad strokes of something and then lets the lower level minions handle details and then the impl. turns out to have a bunch of details wrong that you didn't catch initially. And you wonder why there's so many bugs in software these days! I think the AI model is helpful, but the specification being ambiguous or under-specified is the problem, and…

I'm sure in the early days of compilers (I wasn't around back then, so I'm just assuming) they could also be fairly unreliable. Maybe they translated something in a completely idiotic way, lots of bugs, etc... But over time they improved and improved to the point that 99% of programmers never worry about anything except high level abstractions. This could be signalling the beginning of another such paradigm shift to a higher level abstraction
Post reply on HN