Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is phenomenal at Code

github.com

Post reply on HN
Print view View original

191–200 of 221 posts

Re: GPT-4 is phenomenal at Code

#191

Post by Jensson »

famouswaffles wrote: post #119
zamnos wrote: post #61

Is it? According to OpenAI's paper GPT-4 GPT-4 (no vision) GPT-3.5 Leetcode (easy) 31 / 41 31 / 41 12 / 41 Leetcode (medium) 21 / 80 21 / 80 8 / 80 Leetcode (hard) 3 / 45 3 / 45 0 / 45 https://cdn.openai.com/papers/gpt-4.pdf Table 1; page 5. So it's better than GPT-3.5, but still pretty pathetic at hard Leetcode problems. If your programming job is closer to leetcode easy problems you might be in trouble, but for the…

My dude, putting aside the fact that leetcode problem solving isn't factoring in the majority of programming use cases... you do realize this is better than a lot of professional programmers right ? It's truly amazing seeing the posts shift in real time.

AI will overperform at questions whose answers are in its training set. These leetcode questions has lots of answers posted online, any human with internet access can just google those and copy paste the solution and get a better score than these language models.

This effect is stronger the larger the model, so likely most of the improvements here is that the model has better memory of the solutions that are already out there.

Re: GPT-4 is phenomenal at Code

#192

Post by nottathrowaway3 »

1attice wrote: post #190
prcrstntr wrote: post #122

Earlier quoted context omitted.

But it will have zero credibility, so no.

It will absolutely have great credibility. Did you read any of the code samples in the GH repo linked by OP? Here's the algorithm to get you appropriately scared. - Measure the amount of time it took to go from GPT 2 to GPT 3 - Measure the amount of time it took to go from GPT 3 to GPT 4 - Measure the code quality improvement for the same spans. Go ahead. Pick a metric. You'll notice that, not only is GPT-4 really fs…

> And there is no ceiling yet in sight

We are approaching the time when we find out if the human brain uses language encoding as its core mechanism of thought, or if it has any tricks left up its damp, fleshy sleeve.

Re: GPT-4 is phenomenal at Code

#193

Post by dragonwriter »

1attice wrote: post #190
prcrstntr wrote: post #122

Earlier quoted context omitted.

But it will have zero credibility, so no.

It will absolutely have great credibility. Did you read any of the code samples in the GH repo linked by OP? Here's the algorithm to get you appropriately scared. - Measure the amount of time it took to go from GPT 2 to GPT 3 - Measure the amount of time it took to go from GPT 3 to GPT 4 - Measure the code quality improvement for the same spans. Go ahead. Pick a metric. You'll notice that, not only is GPT-4 really fs…

> What you are likely looking at is a classic hockey-stick exponential graph. And there is no ceiling yet in sight.

Its a pretty ironclad rule that all growth charts that look exponential are really closer to logistic and seem like they are going to go up more rapidly forever until they suddenly flatten out.

Re: GPT-4 is phenomenal at Code

#194

Post by 1attice »

dragonwriter wrote: post #193
1attice wrote: post #190

Earlier quoted context omitted.

It will absolutely have great credibility. Did you read any of the code samples in the GH repo linked by OP? Here's the algorithm to get you appropriately scared. - Measure the amount of time it took to go from GPT 2 to GPT 3 - Measure the amount of time it took to go from GPT 3 to GPT 4 - Measure the code quality improvement for the same spans. Go ahead. Pick a metric. You'll notice that, not only is GPT-4 really fs…

> What you are likely looking at is a classic hockey-stick exponential graph. And there is no ceiling yet in sight. Its a pretty ironclad rule that all growth charts that look exponential are really closer to logistic and seem like they are going to go up more rapidly forever until they suddenly flatten out.

Fine, but you missed the second sentence that you quoted. There is no ceiling in sight, so we have no way of knowing how high this curve is likely to go.

For now, it approximates exponential. That's bad.

Re: GPT-4 is phenomenal at Code

#195

Post by 1attice »

nottathrowaway3 wrote: post #192
1attice wrote: post #190

Earlier quoted context omitted.

It will absolutely have great credibility. Did you read any of the code samples in the GH repo linked by OP? Here's the algorithm to get you appropriately scared. - Measure the amount of time it took to go from GPT 2 to GPT 3 - Measure the amount of time it took to go from GPT 3 to GPT 4 - Measure the code quality improvement for the same spans. Go ahead. Pick a metric. You'll notice that, not only is GPT-4 really fs…

> And there is no ceiling yet in sight We are approaching the time when we find out if the human brain uses language encoding as its core mechanism of thought, or if it has any tricks left up its damp, fleshy sleeve.

This would certainly vindicate several fields, 20th century analytic phil among them, with its laser focus on linguistic analysis.

It's breathtaking for a philosopher to live through this era. I just wish I wasn't so frightened by the likely practical implications of handing out such a powerful tool-that-can-be-a-weapon (a TTCBAW) to anyone who can pull down a torrent.

Re: GPT-4 is phenomenal at Code

#196

Post by kamaal »

dougmwne wrote: post #77
nottathrowaway3 wrote: post #27

> is phenomenal at Code Can it - attend bullshit meetings - lick my manager's boot - tell them that their ideas are good but in the masochistic, self-deprecating way they like to hear it from me in Not yet? Darn it, guess my job is safe for now.

I think it can actually do all those things reasonably well. I’d like to start getting meeting summaries that tell me my action items based on a transcript. And it can generate infinite boot-licking butt kissing if you just ask it. Generate and click send to your boss!

>>I think it can actually do all those things reasonably well.

I don't you get it. Or even AI gets it.

Having pet slaves is a very different feeling. It is not about getting work done. It is about the feeling of power, control. To both create and destroy.

GPT won't be a problem for the same reason why programming itself never automated programmers out of jobs. The current state of programming is already good enough to put nearly all of programming people out of work. It hasn't because people like to rule over pyramid structures. That won't change with GPT.

Re: GPT-4 is phenomenal at Code

#197

Post by dwohnitmok »

famouswaffles wrote: post #162
LightMachine wrote: post #152

Earlier quoted context omitted.

Even if that was the case, it'd still be massively useful because it would allow code to be transposed between languages idiomatically. For example, certain things like game engines were only implemented in very messy C++ code. If GPT understands how these libraries work, could it recreate all these game engines as cleaned up, elegant Haskell code?

Bilingual LLMs are already excellent at translating human languages. I imagine we'll find that GPT can translate between existing programming languages very well. We could soon see a future where "Damn this useful paper/code is in x language. Have to wait for someone to port the code over to y language" is a thing of the past.

They are excellent. Not quite human level, but very very close. I was curious about Chinese-English translation capabilities of the latest crop of models and on more difficult texts a bilingual model like GLM-130B makes several errors per page while GPT-4 is down to probably just around one.

Interested to see how that plays out for programming languages.

Re: GPT-4 is phenomenal at Code

#198

Post by [deleted] »

1attice wrote: post #195
nottathrowaway3 wrote: post #192

Earlier quoted context omitted.

> And there is no ceiling yet in sight We are approaching the time when we find out if the human brain uses language encoding as its core mechanism of thought, or if it has any tricks left up its damp, fleshy sleeve.

This would certainly vindicate several fields, 20th century analytic phil among them, with its laser focus on linguistic analysis. It's breathtaking for a philosopher to live through this era. I just wish I wasn't so frightened by the likely practical implications of handing out such a powerful tool-that-can-be-a-weapon (a TTCBAW) to anyone who can pull down a torrent.

[deleted]

Re: GPT-4 is phenomenal at Code

#199

Post by [deleted] »

1attice wrote: post #195
nottathrowaway3 wrote: post #192

Earlier quoted context omitted.

> And there is no ceiling yet in sight We are approaching the time when we find out if the human brain uses language encoding as its core mechanism of thought, or if it has any tricks left up its damp, fleshy sleeve.

This would certainly vindicate several fields, 20th century analytic phil among them, with its laser focus on linguistic analysis. It's breathtaking for a philosopher to live through this era. I just wish I wasn't so frightened by the likely practical implications of handing out such a powerful tool-that-can-be-a-weapon (a TTCBAW) to anyone who can pull down a torrent.

[deleted]

Re: GPT-4 is phenomenal at Code

#200

Post by mock-possum »

bob1029 wrote: post #81

My current take is that it's actually kind of bad at code in a lot of very important ways, but it is so good at getting to ~95% that you don't even feel the missing pieces once you are in flow. Having skills in the target area are important, but some quick runtime testing usually closes the gap. The added confidence from being able to scream a horrible demand into the void and get back something nearly complete is be…

yeah that's where I'm at too - it's great that it gets you 95% there, genuinely, but unless you know how to recognize the missing 5%, it's not going to mean much. Broken code is broken code - a single wrong character can render the entire thing unusable.

What's fun is asking it for code solutions, and then asking it to debug its own functions - if it's 95% effective at debugging code that is 95% correct, where does that leave you?

Post reply on HN