Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is phenomenal at Code

github.com

Post reply on HN
Print view View original

131–140 of 221 posts

Re: GPT-4 is phenomenal at Code

#131

Post by jabloczko »

What I've noted is that GPT is really good at things that have really good documentation.

The example of kube-tf in this repo is a perfect example. The Kubernetes documentation and all of Hashicorp's documentation is excellent. GPT will have infinite examples of good code to stitch together code for the task in this example.

Now I've been running a private cloud at work on the OpenNebula platform, which has documentation that is definitely lacking. I tried to ask GPT to write some basic code in Python such as "Give me a list of VMs from the OpenNebula API in a powered off state that have a start time older than 30 days."

What I noted was that it would spit out code that looked correct on the surface, but would not run. It would take a decent amount of me modifying the code until I got my desired result. Since there was no documentation, I was just reading through the OpenNebula packages themselves to understand what to do.

The nice thing, though, is that it was a great starting point. Much in the same way I might take a code snippet from StackOverflow and modify it to suit my own needs.

I listened to a great podcast titled "The Trouble with AI" on Making Sense with Sam Harris. One of the key takeaways I grabbed from it was that GPT is an LLM not an AI. What it is very good at is predicting the next correct character or word in a sequence based off of other examples. But it does not actually fundamentally understand what it is outputting.

In order to demonstrate, open up a session with ChatGPT really quickly and ask it a single digit multiplication question. Such as "What is 3 multiplied by 4?" and you will see a correct answer.

Next, ask it something a bit larger, like "What is 12366 multiplied by 981632?" and you will get an incorrect answer but one that looks pretty close to correct. Validate with a calculator yourself.

The reason being, as an LLM it doesn't actually understand multiplication. Instead it has just seen 3 multiplied by 4 countless times in the data it ingested when it was being "trained", but never has seen larger number examples of multiplication. Not that it knows multiplication in the first place.

GPT is fantastic, but as of right now it needs to be used as a starting point towards knowledge or something concrete. I wouldn't trust it as an authoritative source on anything quite yet. It is fantastic for generating a bit of code and then allowing the developer to tweak that code until it actually works.

Re: GPT-4 is phenomenal at Code

#132

Post by zamnos »

capableweb wrote: post #117
zamnos wrote: post #61

Is it? According to OpenAI's paper GPT-4 GPT-4 (no vision) GPT-3.5 Leetcode (easy) 31 / 41 31 / 41 12 / 41 Leetcode (medium) 21 / 80 21 / 80 8 / 80 Leetcode (hard) 3 / 45 3 / 45 0 / 45 https://cdn.openai.com/papers/gpt-4.pdf Table 1; page 5. So it's better than GPT-3.5, but still pretty pathetic at hard Leetcode problems. If your programming job is closer to leetcode easy problems you might be in trouble, but for the…

What is "Leetcode (hard)" specifically? Searching for it brought me to a website ( https://leetcode.com/ ) but I'm assuming that is referring to some special list of problems that are meant to be harder than others, but I cannot find that.

Pick from this list: https://leetcode.com/problemset/all/?difficulty=HARD

Re: GPT-4 is phenomenal at Code

#133

Post by [deleted] »

amelius wrote: post #84
LightMachine wrote: post #9

It managed to write, in one shot, a working λ-calculus parser, using a very specific programming style I asked it in JavaScript, and then translated it all to Python, including sarcastic, rhyming GLaDOS comments. https://twitter.com/VictorTaelin/status/1635726202231988225 It also seems to be extremely competent at writing Agda types and proofs. We need some tool that highly integrates it with entire codebases, allowi…

I don't think this is a good test. Every CS graduate student has to write a lambda-calculus parser. There must be thousands of implementations on the web. It really is not strange that GPT-4 can reproduce this.

[deleted]

Re: GPT-4 is phenomenal at Code

#134

Post by sualehasif »

impulser_ wrote: post #114

Unless these models are updated frequently it not going to be better than a human. ChatGPT doesn't even know Go has generics yet. Now imagine is suggesting you a inferior way of doing something because it doesn't know that a bug as been fixed or a better feature/version has been implemented. It often recommends made up libraries, or libraries that don't exists anymore. It even recommended using a Go wiki page as an i…

GPT-4 certainly is aware of go generics. Do you have an example you would like to test?

Re: GPT-4 is phenomenal at Code

#135

Post by dougmwne »

nottathrowaway3 wrote: post #120
dougmwne wrote: post #77

Earlier quoted context omitted.

I think it can actually do all those things reasonably well. I’d like to start getting meeting summaries that tell me my action items based on a transcript. And it can generate infinite boot-licking butt kissing if you just ask it. Generate and click send to your boss!

That would actually be an interesting product: spell-check but it edits your email or slack message to be more sycophantic. I'd use it.

I should probably start doing that. I’m a self-important jerk usually.

Re: GPT-4 is phenomenal at Code

#137

Post by CapstanRoller »

moonchrome wrote: post #107
CapstanRoller wrote: post #73

Earlier quoted context omitted.

>it would let me handle at least two or three IC jobs At that point you'll need to have at least two or three IC jobs just to survive, because wages will plummet.

I think experience is going to become a multiplier of value here, there will be a first adopter advantage and lower cost is going to create demand.

"Tendency of the rate of profit to fall" affects labor, too.

Sure, you may have a temporary advantage by leveraging AI and effectively doing labor arbitrage (the same way some craftsmen could gain a temporary advantage over others if they had access to rare/expensive tooling in the early days of the Industrial Revolution) but this advantage will quickly disappear as AI tooling becomes cheap and ubiquitous.

Re: GPT-4 is phenomenal at Code

#138

Post by sualehasif »

Iamnotthatbad wrote: post #32
sualehasif wrote: post #3

Earlier quoted context omitted.

If you know of any other cool examples or would like us to try something out and add to the repo, please let us know. If you have access, the repo should be fairly replicable.

Your example of the commenting doesn't seem like "well-commented code" to me. Just re-stating what is in code in plain language isn't helpful and is kind of annoying to read. Can you explain why you think it is helpful?

The commenting was for the model to learn what the types were so it could translate it to gather. Doing that one shot is surprising to me because it is somewhat hard to get pytorch tensors to work out to the correct shapes.

Re: GPT-4 is phenomenal at Code

#139

Post by LightMachine »

amelius wrote: post #84
LightMachine wrote: post #9

It managed to write, in one shot, a working λ-calculus parser, using a very specific programming style I asked it in JavaScript, and then translated it all to Python, including sarcastic, rhyming GLaDOS comments. https://twitter.com/VictorTaelin/status/1635726202231988225 It also seems to be extremely competent at writing Agda types and proofs. We need some tool that highly integrates it with entire codebases, allowi…

I don't think this is a good test. Every CS graduate student has to write a lambda-calculus parser. There must be thousands of implementations on the web. It really is not strange that GPT-4 can reproduce this.

I sort of agree, but it still amazes me that it can get the code is correct, even though I asked for a highly specific style (i.e., use of recursion, representing "None" as null, the format of the JSON, making local functions, etc.). So, even thought it has never seen that exact implementation, it still assembles a working function that just works. If it was just mixing up different code it recalled from memory, it would likely have a bunch of silly errors here and there that I'd have to fix manually, but no, it just works. That's what impressed me.

Re: GPT-4 is phenomenal at Code

#140

Post by mjr00 »

aiappreciator wrote: post #118
mjr00 wrote: post #66

Earlier quoted context omitted.

I remember it being vastly overhyped about how artists were now obsolete, but it really hasn't happened. People have quickly became very attuned to the "AI art style"--humans are excellent at pattern recognition, after all--and if anything, it's made people appreciate how important an artists' individual style and attention to detail is.

It absolutely has happened. I've seen intense rage and despair for even top tier 2D artists, Japanese, Chinese, Western all alike. Like top of industry, massive fanbase, they all understand there is no sugarcoating it anymore. AI improved from 5-year-old scribble to crushing 90% of artists in technical execution within 9 months. So they had no time to process it at all, and had to cope with shock and denial. Now virt…

> It absolutely has happened.

Could you list some examples of companies that have laid off their art departments to use DALL-E/SD/etc?

Post reply on HN