Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

121–130 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#121

Post by anonzzzies »

whateveracct wrote: post #52
maxdoop wrote: post #50

Earlier quoted context omitted.

I don’t think you’ve tried GPT-4. If you have, I don’t think you are like majority of devs (maybe not on HN, but in real life). You sound lucky to have true, novel problems to solve each day. I’m with many here commenting that this is quite powerful stuff, especially when my day-to-day is writing simple CRUD apps, or transforming data from one format to another within an API, or configuring some new bit of infra or C…

There's not such thing as actually boring CRUD. I've worked at many companies and founded my own. Even when it felt like CRUD, a year+ in it was clear that tasteful decisions pay off and iteration and cost gradients matter. GPT doesn't sniff that. I agree with GP - day 2 dev me outclasses it, which means it isn't replacing anyone.

HN echo chamber. Generally the better/best programmers hang around here; your day 2 was probably better than many coders hope to achieve in their whole lives. It is replacing people already; I personally know about 10 people who have been fired or assigned to a completely different position because gpt did a faster and better or equal job. So ‘not anyone’ is simply nonsense; I am only one person, there will be many many more who see the same thing. Of course they are not told they are replaced by ai when fired, but their managers or more skilled colleagues know.

I do agree that there is no boring crud; that’s why gpt or no code tools are not good for full solutions (yet), but it’ll get there I am sure.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#122

Post by anonzzzies »

Jensson wrote: post #113
anonzzzies wrote: post #112

Earlier quoted context omitted.

They went from 4K to 32k in a few months. Wait a few years.

GPT-3 is 3 years old, not a few months.

Sure but the token jump to 32k is new isn’t it? Even 8k to 32k is significant and that was this year.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#124

Post by jrumbut »

qlm wrote: post #29

I had a similar experience earlier. Described a problem that isn't even that hard - very similar to something there are probably lots of examples of online but subtly different. I wanted to see if handled these subtly different requirements. It failed miserably, even with repeated instructions. It just assumed I wanted the more common problem. Every time I pointed out the problem it would say "sorry for the confusion…

It has failed every meaningful programming challenge I've given it (to be fair, I only ask when I've got something difficult in front of me).

I do wonder if part of it is that my prompts are made worse because I have a partial solution in mind.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#125

Post by malborodog »

noitpmeder wrote: post #76
jacobedawson wrote: post #16

Earlier quoted context omitted.

I think most people see where the puck is going, and even where it is right now is very impressive. It's not hard to see that it will be likely less than 5 years before it will be able to do what you did on day 2, and much more, at a tiny fraction of the cost, with no downtime, no attitude problems, sick days, etc. The things you mentioned (taking on board feedback, testing across devices, iterating on solutions) doe…

Personally I think it's 99% hype. The current iteration and architecture of these systems means they will never be at the level where they can actually replace a programmer. The best they will ever get is barfing up snippets for a programmer (who still needs the industry-specific knowledge to VERIFY the snippets). Additionally, "the rate of increase in capabilities" is very much a false flag. Past performance (especi…

But obviously increasing avg and sr programmer productivity by many x implies many fewer jobs for programmers. That’s the point we’re interested in.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#126

Post by pixl97 »

theshrike79 wrote: post #30

I just used GPT-4 yesterday to write a Go-parser for a specific JSON input. Within two prompts it could read the JSON data from a stdin stream, unmarshal it to Go structs and print the correct fields to stdout as a human-readable line of text. Then I told it to colour the timestamp and id fields using the fatih/color -package, and it did it correctly. In total it took me about 4-5 prompts to get where I wanted. I jus…

>Then I asked it to write unit tests for the code, and it confidently started writing correct-looking code that would take the same input and expect the correct output, but just stopped in the middle.

One of two things. First ask it to continue. Sometimes it just stops half way thru code foe whatever reason.

The other possibility is you filled up the token context window. Not much you can do but wait for the 32k model.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#127

Post by spaceman_2020 »

nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I started learning how to code about 6 months ago, mostly to build prototypes of a couple of app ideas. I have no intention of getting a coding job - if the prototype is successful, I'll seek a technical co-founder.

Last couple of months, I've been using chatGPT to write a lot of features and functions. I don't think it has made me a better coder, but it has made me massively more productive. Things like scraping data from a URL - something I would have had to sit through an hour long tutorial to learn - is accessible with a single query.

I also think that the code quality has improved over the last few iterations. It makes fewer mistakes now.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#128

Post by TylerGlaiel »

FeepingCreature wrote: post #13

Yep: the biggest remaining weakness is that it's incapable of thinking deeply and iteratively. This is an architectural limitation (lack of reflectivity), but to fix it will probably usher in the singularity, so maybe we should be glad for it. I suspect if you poked GPT-4 just right (starting with a detailed design/analysis phase?) it could find a rhetorical path through the problem that resulted in a correct algorit…

I have not tried, but I suspect if I described the algorithm I have instead of the problem that it could translate the algorithm into code pretty well. But I'm also unsure of that, some experiments with GPT 3.5 I did would definitely cause it to default to a common solution (ex, A) if the description was sufficiently similar to A, or not realize that a small deviation was intended. But also like... the point here was to see if it could solve a hard problem that has a non-obvious solution. not if it can translate an english description of an algorithm into code.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#129

Post by Jensson »

anonzzzies wrote: post #122
Jensson wrote: post #113

Earlier quoted context omitted.

GPT-3 is 3 years old, not a few months.

Sure but the token jump to 32k is new isn’t it? Even 8k to 32k is significant and that was this year.

Yeah, but the jump to 32k didn't take a few months, it was years in the making. Otherwise you could extrapolate with "yesterday we had 4k, today we have 32k, tomorrow we will have 256k", that isn't how we do it. If we follow the same exponential pace 256k would have to wait 3 years, and even that is unlikely.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#130

Post by vincnetas »

bboygravity wrote: post #115
nitwit005 wrote: post #58

Earlier quoted context omitted.

If all you're doing is very simple crud apps and transforming API responses, your coding is already replaceable with "no code" tools.

Except it isn't and it won't be until 10 layers of management learn what no-code tools are and why they should want them.

And i guess this will never happen as management at the first sign of trouble making AI do that they want, will gladly pay someone else to do it for them. Of course as little as possible, but either way they will be glad to delegate.

It's same for me. I can learn how to tile my bathroom or repair my car, but i just don't feel like and am happy to pay someone else.

Post reply on HN