Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

71–80 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#71

Earlier quoted context omitted.

This post feels like the people that go into linux forums and say linux sucks because I can't get it to do X but microsoft can, but then get 400 replies and several that show GPT has limited reasoning but given enough knowledge of the problem you can coerce it to do surprising things so long as you can relate it to something in else in the knowledge base. Given how big that knowledge base is, you can get lucky surpri…

the big thing to me is programming isn't hard I've worked on adtech, crypto, fintech, gamedev, startup founder, BigCo. Not once was programming something that was a time sink. Makes me feel like GPT is marketing to the incompetent or something.

> Not once was programming something that was a time sink.

Can you please expound on that? Do you mean not hard as in comparison to something else?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#72

Earlier quoted context omitted.

This post feels like the people that go into linux forums and say linux sucks because I can't get it to do X but microsoft can, but then get 400 replies and several that show GPT has limited reasoning but given enough knowledge of the problem you can coerce it to do surprising things so long as you can relate it to something in else in the knowledge base. Given how big that knowledge base is, you can get lucky surpri…

the big thing to me is programming isn't hard I've worked on adtech, crypto, fintech, gamedev, startup founder, BigCo. Not once was programming something that was a time sink. Makes me feel like GPT is marketing to the incompetent or something.

Exactly, we know how much code gets written per programmer hour, its in the single digits on average. Speeding that up wont make a big difference.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#73
post #50
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I don’t think you’ve tried GPT-4. If you have, I don’t think you are like majority of devs (maybe not on HN, but in real life). You sound lucky to have true, novel problems to solve each day. I’m with many here commenting that this is quite powerful stuff, especially when my day-to-day is writing simple CRUD apps, or transforming data from one format to another within an API, or configuring some new bit of infra or C…

On the flip side, I am having a great time charging enterprises exorbitant hourly rates to come in and help them fix their crappy crud programms.

But I still don’t know if I am thrilled by the prospect of being called in to make their monkeypatched, GPT generated project work.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#74

A modified A* that solves the fire routing problem (less efficiently than OP's I think). Each A* location stores where it comes from, how long it takes to get to it, and how many fires it passed through to get there. The algorithm only considers fire cells neighbors if the current number of fires passed through is less than the current fireWillingness global. 1. count fire tiles within movement range 2. run A* from s…

That algorithm (implemented efficiently) is just A* using a different concept of distance. The distance specifically would be `fire*episilon + steps if steps < max else inf`

It doesn't work if you just change the distance. Having implemented similar variations of A* I agree with Tyler. You need to change more than distance to get this to work.

Usually you need to change the search space and increase the number of states you go through to get the algorithm to differentiate between things you want and things you don't want to happen in your final result.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#75
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

Eh, it's still impressive that these systems can write such good code despite pretty much just predicting the next word. I guess it's a matter of perspective. You can either be astounded by how much it can do relative to your expectations from 2018 or you can be skeptical relative to the volume of excitement. I think some excitement is also people extrapolating to the future: if predicting the next word gets you this…

> despite pretty much just predicting the next word

Is GPT really like the old school NLP trigram stuff?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#76
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I think most people see where the puck is going, and even where it is right now is very impressive. It's not hard to see that it will be likely less than 5 years before it will be able to do what you did on day 2, and much more, at a tiny fraction of the cost, with no downtime, no attitude problems, sick days, etc. The things you mentioned (taking on board feedback, testing across devices, iterating on solutions) doe…

Personally I think it's 99% hype. The current iteration and architecture of these systems means they will never be at the level where they can actually replace a programmer. The best they will ever get is barfing up snippets for a programmer (who still needs the industry-specific knowledge to VERIFY the snippets).

Additionally, "the rate of increase in capabilities" is very much a false flag. Past performance (especially for second-order things like 'rate of improvement') is an absolute batshit insane metric for predicting future success.

https://russellinvestments.com/us/blog/past-performance-no-g...

The advancements have eaten low hanging fruit. Once all the low hanging fruit is gone, we'll all realize GPT will never be tall enough to reach the good stuff.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#77
Yesterday evening I thought it would be fun to try to do a Raycaster in Javascript with the help of GPT-4. Experience was mixed.

1. Basic rendering logic was a breeze. I barely had to change anything, just copy paste, and I have a map with walls that were darker the further away they were, using textures, and basic movement using arrow keys. For an inexperienced graphics programmer like me probably saved hours getting to that point.

2. I asked it to add a minimap. Did not work perfectly at the first try, but after a few minutes of exchanging messages, it worked and looked okay.

3. I asked for an FPS display. Worked on first try.

4. Now I asked for a solution to render walls of different heights. Here I had to correct it a few times, or suggest a different approach, but it got it working halfway correct (but not very performant). Definitely took way longer than steps 1 to 3 combined (30+ minutes).

5. I asked for floor rendering (often called "floorcasting"). Here it completely failed. The code it suggested often looked like it might be the right approach, but never really worked. And the longer we exchanged messages (mostly me giving feedback whether the code worked or suggesting possible fixes), the more it seemed to hallucinate: very often variables suddenly appeared that were defined nowhere or in a different scope. At that point, it became increasingly frustrating for me, and I often closed the chat and "reset", by posting my complete working code, and again prompting for a solution to the floor rendering. Still, until I went to bed, it did not produce any working solution. In retrospect, it would probably have been faster to read a tutorial how the floorcasting should work, and implement it myself like a caveman, but that was not what I was aiming for.

It was definitely fun, and I can clearly see the potential time-savings. But maybe I have to learn when to recognize it won't bring me past a certain point, and I will save time and nerves if I switch to "manual control".

Re: Testing GPT 4's code-writing capabilities with some real world problems

#78

What's most astounding is that it was expected to work when so much of the prompt just doesn't make sense: > Objects should only move if they will end up on an empty tile after their move > "An object is free to move onto a tile that another object moves onto if that other object is moving off of its tile"

If the issue was the prompt, I expect someone will prompt it with a better wording of the problem and get a decent algorithm, and post it in this conversation. Let's see what happens...

Re: Testing GPT 4's code-writing capabilities with some real world problems

#79

To be fair, in 15 years writing code, I have spent the vast majority of my time working on minor variations of problems already solved. I am not a fancy developer coming up with new algorithms. I make sign up flows, on-boarding flows, paginated lists, integrations to other apis. And I definitely feel that my job might be threatened by LLMs.

Tbf, most of my time as a programmer was neither spent solving old problems nor solving new problems. Most of it was spent either fiding the bug hidden somewhere in the huge code base, or trying to get the business people to be clear on what their requirements actually are.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#80
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I feel it's the average programmer who is enamoured with GPT. The programmers working on hard stuff know that AI writing programs is still far away.

It is not just those who are working on hard problems who are of that opinion.

Good luck having a ML model understand a 20 year old undocumented dataformat developed inhouse at a specific research lab to be used in their proprietary systems which are also undocumented and are a part of a spiderweb of interconnected systems at that lab (I have worked at this particular lab).

It will be a long time (if ever) until a ML model will be able to handle these situations (and I hazard a guess that most of the worlds active code is something akin to this).

As a supporting tool for the software engineers working there, sure. Just like a linter.

Post reply on HN