I had the exact same experience. Writing code for existing popular problems is phenomenal. But when you diverge slightly, it breaks down.
I asked it to write a regex which finds all html tags that has a specific class name, but does not contain another specific class name. I assume this problem has been tackled many times by scores of developers. It had outputted an excellent regex.
I asked it to ignore texts in inline script (such as event handlers), and it presented an invalid regex.
I tried to point out the problem but it just went into a loop of bad regex code.
Testing GPT 4's code-writing capabilities with some real world problems
51–60 of 677 posts
Re: Testing GPT 4's code-writing capabilities with some real world problems
#52In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…
I don’t think you’ve tried GPT-4. If you have, I don’t think you are like majority of devs (maybe not on HN, but in real life). You sound lucky to have true, novel problems to solve each day. I’m with many here commenting that this is quite powerful stuff, especially when my day-to-day is writing simple CRUD apps, or transforming data from one format to another within an API, or configuring some new bit of infra or C…
There's not such thing as actually boring CRUD. I've worked at many companies and founded my own. Even when it felt like CRUD, a year+ in it was clear that tasteful decisions pay off and iteration and cost gradients matter. GPT doesn't sniff that.
I agree with GP - day 2 dev me outclasses it, which means it isn't replacing anyone.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#53In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…
This post feels like the people that go into linux forums and say linux sucks because I can't get it to do X but microsoft can, but then get 400 replies and several that show GPT has limited reasoning but given enough knowledge of the problem you can coerce it to do surprising things so long as you can relate it to something in else in the knowledge base. Given how big that knowledge base is, you can get lucky surpri…
the big thing to me is programming isn't hard
I've worked on adtech, crypto, fintech, gamedev, startup founder, BigCo. Not once was programming something that was a time sink.
Makes me feel like GPT is marketing to the incompetent or something.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#54Actually, if the state A* searches through is not "tile reached" but "tile reached + count of fires on path", then it just becomes regular A*. This solves the A to C doesn't always go through B, because it turns B into multiple distinct states, some with fires, one without. There are a few issues with this. Search state is bigger (performance goes down), might not scale if other search features are needed in the game…
Did another pass through the article, and checked your code and GPT's code. The fun thing is you DID have similar insights, of changing the search space (including desire and bends in the cell). GPT never bothered to try (at least in the samples you provided).
Re: Testing GPT 4's code-writing capabilities with some real world problems
#55So, for a bit of fun, I signed up to GPT-4 thingy plus and I picked a fairly common web application and built it from scratch, only by talking to GPT-4 and copy pasting the code bits. I'm actually taken back by how well it's doing; including providing me some refreshers on stuff I forgot how it should work. I can see it failing at solving complex problems, but like the blog post mentions, most programming isn't new o…
Programming takes little effort. It's at most 20% of my effort. Feels like GPT is programming with extra steps.
Maybe it's for people who can never think programming is easy. Clearly there's a lot of such types. Explains a lot.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#56One thought that came to mind when the author mentioned that GPT-4 seemed to want to use A* was that maybe the problem statement was not formed in a way that would get the best response. I'm not entirely sure if this type of back and forth would be useful, but just struck me as interesting to try. I put the following prompt into ChatGPT (no GPT-4 access for me yet): *A problem statement is given as follows: An agent…
I really like that approach of asking it what needs to be clarified. That's going straight into my GPT toolbox.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#57A modified A* that solves the fire routing problem (less efficiently than OP's I think). Each A* location stores where it comes from, how long it takes to get to it, and how many fires it passed through to get there. The algorithm only considers fire cells neighbors if the current number of fires passed through is less than the current fireWillingness global. 1. count fire tiles within movement range 2. run A* from s…
That algorithm (implemented efficiently) is just A* using a different concept of distance.
The distance specifically would be `fire*episilon + steps if steps < max else inf`
Re: Testing GPT 4's code-writing capabilities with some real world problems
#58In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…
I don’t think you’ve tried GPT-4. If you have, I don’t think you are like majority of devs (maybe not on HN, but in real life). You sound lucky to have true, novel problems to solve each day. I’m with many here commenting that this is quite powerful stuff, especially when my day-to-day is writing simple CRUD apps, or transforming data from one format to another within an API, or configuring some new bit of infra or C…
If all you're doing is very simple crud apps and transforming API responses, your coding is already replaceable with "no code" tools.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#59I think what GPT-4 is missing: a feedback loop.
Imagine you were GPT-4 and being asked to write a small program, but you can't try it out yourself.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#60The job of a programmer, in a business context especially, is to take real-world requirements, and convert them into clearly defined systems where they can be solved / reasoned with.
I once had a manager telling me what needed to be done. Even with an actual person (me) in the loop, the code produced would often have glaring differences from what he wanted.
By its very nature, code requires a lot of assumptions. In any business context, a lot of things are implicitly or explicitly assumed. If you need a computer, or another person to give you exactly what you desire, you need to be able to spot the assumptions that are required to be made, and then clearly state them. And after a point, that's just programming again.
So this, or some other AI, is more likely to replace JS and python, or create another level of abstraction away from systems programming. But programmers will still always be required to guide and instruct it.