[deleted]
Testing GPT 4's code-writing capabilities with some real world problems
21–30 of 677 posts
Re: Testing GPT 4's code-writing capabilities with some real world problems
#22In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…
This post feels like the people that go into linux forums and say linux sucks because I can't get it to do X but microsoft can, but then get 400 replies and several that show
GPT has limited reasoning but given enough knowledge of the problem you can coerce it to do surprising things so long as you can relate it to something in else in the knowledge base. Given how big that knowledge base is, you can get lucky surprises where things just work if you fish around enough
Re: Testing GPT 4's code-writing capabilities with some real world problems
#23Code is just the rendering of an algorithm.
It might be best to prompt it with a high level description of an algorithm, then iteratively prompt it to refine its prior output or add more detail. Render to code should be the final step.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#24I skipped the article and only read the prompt. I have no idea what to do with fire. Why does it matter what it costs if you must avoid it?
Re: Testing GPT 4's code-writing capabilities with some real world problems
#25In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…
I think most people see where the puck is going, and even where it is right now is very impressive. It's not hard to see that it will be likely less than 5 years before it will be able to do what you did on day 2, and much more, at a tiny fraction of the cost, with no downtime, no attitude problems, sick days, etc. The things you mentioned (taking on board feedback, testing across devices, iterating on solutions) doe…
That's what they said about Watson and Wolfram Alpha. You only hear this nonsense on HN.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#26What's most astounding is that it was expected to work when so much of the prompt just doesn't make sense:
> Objects should only move if they will end up on an empty tile after their move
> "An object is free to move onto a tile that another object moves onto if that other object is moving off of its tile"
Re: Testing GPT 4's code-writing capabilities with some real world problems
#27Earlier quoted context omitted.
I think most people see where the puck is going, and even where it is right now is very impressive. It's not hard to see that it will be likely less than 5 years before it will be able to do what you did on day 2, and much more, at a tiny fraction of the cost, with no downtime, no attitude problems, sick days, etc. The things you mentioned (taking on board feedback, testing across devices, iterating on solutions) doe…
That's what they said about Watson and Wolfram Alpha. You only hear this nonsense on HN.
Demonstrably untrue, it’s taken up semi-residence in everything from oped pages to boardrooms since the New Year. Welcome to the mid singularity, my friend.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#28GPT3 worked well for me with smaller programming tasks.
I.e, helper functions, api calls, etc
In those cases it was easier to type : Write a javascript function that does X
It totally failed for me creating a nice looking website using bootstrap. While GPT3 created a workable outline, it never looked right and the css adjustments never worked.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#29I had a similar experience earlier. Described a problem that isn't even that hard - very similar to something there are probably lots of examples of online but subtly different. I wanted to see if handled these subtly different requirements.
It failed miserably, even with repeated instructions. It just assumed I wanted the more common problem. Every time I pointed out the problem it would say "sorry for the confusion, I've fixed it now" and give me back identical code. I even asked it to talk me through test cases. It identified that its own code didn't pass the test cases but then still gave me back identical code.
I eventually gave up.
Re: Testing GPT 4's code-writing capabilities with some real world problems
#30I just used GPT-4 yesterday to write a Go-parser for a specific JSON input.
Within two prompts it could read the JSON data from a stdin stream, unmarshal it to Go structs and print the correct fields to stdout as a human-readable line of text.
Then I told it to colour the timestamp and id fields using the fatih/color -package, and it did it correctly.
In total it took me about 4-5 prompts to get where I wanted. I just needed to fine-tune the printing to stdout part a bit to get it just how I liked, but it saved me a ton of boring template code writing and iteration.
I could've done it easily myself, but there were a few fiddly bits that would've required me to look up the documentation to check the exact way to do things. GPT4 had it correct from the start.
Then I asked it to write unit tests for the code, and it confidently started writing correct-looking code that would take the same input and expect the correct output, but just stopped in the middle. Three times. I stopped trying.
And another case:
I tried to use GPT-3.5 to write me a program that would live-tail JSON-logs from Sumo Logic and pretty-print them to stdout. It confidently typed out completely correct code with API endpoints and all. ...except the endpoints didn't exist anymore, Sumo Logic in their great wisdom had removed them completely. The only solution is to use their 5 year old binary-only livetail executable.
GPT4 with the same input gave me a shell-script that starts a search job with the correct parameters and polls the endpoint that returns the result when it's done.
The speed at which this is developing is really fascinating, I'm not really afraid for my job but I do love how this will automate (some of) the boring stuff away a bit like GitHub CoPilot did, but better.