Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

111–120 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#111

Post by TylerGlaiel »

akamoonknight wrote: post #12

One thought that came to mind when the author mentioned that GPT-4 seemed to want to use A* was that maybe the problem statement was not formed in a way that would get the best response. I'm not entirely sure if this type of back and forth would be useful, but just struck me as interesting to try. I put the following prompt into ChatGPT (no GPT-4 access for me yet): *A problem statement is given as follows: An agent…

Interesting, not a single one of these things it asked for clarifications on are things it actually got wrong in its suggested solution

Re: Testing GPT 4's code-writing capabilities with some real world problems

#112

Post by anonzzzies »

theK wrote: post #100
hnfong wrote: post #83

Earlier quoted context omitted.

"Hey GPT-X: Can you refactor the codebase in gpt@monkeypatched-crap.git for me? Preferably in the style of gpt@crappycrud.git that I worked on last year."

Token cap will probably be the biggest problem here. After validation. After getting the changes to disk, documented, actually compiling, etc… But the biggest problem is that transferring the nuance that is external to the code base is typically really tiresome and lengthy grunt work and again token cap.

They went from 4K to 32k in a few months. Wait a few years.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#113

Post by Jensson »

anonzzzies wrote: post #112
theK wrote: post #100

Earlier quoted context omitted.

Token cap will probably be the biggest problem here. After validation. After getting the changes to disk, documented, actually compiling, etc… But the biggest problem is that transferring the nuance that is external to the code base is typically really tiresome and lengthy grunt work and again token cap.

They went from 4K to 32k in a few months. Wait a few years.

GPT-3 is 3 years old, not a few months.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#114

Post by pixl97 »

photonbeam wrote: post #64
dr_dshiv wrote: post #43

Earlier quoted context omitted.

Except Watson and Alpha weren’t useful out of the box to everyone. They weren’t the fastest growing app of all time and couldn’t even remotely play that way. ChatGPT isn’t hype because it is so useful — it’s already on the “Plateau of Productivity”. But it looks like an exponentially steep climb from here. indefinitely

They said the same thing about transistor scaling

Being that you and I are talking to each other by forming thoughts in meat that us causing other meat to move around tells me that digital thought still has a lot of scaling room ahead of it.

Maybe superintelligence isn't possible for some reason, but the fact we exist should tell you that general intelligence is not magic.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#115

Post by bboygravity »

nitwit005 wrote: post #58
maxdoop wrote: post #50

Earlier quoted context omitted.

I don’t think you’ve tried GPT-4. If you have, I don’t think you are like majority of devs (maybe not on HN, but in real life). You sound lucky to have true, novel problems to solve each day. I’m with many here commenting that this is quite powerful stuff, especially when my day-to-day is writing simple CRUD apps, or transforming data from one format to another within an API, or configuring some new bit of infra or C…

If all you're doing is very simple crud apps and transforming API responses, your coding is already replaceable with "no code" tools.

Except it isn't and it won't be until 10 layers of management learn what no-code tools are and why they should want them.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#116

Post by paganel »

Maxion wrote: post #99
shinycode wrote: post #93

Earlier quoted context omitted.

Also if all the company is doing is boring crud app as a service then the whole company is going to disappear once anyone can ask ChatGPT to create a version of the service for themselves

Most CRUD apps in enterprise settings relate to something in the real world. E.g. ordering systems, warehouse management and so on. The hard part here isn’t really the coding - but understanding the business logic and getting it out of the customer what they need. Often the customer hasn’t even got a good enough grasp on their own business to be able to formulate what they need. In the short term this ai tools will m…

Excellent response.

I also wanted to add about the myriad of incoming data formats that need to be processed and the myriad of data exports that one has to implement for most of those "boring" CRUD apps.

If one hasn't written code that includes comments like "Special case, California does it this way" or "Alberta needs an .xml export, not an .xsl one", with a link to a .pdf spec that points to somewhere on the internet, then he/she hasn't got to know what it really means to write a true and "boring" CRUD app.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#117

Post by [deleted] »

scg wrote: post #19

As a human programmer I didn't quite understand the problem statement until I read the whole article and the tests. I believe the goal is to find a path with the fewest possible "fire" cells and the minimum cost as a tie breaker. The cost of a path is the sum of its cells' cost and it can't be greater than 5. If I understood the assignment correctly, I don't think the problem statement is equivalent to what's include…

[deleted]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#118

Post by Jensson »

noitpmeder wrote: post #108
Jensson wrote: post #97

Earlier quoted context omitted.

If you do nontrivial coding then GPT wont help you, so that isn't relevant for the discussion.

I disagree. GP, by their statement and appeal "I've worked on adtech, crypto, fintech, gamedev, startup founder, ...", implies that he has worked on complicated software issues, and that he thinks GPT is a promising replacement for developers working on those problems.

> implies that he has worked on complicated software issues

No it doesn't, there is a lot of simple "gluing APIs together" to do in all of those. The hard part then is figuring out what you want to do, not getting the code to do it.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#119

Post by copperx »

akamoonknight wrote: post #12

One thought that came to mind when the author mentioned that GPT-4 seemed to want to use A* was that maybe the problem statement was not formed in a way that would get the best response. I'm not entirely sure if this type of back and forth would be useful, but just struck me as interesting to try. I put the following prompt into ChatGPT (no GPT-4 access for me yet): *A problem statement is given as follows: An agent…

This is perhaps one of the most impressive responses I've read. It truly seems like there is some reasoning happening. I don't understand how this can be the output of a generative LLM.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#120

Post by TylerGlaiel »

photochemsyn wrote: post #14

The only take-home message here is that people who claim to write 'self-documenting code' are well, let's not be hyperbolistic, but come on. No comments on that code example? Every line could have an explanatory comment, then the author could remember what they were thinking at the time and it would probably help the AI out too. > "People who claim code can document itself considered harmful"

chill, I'm the only programmer on the project, and I don't have any problems understanding what the code is doing (only lost track of some of the "why", the process that led me there. which was only relevant here because I was trying to recreate that process with ChatGPT). The original algorithm involved a ton of trial and error from my end, so the "why" is really just "I tried a bunch of permutations of this and ended up with this as the version that worked".
Post reply on HN