Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

461–470 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#461

Post by sgregnt »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Make the AI handle the conversation on both sides? No need to wait a few days for back and forth

Re: Testing GPT 4's code-writing capabilities with some real world problems

#462

Post by gwoolhurme »

IsaacL wrote: post #385
gwoolhurme wrote: post #300

I guess my genuine question is to the people who are saying this is a big deal and it will take our jobs. I am a bit lucky to where at the moment I am working in a "novel" field. Lets say though for the sake of argument that AI does come for SWE jobs. To be honest? I don't know what to do in that case, I have no backup plan, not enough to retire. The country I've lived in for 9 years is still through a work visa (hop…

Check out my comments higher up in the thread (eg https://news.ycombinator.com/item?id=35197613 ), I really do believe that GPT4+ will be primarily useful as augmenters for capable and dedicated engineers, rather than replacements. It's like a very eager and brilliant junior dev that can solve many problems that you throw at it, but still needs hand-holding, error-checking, and someone who knows how to piece the actu…

Thanks for the reply! I don’t know what the future holds. The country I live in will even just on the job train more programmers if need be. It is kind of nice in that sense. If what you say is right? Then for the most part everyone is happy. The problem is, if we assume the worst case scenario, that it does take all our jobs, well then what? To people with those opinions, my question is what do I do? What does anyone do?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#463

Post by GrumpyNl »

I asked it to write the code for all the unique combinations of A,B,C,D in PhP, after 27 tries it succeeded. The i asked it to solve the problem of a horse is 15 dollar, a chicken one dollar and a egg .25 dollar, i can spend 100 dollar for 100 items, some of each. After 2 hours, it was not able to solve it. One time it gave 5 possible answers, with the correct one also, but it did not recognize the correct one.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#464

Post by thingification »

r3trohack3r wrote: post #422

Watching GPT write code hurts my brain. It starts at the first character, works forward one “token” at a time, and ends at the last character. Never moving back. It feels like it knows where it’s going at the first character, even though it doesn’t. It’s like it starts speaking a sentence and, by the time it’s done speaking, it’s written a syntactically correct Node.js application. The way GPT communicates in English…

Speaking from ignorance (I've not studied attention nor transformers): This is my feeling too. I feel like the next step isn't far away: a more explicit model of the world with a mechanism to both query and "feed back on" that model, correcting mistakes.

If it's possible to get so far when that functionality seems in an important sense basically missing, imagine how far it'll go when that does happen.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#465

Post by pixl97 »

swexbe wrote: post #336
pixl97 wrote: post #114

Earlier quoted context omitted.

Being that you and I are talking to each other by forming thoughts in meat that us causing other meat to move around tells me that digital thought still has a lot of scaling room ahead of it. Maybe superintelligence isn't possible for some reason, but the fact we exist should tell you that general intelligence is not magic.

Your brain is analog. Trying to emulate this with a digital computer is bound to come with some computational cost.

The world created by human thought has given us such excess means of power production that we have literal gigawatts of electricity we can use in a trade off where it doesn't take us 4 billion years to make efficient circuits.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#466

Post by jdminhbg »

ruined wrote: post #441
jdminhbg wrote: post #394

Earlier quoted context omitted.

5X is a gigantic underestimate of how much developer time has improved. How long would it take a good assembly programmer to implement a CRUD web server connected to a database? Way more than 5x longer what a mediocre Python programmer would need.

an assembly programmer wouldn't write a web server connected to a database, they would implement a tui over tty, and it would be faster.

How good would it be at letting business users attach photos and emailed documents?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#467

Post by ben7799 »

I’m curious how long till we figure out if these algorithms are plagiarizing OSS or other code they come across like GitHub Copilot.

It requires special tools to actually figure out if this is happening. Having seen tests with such tools the problem seems a lot worse than commonly discussed.

Inserting stolen code or using OSS code in violation of licenses is going to be a big mess. Copying snippets versus pulling in dependencies creates tons of issues. Even if you get away with violating licenses you set yourself up for security issues if the tools plagiarize code with vulnerabilities in a way that won’t get updated.

It might mean this stuff is a useful tool for someone with a clue but not for someone who doesn’t know what they’re doing.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#468

Post by anon7725 »

theK wrote: post #131
anonzzzies wrote: post #112

Earlier quoted context omitted.

They went from 4K to 32k in a few months. Wait a few years.

Few years, yes. I’m with you “the change is coming” but we still need to transfer millions of tokens in and out to cater for context and out of repo intricacies.

Large enterprises will just pay to fine tune over their documents/code/etc.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#469

Post by evo_9 »

It's not bad. I just had it write a small API call to the NHL endpoint to gather some data for something I was curious about stat-wise.

Anyway, I initially had it write it in Python, and it mostly worked, but I was having some issues getting the data exactly right, and formatted the way I wanted.

Once I had it more / less right in Python, I had it rewrite it as a dotnet console app (C#), which is what I know best.

The only real issue I ran into is it would randomly stop before completing the conversion to dotnet. Like it would write 85% then just stop in the middle of a helper function. Not a huge deal, I just had it complete the last function, and with a little bit of fiddling in VS Code got it running pretty much the way I wanted.

So overall, yeah, not bad. Probably saved me an hour or so, plus I couldn't find great docs for the NHL endpoint, and ChatGPT was able to sus out the correct syntax to get to the data I needed.

I wonder how git Copilot compares, has anyone tried out both?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#470

Post by zamalek »

I have been writing a text editor, and I'm currently working on the VT100 stuff. Unit testing VT100 is a lot of busy work. There's a bunch of different message frames (DCS, OSC, CSI, etc.) and many, many, escape codes.

I decided to try out CodeGPT.nvim, and it was a massive help. It didn't provide perfect code, not by a long shot, but it gave me extremely valuable starting points - and did a somewhat decent job of exercising most of the branches (certainly enough for me to be happy): https://gitlab.com/jcdickinson/moded/-/blob/main/crates/term...

Many people have said it, and it's true. Expecting GPT to write a complete solution is just asking for problems, but it is an incredible assistant.

Post reply on HN