Testing GPT 4's code-writing capabilities with some real world problems • Hacker News Board

Re: Testing GPT 4's code-writing capabilities with some real world problems

#511

Post by kneel » Fri, Mar 17, 2023, 5:43 PM UTC

I self taught myself how to code and have never been very good, I don't code often and when I do I spend a lot of time relearning some simple programming detail I forgot.

ChatGPT (also copilot) allows me to focus on the project that I'm working on and offload the stack overflow searches to prompting. I don't have to find a similar error someone else posted on SO and figure out how it applies to my current problem. I can keep a high level view of the project and not get bogged down with silly bugs, learning new libraries, or deciphering someone else's code.

I imagine there are a lot of people who are in a similar situation, it's crazy that we've just unleashed this massive productivity booster onto millions of people.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#512

Post by cdchn » Fri, Mar 17, 2023, 5:45 PM UTC

If you look at the Leetcode scores, it looks like GPT-4 can generally do most "basic" leetcode but fails on "medium" or "hard" problems. This seems to align with what I see most people's experience with using GPT-3/3.5/4 to generate code seems to be. Works well for simple cases (which you could probably find examples of online) but stumbles on nuances of incrementally harder problems.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#513

Post by sagebird » Fri, Mar 17, 2023, 5:46 PM UTC

I would ask it to pretend a mathematician knows how to solve it, and is having a conversation with a novice programmer attempting to solve the problem, and pointing out mistakes and hints at each step, and gives examples of where it fails, until a proof is given that the program is correct.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#514

Post by braingenious » Fri, Mar 17, 2023, 5:46 PM UTC

danielbln wrote: post #214

braingenious wrote: post #41

I tried out gpt4 today with the task of “take some html files made by a non technical person using various versions of microsoft word over a decade ago and put the contents into a csv” and it hasn’t done great. Not terrible, but not great. That being said, I don’t know anybody talented enough to handle it that would even look at this project for $20 so ¯\_(ツ)_/¯

An alternative path would be to tell it to write python (or $LANG) code that can parse these HTML files and output the right CSVs.

That’s what I’m doing! I found myself amazed by the sheer terribleness of the html that Microsoft Word output. I managed to get it to write a script to clean up the files last night, but it took a surprising amount of finessing to avoid the myriad footguns involved in dealing with this godawful format.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#515

Post by agilob » Fri, Mar 17, 2023, 5:46 PM UTC

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

- ... 2 days later...

- us: umm, there's something not specified in . What about this part here that says that...

- ... 2 days later...

Can you replace this part with ChatGPT talking to another ChatGPT to generate questions and answers instantly?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#516

Post by kneebonian » Fri, Mar 17, 2023, 5:48 PM UTC

kneel wrote: post #511

I self taught myself how to code and have never been very good, I don't code often and when I do I spend a lot of time relearning some simple programming detail I forgot. ChatGPT (also copilot) allows me to focus on the project that I'm working on and offload the stack overflow searches to prompting. I don't have to find a similar error someone else posted on SO and figure out how it applies to my current problem. I…

> I imagine there are a lot of people who are in a similar situation, it's crazy that we've just unleashed this massive productivity booster onto millions of people.

Maybe it makes me sound like an elitist git, but I remember when the coding bootcamps started "unleashing massive productivity boosts" by promising to make people "full-stack engineers" in just a 6 weeks, and I still shudder to remember the code horrors I've seen as a result of that.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#517

Post by geysersam » Fri, Mar 17, 2023, 5:51 PM UTC

lebuffon wrote: post #357

zeroonetwothree wrote: post #342

Earlier quoted context omitted.

How much more efficient is programming in Python with modern tools vs assembly? I would bet more than 5x

That's an interesting case and use of the word efficient. Developer time reduced by 5X CPU time increased by 500x When CPU cycles are cheap it's a logical tradeoff but it still grates against my sensibilities.

What does it even mean to say that a hypothetical assembly program is 500x faster than an existing Python program, when the assembly program does not even exist?

It's some kind of philosophical question.

Maintaining an assembly version of a modern software stack is not 5x more costly, it's simply not possible.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#518

Post by twic » Fri, Mar 17, 2023, 5:51 PM UTC

vages wrote: post #253

IKLOL wrote: post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

If increased productivity equaled job loss, there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950. A lot of projects today are not even greenlit because they would be too expensive to make. For instance, there are a lot processes in almost every country that require you to file paper forms, even though we have had web forms and…

Correct. This is pretty much Jevons's paradox: https://en.wikipedia.org/wiki/Jevons_paradox

Re: Testing GPT 4's code-writing capabilities with some real world problems

#519

Post by whateveracct » Fri, Mar 17, 2023, 5:52 PM UTC

stormfather wrote: post #184

whateveracct wrote: post #86

Earlier quoted context omitted.

I don't even think it's useful for the incompetent. I do think the incompetent think it's useful tho.

That's like saying a calculator isn't useful just because you're good at mental math. The ability to adapt and extract value from new tools is a mark of competence.

Using natural language to code isn't going to help me. It's less dense informationally already. I don't have a stream of natural language in my head when I program or build software, so it's not free to use GPT.

This may be a Haskell thing. But I did try and even for small programs it was cute but not actually an efficiency gain over me just doing it.

Not to mention in real life, I have to deal with fallout and bugs and I think GPT deprives me of key processes that allow me to be excellent in that regard. If you aren't a coder and code a backend with GPT, what do you do when the shit hits the fan?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#520

Post by logifail » Fri, Mar 17, 2023, 5:53 PM UTC

danielbln wrote: post #486

logifail wrote: post #435

Earlier quoted context omitted.

> Those parts would still be handled with GPT-X talking to another GPT-X in the other company. Even the requirements might be given by a GPT-X. What happens if one (or more) of the GPT-Xs starts having hallucinations while they're busy working on this project? > Second, even if that's not the case, the part of doing the communication can be handled by non-programmers. I was in a meeting with a sales manager and tech…

Are hallucination is a systemic for-ever problem that will not be solved, mitigated or in akne other way rendered inconsequential? Also, having conversations about things you don't understand with a machine, where you don't have to keep up social decorum and can ask the dumbest questions should help a lot with improving the decision making of non-technical personel

> having conversations about things you don't understand with a machine, where you don't have to keep up social decorum and can ask the dumbest questions should help a lot with improving the decision making of non-technical personel

This sales manager didn't know that neither he - nor his customer - properly understand the product until he was specifically called out on it (by me, as it happens. At least the boss laughed). Are we expecting GPT-X to sit in on Teams meetings and ask questions like that, too? Sales manager literally did not know he needed to start asking questions about the product. Tech team were calling BS on it as soon as he reported his conversation with customer.

"It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.", which may (or may not) have been said by Mark Twain.