Testing GPT 4's code-writing capabilities with some real world problems • Hacker News Board

Re: Testing GPT 4's code-writing capabilities with some real world problems

#291

Post by brookst » Fri, Mar 17, 2023, 1:30 PM UTC

flippinburgers wrote: post #210

brookst wrote: post #185

Earlier quoted context omitted.

You seem to be saying that there is fixed demand for programmers and oversupply means less hiring. But the history of technology says that demand is recursive; the more tech produced, the more demand there is for producers. There may be a time when we hit the old “the whole world only needs 5 computers”[0] limit, but I don’t think we’re anywhere close. AI is providing leverage to create net more programming; it is no…

Sounds a bit too much like a perpetual motion machine to me.

Isn’t that true of any economy? Think of how many people are employed in general. How could this number of people have been employed 200 years ago?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#292

Post by tuatoru » Fri, Mar 17, 2023, 1:30 PM UTC

anonyfox wrote: post #195

guax wrote: post #189

Earlier quoted context omitted.

I have the same feeling, people are very concentrated on the ability of this AI generators to create working code from super specific and well formed prompts. When in reality, figuring out what the prompt should be accounts for 80% of the job.

don't fall into this mental trap. you can get into recursion quite easily here, and figuring out what to prompt can start from simple general questions - and there is no need for a developer at all, aside from the current limitations of copy/paste/run workflow has to be done manually

The raison d'être of COBOL (now Cobol) in the 1950s was that "there is no need for a developer at all". Managers can produce their own reports.

How did that turn out? A whole profession was created.

The thing that comes closest to "no developer needed" is Excel. But you really need someone who knows programming if you want a reliable, robust product, even in Excel.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#293

Post by cosmodisk » Fri, Mar 17, 2023, 1:32 PM UTC

IKLOL wrote: post #229

quitit wrote: post #200

Earlier quoted context omitted.

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

I'm kind of on the fence with this one. As someone who does programming at work,I don't want to be partially replaced by a prompt, however there are also lots of sectors, where the problems are big and solving them could help a lot of people but money is an issue,so being able to access resources on a scale of large tech companies would be amazing.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#294

Post by muzani » Fri, Mar 17, 2023, 1:33 PM UTC

superasn wrote: post #219

tarruda wrote: post #198

Earlier quoted context omitted.

Maybe if you train/tune GPT-4 with enough samples of similar interactions, it will learn to do it too.

If anything it will be much better as it won't mind waiting and asking for answers nor get frustrated with incompetence.

Honestly one of my favorite early applications of AI was as customer service. The CS people I talk to always seem pissed with something.

One conversation:

Me: "Hello, I'd like to ask why all the loyalty points from my MILES card is gone."

CS: "Sir, it says here that you upgraded your MILES card to a SMILES card. If you did not request a points transfer, they are gone."

Me: "Okay, how do I do that?"

CS: audible eyerolling "You should have done that while applying for an upgrade."

Me: "You can verify my details for both, right? Can't you transfer the points over?"

CS: "Ugh." Hangs up

Whereas AI would always politely answer questions, even the dumb ones like this.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#295

Post by MavisBacon » Fri, Mar 17, 2023, 1:36 PM UTC

I think the question also needs to be asked of- can GPT-4 write accessible/WCAG compliant code? I just spent the last two days at a digital accessibility conference called axe-con, financed by a firm called Deque who primarily make AI/ML powered software for the detection of accessibility issues in code

The resounding attitude seems to be that AI/ML is a friend to disabled users and can help do a lot of lifting with writing, maintaining, auditing code- but we are a long long ways away from fully automated processes that account for accessibility and produce websites that will work with assistive tech like screen readers, if it is possible at all

Re: Testing GPT 4's code-writing capabilities with some real world problems

#296

Post by Sunhold » Fri, Mar 17, 2023, 1:36 PM UTC

Planktonne wrote: post #258

Winsaucerer wrote: post #167

Earlier quoted context omitted.

That's not accurate at all for me. I'm not impressed with the output when rating it against humans who are good at said task. It's obviously not as good as those who are good in that relevant field. But it's certainly far better than humans who are not skilled at those tasks, and that is what I find to be very impressive. I just didn't realise that these models could be this good, and they're not even as good as they…

I think we're talking at cross-purposes here. I'm not underwhelmed by it being not as good as good humans at [task]; that's not a surprise. I'm saying that most people I see who are impressed by it for any given task are people who are not well-equipped to judge performance on that task.

I'm impressed by the talking dog because it's a dog that can talk, not because I think it's a great orator.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#297

Post by jmull » Fri, Mar 17, 2023, 1:36 PM UTC

sgarland wrote: post #241

There are plenty of edge cases where it fails. However, the one thing that made me think it actually knows (for certain definitions of the word "knows") what it's doing was asking it to re-write a non-trivial SQL query into equivalent relational algebra. I created a simplified schema from Northwind [0], gave it the CREATE TABLE statements for tables, and then some sort-of-TSV files for the values. It was able to not…

It's not just edge cases where it fails. It fails all the time at all kinds of things.

I've been using chatgpt in my work, but I have to essentially know the answer it's going to give me because I have to catch all of its mistakes. It really, really nice for certain kinds of drudge work.

Using northwind is probably not a good thing to use to evaluate chatgpt's general capability. It is very commonly used for examples of almost anything database-related, which means it's extremely well represented in chatgpt's training data. Chatgpt probably doesn't need to generalize or understand much of anything about northwind to answer your questions in terms of it. You need to try it on wacky problems specific to you.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#298

Post by alex201 » Fri, Mar 17, 2023, 1:37 PM UTC

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Me feeling philosophical. To me, solving a problem is living. No machine will do living, unless we know what living is, so that we can try to bake it into a machine. We can use ChatGPT to take blind, systematic steps in the way of solving a problem, but never to 'solve' the problem. For solving is living.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#299

Post by zitterbewegung » Fri, Mar 17, 2023, 1:38 PM UTC

I had some Matlab code and I wanted it to be ported to numpy. I couldn't get it running on Python and it wasn't doing it correctly on chatGPT.

On the other hand it could regurgitate code to use fastapi and transformers and it looked correct to me.

When you think about it this is very very similar to a stack exchange or google search but with a much different way to search and it can synthesize simple things which limits the complexity of what you want to do. So I don't really think it can write code but it can surely get you something that gets you 50% there.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#300

Post by gwoolhurme » Fri, Mar 17, 2023, 1:38 PM UTC

I guess my genuine question is to the people who are saying this is a big deal and it will take our jobs. I am a bit lucky to where at the moment I am working in a "novel" field. Lets say though for the sake of argument that AI does come for SWE jobs. To be honest? I don't know what to do in that case, I have no backup plan, not enough to retire. The country I've lived in for 9 years is still through a work visa (hopefully at least that changes soon). I am just comfortable enough with my salary. If all that is pulled from under me, I lose my job tomorrow, I lose my profession, my visa, my home. I honestly would like to ask to the people who say this is and will come for us soon. Well OK, but what is your advice for someone like me? It's true society doesn't owe me anything, nobody does. So it is also just an answer that some of us will be dropped by the wayside. That's what happened before. Just curious what anyone's advice would be assuming they are right and it does take our jobs.