Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

311–320 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#311

Post by anonzzzies »

oblio wrote: post #264
anonzzzies wrote: post #164

Earlier quoted context omitted.

> Please elaborate. I simply know firsthand (i'm old, i have manager, cto, ceo friends who I go golf and play squash with) that people in data entry and programming have been let go in the past weeks because 1 person could take over their work using the gtp/chatgpt api's and do their work faster with less errors. I am recommending the same in my company as a lot of my colleagues are doing nothing anymore as the skill…

How does this work from an IP perspective? Aren't you feeding OpenAI and Microsoft with your confidential information? Also, what license is the code under when you get it back?

I take care not to feed it secrets ; this is just boring ERP stuff without the magic numbers (they are not needed to create or test; we use test data normally as well, as we cannot give that to outsourcing companies either, so there is no difference in work).

Re: Testing GPT 4's code-writing capabilities with some real world problems

#312

Post by cableshaft »

caleb-allen wrote: post #236
anonyfox wrote: post #195

Earlier quoted context omitted.

don't fall into this mental trap. you can get into recursion quite easily here, and figuring out what to prompt can start from simple general questions - and there is no need for a developer at all, aside from the current limitations of copy/paste/run workflow has to be done manually

It's astonishing to see the goalposts move so quickly. The cope of "well, okay, it can do that, but that's not even the hard part!" when just a year ago this entire product was almost unimaginable.

I'm impressed. But I want it to be better, so I don't have to spend so much time coding and can trust it to give me good code so I can offload a lot of my efforts on to it. Right now I have to check and verify or go back and forth with prompts so much I almost would have been better off writing the damn code in the first place.

Meanwhile I've got a backlog of about 100 personal projects I've never gotten around to making (especially creating digital versions of my board game designs, of which I have about 70 at this point) that I just haven't had the energy to work on myself, that I'd love to be able to just upload the rules document for it and get some sort of working game spit out the other end, or at least a good chunk of it, and I can take it from there.

And then I can test a new rule by just rewriting the rules document, as opposed to having to manually code every rule change into the game.

I don't think I'd run out of things for it to make if it was as good at complex software generation as A.I. is right now with art generation.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#313

Post by numbsafari »

Hey GitHub / Microsoft / OpenAI

How about this prompt:

I have a web page where customers see their invoice due. When they enter their credit card information, sometimes the page just refreshes and doesn't show any kind of error information whatsoever, but the invoice remains unpaid. This has been going on FOR YEARS NOW. Can you write some code to fix this as we have been busy laying off all the umans.

Oh, or this one:

I have this page called "Pull Reqeuest", at the bottom there is a button that says "Comment" and right next to it is a button that says "Close this PR". We probably shouldn't have a button that performs a destructive action immediately next to the most common button on the page. This has also been going on for years, but, you know, no umans.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#314

Post by korijn »

IKLOL wrote: post #229
quitit wrote: post #200

Earlier quoted context omitted.

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

If as GP says AI could automate 1% of a single programmer's job (the boring part where you write code), then how on earth can you derive that a single programmer could do the job of 5 with AI? It's completely illogical.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#315

Post by vidarh »

danuker wrote: post #282
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

We could have this already. - Have a cron job that checks email from a certain sender (CRM?). - Instruct an OpenAI API session to say a magic word to reply to an e-mail: "To reply, say REPLY_123123_123123 followed by your message." - Pipe received email and decoded attachments as "We received the following e-mail: " to the OpenAI API. Make it a 1-click action to check if there is a reply and confirm sending the messa…

I just had ChatGPT write me a JMAP client (for Fastmail) that'd create a draft. Then I asked it to:

"Write an OpenAI API client that would take "new_message" from the above, feed it to the API with a prompt that asks the model to either indicate that a reply is needed by outputting the string "REPLY_123123_123123" and the message to send, or give a summary. If a reply is needed, create a draft with the suggested response in the Draft mailbox."

It truncated the "REPLY_123123_123123" bit to "REPLY_", and the prompt it suggested was entirely unusuable, but the rest was fine.

I tried a couple of times to get it to generate a better prompt, but that was interestingly tricky - it kept woefully underspecifying the prompts. Presumably it has seen few examples of LLM prompts and results in its training data so far.

But overall it got close enough that I'm tempted to hook this up to my actual mailbox.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#316

Post by anonzzzies »

suslik wrote: post #233
DaiPlusPlus wrote: post #154

Earlier quoted context omitted.

> I personally know about 10 people who have been fired or assigned to a completely different position because gpt did a faster and better or equal job Please elaborate. And, if true, this would be a major news story that Vox or any number of major newspapers would love to write about - so have you approached the media about this? If not, why not?

Manager here. Did not fire anyone (and hope not to), but I am starting to look into implementing GPT-4 as a part of our development practice and, potentially, reduce future hiring. This might have a positive impact on our engineering budgets, which are always tight. Many of my colleagues are doing the same - this has been a cooler topic with other managers for weeks now (strange how progress is now measured in weeks)…

If your engineers are actually engineering then yes, really by all means keep them and integrate GPT in workflows as is suitable.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#317

Post by Havoc »

Not sure it matters? If the majority of coding is gluing things together and it can replace that then you've suddenly got 10x as many coders gunning for the remaining gigs that have hard problems.

Good for whoever comes out on top, but not sustainable from a societal perspective

Re: Testing GPT 4's code-writing capabilities with some real world problems

#318

Post by globalise83 »

staminade wrote: post #211
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills. What they failed to predict was that…

"How far can you get if you put prompt-generated code at the heart of the workflow and make everything else subservient to it"

OK, challenge accepted - will go down your suggested route - thanks :)

Re: Testing GPT 4's code-writing capabilities with some real world problems

#319

Post by nonethewiser »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Sure, but in fairness to the original post, it's about whether Chat GPT can code. Not replace software engineers.

And in your scenario where chatGPT can code but someone needs to gather requirements, it still doesn't necessitate software engineers. I'm not worried about that personally but I don't think the "communicating between stakeholders" skill is such a big moat for the software engineering profession.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#320

Post by segmondy »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Yes it can write code, some demoed developing "Doom", ray tracing/ray casting with GPT-4 the very first day it came out and it was impressive. Programmers would still program, but the program will no longer be code but will be "GPT prompts". I suspect with time tho, we won't need to write amazing prompts, you ask GPT for a solution, it will then figure out the edge cases by asking you questions. If it's 2 people, it would query you both and resolve your conflicts. Programmers will be replaced with AI. We need to get over it, the question we should be asking is, what's next?
Post reply on HN