Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

401–410 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#401

Post by shagie »

anileated wrote: post #188
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Why aren’t you thinking rather that instead of talking to you, “they” would already be talking to the LLM (likely trained on your code, among other data)—while you get 0 total billable workload in the first place?

The issue being that neither they, nor the LLM has the proper model for the problem domain and so don't ask the right questions when trying to extract business requirements.

Additionally, this is "stateless" to an extent. There's no architectural plan for how it should work when you have an LLM do it. "We're using X now but there are plans to switch to Y in some number of months." This could lead to making an abstraction layer for X and Y so that when the switchover happens there is less work to be done - but that requires forward looking design.

If "they" only describe the happy path, there is no one to ask about all the unhappy paths, edge cases and corner cases where the naive implementation of the problem description will fail.

Hypothetically, yea, "they" could be trained to think through every possible way the generated code could go wrong and describe how the code should work in that situation in a way that isn't contradictory... but that remains an unsolved problem that has nagged developers for decades. Switching to an LLM doesn't resolve that problem.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#402

Post by ryandrake »

lebuffon wrote: post #357
zeroonetwothree wrote: post #342

Earlier quoted context omitted.

How much more efficient is programming in Python with modern tools vs assembly? I would bet more than 5x

That's an interesting case and use of the word efficient. Developer time reduced by 5X CPU time increased by 500x When CPU cycles are cheap it's a logical tradeoff but it still grates against my sensibilities.

We should stop thinking about the trade-off as between developer time and CPU time. The CPU is an inanimate object which doesn't mind pushing around more electrons. What we're really throwing under the bus when we optimize for developer time is the customer's time.

If an inefficiency saves a developer 10 days of work, but results in an operation taking 100 milliseconds longer, and you have 50M customers who do that operation just one time, then you've wasted ~58 customer-days.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#403

Post by awb »

How are people generating multiple files for larger applications?

I gave it a prompt and asked it to respond with a list of file names required to build the app. Then when I prompted a file name it should print the code for that file along with a list of ungenerated file names. It got through two before it got confused.

I’m stuck with having it write one function at a time.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#404

Post by soperj »

IKLOL wrote: post #229
quitit wrote: post #200

Earlier quoted context omitted.

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

Yeah, but they would then just start 10 times the number of projects at once and watch the majority of them fail.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#405

Post by rootusrootus »

caleb-allen wrote: post #236
anonyfox wrote: post #195

Earlier quoted context omitted.

don't fall into this mental trap. you can get into recursion quite easily here, and figuring out what to prompt can start from simple general questions - and there is no need for a developer at all, aside from the current limitations of copy/paste/run workflow has to be done manually

It's astonishing to see the goalposts move so quickly. The cope of "well, okay, it can do that, but that's not even the hard part!" when just a year ago this entire product was almost unimaginable.

The realization that software development is only maybe 5% coding came way before GPT.

> The cope of

Whenever I see someone use 'cope' that way, I immediately think they're making a bad faith argument. Is that intentional?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#406

Post by eternalban »

sgarland wrote: post #215
Jensson wrote: post #135

Earlier quoted context omitted.

Quoting famous people sounds smart until you realize they just memorized a ton of trivia. These models have demonstrated that they don't learn logical models, instead they learn to generate text that looks logical at first glance but is nonsense.

I asked GPT-4 something fairly niche that I happen to know a fair amount about: to explain the concept of Xenon poisoning in a nuclear reactor. Other than skipping Te-135 being the initial fission product that starts the decay chain (and tbf, operationally it can be skipped since the half-life is 19 seconds), it got everything correct. I'm sure if I kept probing on smaller and smaller details it would eventually fail…

I couldn't get ChatGPT to do Wordle. I don't have access to GPT-4. See if it can do that.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#407

Post by importantbrian »

> After going in circles a few more times, I decided that was it. It got close. It seemed to understand the problem, but it could not actually properly solve it.

I had this same loop issue with Chat-GPT. I had something I wanted to do with asyncio in Python. That's not something I work with much so I thought I'd see if Chat-GPT could help me out. It was actually good at getting me up to speed on ansycio and which parts of the library to look at to solve my problem. It got pretty close, but it can't seem to solve edge cases at all. I got into this loop where I asked it to make a change and the code it output contained an error. I asked it to fix the error so it gave me a slightly modified version of the code prior to the change. So I asked it to make the change again and the code it spit out gave the same error again. I went through this loop a few times before I gave up.

Overall, it's cool to see the progress, but from what I can tell GPT-4 suffers from all the same issues Chat-GPT did. I think we're probably missing some fundamental advance and just continuing to scale the models isn't going to get us where we want to go.

My biggest concern with the current batch of LLMs is that we're in for Stackoverflow driven development on steroids. There's going to be a ton of code out there copy and pasted from LLMs with subtle or not so subtle bugs that we're going to have to spend a ton of time fixing.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#408

Post by irrational »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

This is also one of the reasons India taking over all of the programming work didn’t really happen. There are numerous issues (time zones, language, etc.) but business people not being able to document perfectly what they want to have built, considering all corner cases and paths, is a big one.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#409

Post by [deleted] »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

[deleted]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#410

Post by jasmer »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

ChatGPT will eventually help with debugging, suggestions, idioms, finding security flaws, recommending libraries, boilerplate, finding stuff etc..

Or maybe not ChatGPT but something like it.

Post reply on HN