Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

571–580 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#571

Post by WinstonSmith84 »

lionkor wrote: post #549
WinstonSmith84 wrote: post #472

Earlier quoted context omitted.

GPT4 is very different from 3.5. I've asked it today to write some unit tests given the code of the class (~200 lines) and the methods I wanted to cover and it did that just perfectly. It put asserts where it made sense (without me asking to do it), and the unit test code was better written than some code I've seen written by (lazy) humans. It's not perfect sure and it's easy to get a bad response but give OpenAI a f…

what kind of unit tests are these? is it `check_eq(add(1, 2), 3)` or "check for possible exceptions, cover edge cases, test extremes of this super important db function"

It's Salesforce unit tests, written in Apex, actually a niche language, so it's surprising that even on such a language, it was so good. And no, the unit tests were much more complex than this. It involves creating records, querying data, some business logic runs and then the data is updated. The asserts are after, checking that the business logic performed correctly.

The bot created the whole unit test involving the creation of data with test fields, then queried the output results and put some asserts. That's more than 100 lines of code which were written by GPT4. A (good) Salesforce developer would need a good 30 minutes to write those, and the result would not have been better.

Again, I also have some counter examples were it made some mistakes, but this is really shocking how... a program... figured all this out.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#572

Post by alfalfasprout »

irrational wrote: post #408
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

This is also one of the reasons India taking over all of the programming work didn’t really happen. There are numerous issues (time zones, language, etc.) but business people not being able to document perfectly what they want to have built, considering all corner cases and paths, is a big one.

Ultimately the model that worked was "I have this tighly scoped project that no one really wants to work on that's completely self contained and is going to require a ton of manual work" and hiring contractors to implement your own design.

Otherwise if there's a lot of back and forth required or generating a design, forget it. Companies giant and small have tried it and eventually realized "F it" and gone back to in-house teams.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#573

Post by alfalfasprout »

geph2021 wrote: post #565
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

If we use GPT-X in our company, it will help us with 1% of our workload I think there are many such cases. Another one that comes to mind is adding features to a large/legacy code-base. Writing the new code/function is a small part of the work. The main part of the work is first understanding and agreeing on how/where to implement the changes, sometimes across multiple teams, and the implications/knock-on effects to…

And this is one of those things where it has to be done correctly by someone who knows what they're doing. It's not just "oh, I deployed a change and broke something i'll just revert it". Often it means major migrations, etc.

There's a disconnect between folks writing small CRUD or mobile apps for clients and folks that have to work on large codebases with lots of complex systems.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#574

Post by gnaritas99 »

BurningFrog wrote: post #413
vages wrote: post #253

Earlier quoted context omitted.

If increased productivity equaled job loss, there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950. A lot of projects today are not even greenlit because they would be too expensive to make. For instance, there are a lot processes in almost every country that require you to file paper forms, even though we have had web forms and…

Something like 97% of all jobs have been eliminated since the Industrial Revolution started ~250 years ago. And instead of the 97% unemployment our intuition tell us would happen, we're instead 30x as productive. The same will happen when/if AI makes writing software far more effective.

[dead]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#575

Post by alfalfasprout »

namelosw wrote: post #480
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Consider creating an AI stakeholder that speaks for the client. This approach would allow the client to provide input that is wordy or scattered, and the consultant could receive immediate responses by asking the AI model most questions. Better yet, they can ask in a just-in-time manner, which results in less waste and lower mental stress of collecting all possible critical information upfront. As the project progres…

There's major $$$, legal, and security ramifications for clients in many cases. Having an AI that can't properly deal in ambiguity and hallucinates an outright reckless idea 1% of the time is completely unacceptable.

Writing code, sure. A human ultimately reviews it. I suspect in the legal world a lot of legal writing can also be automated to some degree. But strategic decisions, designs, etc. very much need a human pulling the trigger.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#576

Post by gnaritas99 »

cguess wrote: post #439
coldtea wrote: post #344

Earlier quoted context omitted.

> If increased productivity equaled job loss there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950. The only reason it's not the case in this example is because computers at the time were a tiny early adopter niche, which massively multiplied and expanded to other areas. Like, only 1 in 10,000 businesses would have one in 1950,…

Have no doubt, we will find new places to put computers. In the 80's and even the 90's everyone said the same thing, "Why do I need a computer? I can do everything I do already without a problem?" Well, turns out with computers you could do 12 more things you can never considered. Consider the interoffice memo: it'd take what, 1-2 hours to get a document from one floor to another through the system? Cool, you can wor…

[dead]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#577

Post by weatherlite »

pewpew2020 wrote: post #47
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I feel it's the average programmer who is enamoured with GPT. The programmers working on hard stuff know that AI writing programs is still far away.

OK, most of us are average by definition, what's your point again? And - I think if it can replace my average ass it will sooner than you imagine be able to solve Linux Kernel bugs. I just don't see a huge difference in the computation required between CRUD and Kernel development: It's just statistics for the machine.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#578

Post by syntheweave »

rjmunro wrote: post #243
syntheweave wrote: post #162

Earlier quoted context omitted.

The problem is, indeed, that Mr. Glaiel did not know the category of problem he was dealing with. A correct statement would be: "Given a solution set containing both the shortest path through fire and the shortest path avoiding fire, select the solution that fits within six tiles of movement, preferring the solution that avoids fire where possible." It's a constraint optimization problem in disguise: generate a solut…

Can you not do A* where it normally costs one point per tile, 10 points for a fire tile but 1000000 points for > 6 tiles, so you never explore the > 6 options unless you have run out of shorter routes?

At a glance, it might be OK that way, and I would give it the gold star. It's just unintuitive to make the leap to "apply a cost to the total length of the path" as a way of expressing preferences among distinct categories of path.

Implementation of the categories as completely independent paths falls out of the clarified problem definition directly. It's really in nailing the specification that the problem is hard, i.e., even with GPT we're still programming.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#579

Post by weatherlite »

yucky wrote: post #169
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

> I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Equally fascinating is all of the "this is fine" posts from programmers suddenly realizing they are not the gods they once thought. But fret not, programming is not the first industry that has been automated into a shell of itself. Yes, the industr…

> most programmers will have to learn to do something else. Humans are resilient and will adapt.

Like what? Seriously, which cognitive occupation is safe then? I think if one wants to stop competing with the machines (who appear very close to becoming superior to humans by what you are saying), it's some kind of child care / social work job. We still don't want robots in those (for now. Eventually they may do those better than us as well).

Re: Testing GPT 4's code-writing capabilities with some real world problems

#580

Post by syntheweave »

TylerGlaiel wrote: post #557
tinco wrote: post #181

Earlier quoted context omitted.

By the time you've formulated the problem as: "Give me the shortest route with a cost of 5 or lower that doesn't go through fire, and if that doesn't exist, the shortest route with a cost or 5 or lower that goes through fire." Then you've basically formulated the algorithm as well. That's also precisely where one of the programmer's greatest challenges lies, to carefully translate and delineate the problem. I agree i…

thats also not a correct formulation of the problem, as it needs to minimize the number of fire tiles it passes through. which is where a lot of the complication comes from.

Solve once for each number of fire tiles n.
Post reply on HN