Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

241–250 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#241

Post by sgarland »

There are plenty of edge cases where it fails. However, the one thing that made me think it actually knows (for certain definitions of the word "knows") what it's doing was asking it to re-write a non-trivial SQL query into equivalent relational algebra. I created a simplified schema from Northwind [0], gave it the CREATE TABLE statements for tables, and then some sort-of-TSV files for the values.

It was able to not only produce reasonable outputs from various queries, but also to produce valid relational algebra for them. To me, that shows a fairly deep level of understanding of the underlying concepts.

[0]: https://en.wikiversity.org/wiki/Database_Examples/Northwind

Re: Testing GPT 4's code-writing capabilities with some real world problems

#242

Post by iambateman »

The replies here are defensive and I think misguided. Yes, a programming job doesn’t solely require typing code. But the reason we have well-paid programming jobs is because there is a specialized skillset required to understand a body of syntax that takes several years to really grasp.

The difference is that writing a well-formed prompt is massively easier to teach than writing the code itself, for similar results. That’s not to say prompt writing requires no skill - it will certainly need understanding of systems and the scope of what is possible within a language. Asking GPT-4 to write a jQuery plug-in that generates an original Bob Dylan song will probably just not work.

But it is wildly easier to teach someone what is possible with JavasScript and let them spend a month watching someone prompt the system and let them go from there.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#243

Post by rjmunro »

syntheweave wrote: post #162
scg wrote: post #19

As a human programmer I didn't quite understand the problem statement until I read the whole article and the tests. I believe the goal is to find a path with the fewest possible "fire" cells and the minimum cost as a tie breaker. The cost of a path is the sum of its cells' cost and it can't be greater than 5. If I understood the assignment correctly, I don't think the problem statement is equivalent to what's include…

The problem is, indeed, that Mr. Glaiel did not know the category of problem he was dealing with. A correct statement would be: "Given a solution set containing both the shortest path through fire and the shortest path avoiding fire, select the solution that fits within six tiles of movement, preferring the solution that avoids fire where possible." It's a constraint optimization problem in disguise: generate a solut…

Can you not do A* where it normally costs one point per tile, 10 points for a fire tile but 1000000 points for > 6 tiles, so you never explore the > 6 options unless you have run out of shorter routes?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#244

Post by jonnycat »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

This is exactly right. Actually writing the kind of code that ChatGPT produces is a vanishingly small part of my job. And there's a ton more specialized scenarios to deal with, like next week when the third-party company is breaking the contract in .

If you want to hire a developer to implement qsort or whatever, ChatGPT has them beat hands-down. If you want to build a product and solve business problems, there's way more involved.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#245

Post by gonzo41 »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

It's good at tech doco. ie take my dot points and make prose. Great for copy, great for having that nice corporate drone voice.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#246

Post by SketchySeaBeast »

IKLOL wrote: post #229
quitit wrote: post #200

Earlier quoted context omitted.

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

I don't know if that's going to be the case - companies can never have enough software and typically they just go until the budget runs out as the software is never "done". I think being able to build 5x the software with one dev means that each company is going to build 5x the software.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#247

Post by jackcviers3 »

Quarrelsome wrote: post #228
staminade wrote: post #211

Earlier quoted context omitted.

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills. What they failed to predict was that…

That's an extremely interesting thought. Perhaps we will see organisations in the future structure themselves more like a suite of unit tests. Instead of getting a developer or software house to plug a specific need in their org and being entirely outside of the development process: they will reflect the development process organisationally to ensure they catch any problems with the current output and just feed the b…

I firmly believe that we should be studying model analysis and using that to create a field of peompt engineering. Both from a security standpoint and a productivity standpoint.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#248

Post by simonh »

hnfong wrote: post #88
matwood wrote: post #67

Earlier quoted context omitted.

> you can coerce it to do surprising things Yeah, I've gotten it to write some pretty decent contracts, but only because I have written said contracts and can ask it all the right questions/prod it into adding what I need.

And the next iteration of ChatGPT will likely generate a better contract because your interaction with it will be used as training data

It will only be a better contract if it better meets the actual real world requirement, rather than the requirement it was literally given. That means inferring information that isn't there. Now that is possible for common situations but not reliably even for a human expert. The way a human lawyer would do that is by interrogating the client and asking about likely additional requirement and iterating on them to agree on a final requirement. Current LLM architectures aren't capable of this, and it's hard to see how they could be re-architected to do it because it's a very different kind of task, but if that ever is achieved then there may be no limit to what such a system can do.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#249

Post by gonzo41 »

IKLOL wrote: post #229
quitit wrote: post #200

Earlier quoted context omitted.

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

You're not thinking that companies will not just spin up 5 projects, so 5 programmers produce a notional 25 person's worth of work. And hey, maybe they sack the expensive old guy and the AI spend isn't so great. Seems like a win.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#250

Post by yanis_t »

I find it interesting that many people took a defensive position towards AI. For many the discurs seems to be "will this AI thing eventually replace me and kick out of me job".

For me it's more like will that AI thing make me a 10x developer? And the answer I'm leaning for is yes.

I use copilot which saves me time googling and reading stackoverflow. I use chatgpt for writing tests to my code (which I hate to do myself). Sometimes I use it to ping-pong ideas, and eventually set on a good solution to a problem.

It saves me tons of time I use to complete other tasks (or spend with my family).

Post reply on HN