Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

251–260 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#251
post #200
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

[deleted]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#252

There are plenty of edge cases where it fails. However, the one thing that made me think it actually knows (for certain definitions of the word "knows") what it's doing was asking it to re-write a non-trivial SQL query into equivalent relational algebra. I created a simplified schema from Northwind [0], gave it the CREATE TABLE statements for tables, and then some sort-of-TSV files for the values. It was able to not…

Could you share the results?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#253
post #229
post #200

Earlier quoted context omitted.

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

If increased productivity equaled job loss, there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950.

A lot of projects today are not even greenlit because they would be too expensive to make. For instance, there are a lot processes in almost every country that require you to file paper forms, even though we have had web forms and databases for 25 years. This goes even for rich countries. If one programmer is able to do the job of 5 others, we will probably have 5 times as many systems to tend to now that businesses and institutions can afford them.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#254
post #229
post #200

Earlier quoted context omitted.

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

Wouldn’t this actually create more demand for programmers? More businesses can exist that have a need for software. To date every advance in programming efficiency has resulted in more tech, not less.

Additionally there’s the math behind it. If Company A fires 50% of their staff because AI lets the remaining 50% work at twice the productivity then how will they compete with Company B that keeps their staff and now gets 200% efficiency?

The math is in favor of getting more, not fewer, developers.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#255
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills. What they failed to predict was that…

Indeed, a programmer's job feels artisan oftentimes. I think a reason for it is that projects are often ill defined from day one. They are defined by people who do not know enough about the system to set good requirements. The engineer works both bottom-up from the existing primitives of an existing system and top-down from requirements and tries to solve the puzzle where both approaches meet. Such work is very hard to automate. I believe there is a very big opportunity for AI to be used in a workflow such that inconsistencies between requirements, on all levels, and actual systems become clear a lot faster. The feedback loop to the requirements will be shorter, prototypes will exist sooner. The current workflow has little space for an AI worker. I believe this will change and have a major impact on the art of developing products. The AI programmer is still at its infancy, let's talk again 5 years from now.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#256
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

This is exactly why I'm so tired of these "can AI write code" think pieces. I assume people writing this crap aren't actual developers. Maybe its management fan-fiction.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#257
post #10
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

Have you seen this video? https://www.youtube.com/watch?v=9QZlQMpNk-M I think the author is onto something – while AI might not be able to program per se, it can certainly be handed a code snippet and then use its huge corpus of Internet Learning™ to tell you things about it, code that looks like it, and ways (people on the Internet think) it might be solved better. In that sense, it isn't replacing the programmer; i…

This is how I use it. It is the best rubber duck money can buy.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#258

Earlier quoted context omitted.

> Makes me feel like GPT is marketing to the incompetent or something. Absolutely. The common constant I can see in people who are really blown away by GPT's performance at [task] is that they are bad at [task]. Programmers who describe their job as copying from StackOverflow think it's great at coding. People who don't read fiction think it's great at writing fiction, and so on.

That's not accurate at all for me. I'm not impressed with the output when rating it against humans who are good at said task. It's obviously not as good as those who are good in that relevant field. But it's certainly far better than humans who are not skilled at those tasks, and that is what I find to be very impressive. I just didn't realise that these models could be this good, and they're not even as good as they…

I think we're talking at cross-purposes here. I'm not underwhelmed by it being not as good as good humans at [task]; that's not a surprise.

I'm saying that most people I see who are impressed by it for any given task are people who are not well-equipped to judge performance on that task.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#259
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills. What they failed to predict was that…

For that we need to make the AI deterministic or at least shape processes on specific error rates which are probably higher than that of the average smart human.

We had to do that for the industrial approach and it wasn't a simple, fast or intuitive process.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#260
Here's another view. If you're a music composer, you hear the music in your head. But in order to get it out, you need to play and record musical instruments, learn to sign, learn to produce, etc. What if you had a device that takes music from your brain and gives you an mp3 file?

That's what I think AI is doing for developers here.

Post reply on HN