Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

381–390 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#381

Post by trabant00 »

staminade wrote: post #211
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills. What they failed to predict was that…

Two problems with the analogy: The artisans fields where machines took over where hundreds and thousands of years old. We understood them very good. And maybe more importantly factory automation is deterministic and needs to be, as opposed to generative ML.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#382

Post by travisjungroth »

yodsanklai wrote: post #364
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

The problem is that many people here have an extreme point of view. It's either "this is going to make all develops jobless" or "it's useless, my job isn't that". I think It'll help with some tasks, which is always good to take. After all, people tweak their vim settings because they feel it makes them more productive.

This tech is so powerful, cutting so close to the last of human output that’s uncopied, advancing so fast, I have an incredibly hard time imagining it ending in a space that deserves a muted a reaction. It seems like it will either dramatically change society or fail. And I have quite a hard time imagining failure.

It seems on par with the Industrial Revolution, at least. Which, easy to forget, was a huge deal that totally changed society.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#383

Post by deelly »

HervalFreire wrote: post #306

Guys all of this happened within a couple months. If you guys are confident about the entity as it is right now not taking over your job, what if I double the accuracy of gpt output? What if I double it again? Then again? And again? And again? You guys realize this is what's coming right? This thing literally is a baby as of now.

Is it true that GPT4 failure rate is .5 of GPT3?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#384

Post by anileated »

slfnflctd wrote: post #375
anileated wrote: post #188

Earlier quoted context omitted.

Why aren’t you thinking rather that instead of talking to you, “they” would already be talking to the LLM (likely trained on your code, among other data)—while you get 0 total billable workload in the first place?

My intuition here is that it's because people don't always say what they mean, or know how to describe what they want. I've been working on a database migration recently, and I look forward to the rare moments when I get to write queries and analyze actual data. The vast majority of my billable hours are spent trying to tease out the client's needs by going over the same ground multiple times, because their answers k…

> Especially when a higher-ranking person comes back from vacation and says "no, everything you nailed down in the last two weeks is completely wrong".

Yes, and at some point this high-ranking person is fed up with this now-inefficient use of time and money enough that they will just sort this out using an LLM tuned to handle this situation better if not today then tomorrow.

Maybe they will pay someone to coach them for a week how to “talk” to LLM, but other than that the one who gets paid in the end is OAI/MS.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#385

Post by IsaacL »

gwoolhurme wrote: post #300

I guess my genuine question is to the people who are saying this is a big deal and it will take our jobs. I am a bit lucky to where at the moment I am working in a "novel" field. Lets say though for the sake of argument that AI does come for SWE jobs. To be honest? I don't know what to do in that case, I have no backup plan, not enough to retire. The country I've lived in for 9 years is still through a work visa (hop…

Check out my comments higher up in the thread (eg https://news.ycombinator.com/item?id=35197613), I really do believe that GPT4+ will be primarily useful as augmenters for capable and dedicated engineers, rather than replacements. It's like a very eager and brilliant junior dev that can solve many problems that you throw at it, but still needs hand-holding, error-checking, and someone who knows how to piece the actual system together.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#386

Post by airstrike »

dimitrios1 wrote: post #370
coldtea wrote: post #354

Earlier quoted context omitted.

That's irrelevant though, because actual programming is very much about computers, and about writing things the majority of which have already been formalized by computer scientists, and gluing code and programs inti pipelines...

I just adore the hubris on full display here to say what one of the world's best computer science educators has to say is irrelevant.

I think the parent's point is that even highly inspired wisdom can be irrelevant if applied to all situations

More like "it doesn't apply" than "it's inherently invaluable"

Also your comment is technically an appeal to authority

Re: Testing GPT 4's code-writing capabilities with some real world problems

#387

Post by gsamuelhays »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

I run into this a lot myself. In our paper (https://arxiv.org/pdf/2303.07839.pdf) we specify a 'Specification Disambiguation' pattern that attempts to address this very thing.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#388

Post by antegamisou »

lionkor wrote: post #240
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

For me, its mostly that I have used GPT-3.5 a little for programming C++, and I wasnt impressed. For one, it made horrible, glaring mistakes (like defining extern functions which dont exist, using functions which are specific to a platform im not using, etc.), stuff beginners would do. It also decided to sneak in little issues, such as off-by-one errors (calling write() with a buffer and a size that is off by one in…

My thoughts exactly.

There are so many non-CRUD complex disciplines involving programming such as signal processing, robotics, control theory, scientific computation to name a few, the current version, at least, of GPT is not even close to being a good supplement, let alone a substitute.

But then I remember I'm on HN where the technical pinnacle of programming is Backend and DevOps.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#389

Post by underanalyzer »

scg wrote: post #19

As a human programmer I didn't quite understand the problem statement until I read the whole article and the tests. I believe the goal is to find a path with the fewest possible "fire" cells and the minimum cost as a tie breaker. The cost of a path is the sum of its cells' cost and it can't be greater than 5. If I understood the assignment correctly, I don't think the problem statement is equivalent to what's include…

Since this is the comment thread talking about the algorithm I'm gonna add my 2 cents here:

Here's the problem statement as far as I see it: Each tile has a number of move points to spend to go through it (1 for regular and 2 for water). Each tile also has a cost associated with it. Given a max number of move points find the lowest cost path between two tiles or return none if no such path exists.

I'm gonna say this is still modified dijkstra with a small twist. The fire has cost 1, the other tiles have cost 0. However instead of pathing on a 2d grid (x, y) we path on a 3d grid (x, y, moves). All "goal" tiles within (goal_x, goal_y, moves < total_move_points) have a 0 cost edge which brings them to the true goal node. The implementation difference is that the get neighbors function queries neighbors in later grid layers (x+..., y+..., moves + 1 or 2)

Re: Testing GPT 4's code-writing capabilities with some real world problems

#390

Post by Filligree »

swexbe wrote: post #336
pixl97 wrote: post #114

Earlier quoted context omitted.

Being that you and I are talking to each other by forming thoughts in meat that us causing other meat to move around tells me that digital thought still has a lot of scaling room ahead of it. Maybe superintelligence isn't possible for some reason, but the fact we exist should tell you that general intelligence is not magic.

Your brain is analog. Trying to emulate this with a digital computer is bound to come with some computational cost.

Good thing we aren’t trying, then. GPT works in a completely different fashion.
Post reply on HN