Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

321–330 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#321

Post by danparsonson »

iambateman wrote: post #242

The replies here are defensive and I think misguided. Yes, a programming job doesn’t solely require typing code. But the reason we have well-paid programming jobs is because there is a specialized skillset required to understand a body of syntax that takes several years to really grasp. The difference is that writing a well-formed prompt is massively easier to teach than writing the code itself, for similar results.…

The most challenging part of software development (and the reason we have well-paying jobs) is not understanding syntax, it's analysing and abstracting a problem domain into a set of cleanly separated modules that interact to solve those problems. That being the case then, actually none of us is getting replaced by GPT-n any time soon - 'prompt engineering' will just become the new Javascript, only more abstract; just another tool in the toolbox. Hopefully :-)

Re: Testing GPT 4's code-writing capabilities with some real world problems

#322

Post by marstall »

staminade wrote: post #211
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills. What they failed to predict was that…

Iteration and integration, the tasks which take most of at least my time as a developer, could fade significantly - or become automated themselves.

We won't have understanding of our code, similar to how we don't understand the machine language being generated by our compilers now.

We will be using our intuition about GPT to bring into being fully designed and integrated systems with 10 paragraphs at the prompt.

Which could in the end greatly increase the influence of a programmer in a given organization. Our role will be a softer, almost cyborgian one.

But this will indeed require the destruction of all that came before it. Questions like "but does it work with this 3rd party API, or this platform?" must become irrelevant for this future to happen.

A bit similar to how the web destroyed mainframe, perhaps, by first creating its own compelling world, then making the mountain come to it.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#323

Post by woeirua »

lionkor wrote: post #240
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

For me, its mostly that I have used GPT-3.5 a little for programming C++, and I wasnt impressed. For one, it made horrible, glaring mistakes (like defining extern functions which dont exist, using functions which are specific to a platform im not using, etc.), stuff beginners would do. It also decided to sneak in little issues, such as off-by-one errors (calling write() with a buffer and a size that is off by one in…

Two points: GPT4 is significantly better in this regard, and you should be concerned about the rate of progress more than it’s actual capabilities today.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#324

Post by lordnacho »

shp0ngle wrote: post #222
it_citizen wrote: post #3

To be fair, in 15 years writing code, I have spent the vast majority of my time working on minor variations of problems already solved. I am not a fancy developer coming up with new algorithms. I make sign up flows, on-boarding flows, paginated lists, integrations to other apis. And I definitely feel that my job might be threatened by LLMs.

Your job is already threatened by cheap outsourcing. However, the risk with cheap outsourcing is exactly the same as with LLMs - you get what you pay for, and you need to constantly check if it's really doing what it's supposed to be doing.

This is the thing. You need to know what you're doing to know whether you got the thing you wanted. Thus it's still a tool, rather than an expert.

By contrast, you don't know what your pilot or your surgeon is up to, you have to trust their decisions.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#325

Post by roflyear »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Not just with external parties. This dance happens all over.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#326

Post by marcyb5st »

Ah, in my experiments it writes like > 90% of the code correctly.

I got the best results with prompts like:

Given the following python code:

``` Few hundreds python loc here ```

Write tests for the function name_of_function maximizing coverage.

The function in this example had a bit of read/dumps from disk and everything. The code returned correctly created mocks, set up setup and teardown methods and came up with 4 test cases. I only needed to fix the imports, but that's because I just dumped python code without preserving the file structure.

I am amazed how fast these models are evolving.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#327

Post by vsareto »

sequoia wrote: post #290

One other question: can GPT-4 reliably modify code? In A Philosophy of Software Design the author points out that code is written once and modified possibly dozens of times, so ease of maintainability/reading is more important than ease of writing. I wonder whether a) AI can reliably modify code and b) whether AI can reliably write code that is able to be easily modified by humans. If AI starts spitting out machine c…

It can do edits, yes. You can also generally specify what language to use so it shouldn't jump from C++ to assembly unless you tell it to.

The bigger edits (refactoring things across an entire project) is out of reach because of the token limit. You could do it piece-meal through ChatGPT but that seems more tedious than it's worth.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#328

Post by irobeth »

> I think ChatGPT is just kind of bullshitting at this point. It doesn’t have an answer, and cannot think of one, so it’s just making shit up at this point [...] But instead it’s [overconfident] in its own capabilities, and just makes shit up. It’s the same problem it has with plenty of other fields

If anything, the article demonstrates it can write code, but it can't thoroughly reason about problems it hasn't been trained on

So when saying something like "Its possible that similar problems to that have shown up in its training set." as a way to dismiss any scintilla of 'intelligence', how many of these articles reduce to a critique e.g. "Can a Middle Schooler actually understand dynamic programming?"

Like, what is the actual conclusion? That a software model with O(N) parameters isn't as good as a biological model with O(N^N) paremeters? That artisans need to understand the limits of their tools?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#329

Post by coding123 »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

So it sounds like AI researchers should focus on replacing both sides - sounds like it would be much more efficient. (tic)

Re: Testing GPT 4's code-writing capabilities with some real world problems

#330

Post by woeirua »

noitpmeder wrote: post #76
jacobedawson wrote: post #16

Earlier quoted context omitted.

I think most people see where the puck is going, and even where it is right now is very impressive. It's not hard to see that it will be likely less than 5 years before it will be able to do what you did on day 2, and much more, at a tiny fraction of the cost, with no downtime, no attitude problems, sick days, etc. The things you mentioned (taking on board feedback, testing across devices, iterating on solutions) doe…

Personally I think it's 99% hype. The current iteration and architecture of these systems means they will never be at the level where they can actually replace a programmer. The best they will ever get is barfing up snippets for a programmer (who still needs the industry-specific knowledge to VERIFY the snippets). Additionally, "the rate of increase in capabilities" is very much a false flag. Past performance (especi…

The fact that we’re even having this conversation should concern you. Move the goalposts if you want, but the rate of progress should concern you.
Post reply on HN