Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

421–430 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#421

Earlier quoted context omitted.

That's irrelevant though, because actual programming is very much about computers, and about writing things the majority of which have already been formalized by computer scientists, and gluing code and programs inti pipelines...

I just adore the hubris on full display here to say what one of the world's best computer science educators has to say is irrelevant.

It is indeed not particularly relevant to the emergence of AGI and its effect on our work.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#422
Watching GPT write code hurts my brain.

It starts at the first character, works forward one “token” at a time, and ends at the last character. Never moving back.

It feels like it knows where it’s going at the first character, even though it doesn’t.

It’s like it starts speaking a sentence and, by the time it’s done speaking, it’s written a syntactically correct Node.js application.

The way GPT communicates in English does seem similar to how humans communicate. The way GPT writes code doesn’t seem to come close to approximating how humans do - it’s an entirely different mechanism. Humans generally can’t write code without a cursor and backspace.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#423
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

sorry, is the debate here whether gpt can engage in a conversation with someone and respond using previous context? why would any of this present a challenge given its known abilities?

This was also my reaction to the comment. Go play out this type of conversation with ChatGPT. Does it actually do worse at this than at writing the code? Or is "but code generators can't do the requirements part" just a cached thought?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#424
post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

> Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors. That is already possible without AI and has been the case for a long time... the issue is nobody will stay to be that highly paid senior running entire projects because at that point you can just run your own shop and pocket the full profits.

It also causes a "single point of failure". If that senior gets hit by a bus then what? Can the company afford to bring in another senior that will take ~6 months to become productive?

I'm not disagreeing with you. Im thinking of going solo myself.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#425
post #229
post #200

Earlier quoted context omitted.

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

Fixed pie fallacy. More likely, there will be 5x as many apps / companies with just as many jobs available. And the cost of everything will go down, except truly scarce things like Manhattan real estate.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#426
People need to understand that AI doesn't think, doesn't have insight or intuition. AI just repeat patterns it saw in a huge database, but are not able to understand what is going on.

Nobody can really understand what's inside a trained neural network, and nobody is really looking.

No psychologist or neuro-scientist can really understand how a human brain, a mouse brain or even an ant brain or a fly brain even works, so don't expect computer scientists to have any insight about doing something relevant with just a small collection of sophisticated statistical methods.

AI is soon going to become the pseudo-scam status that bitcoin experienced.

ChatGPT is an improved search engine at best.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#427
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

> Basically, 99% of the effort is NOT WRITING CODE. It's all about communication with people, and problem solving. If we use GPT-X in our company, it will help us with 1% of our workload First, if you did have GPT-X (say GPT-10) in your company, there wouldn't be much back-and-forth communication either. Those parts would still be handled with GPT-X talking to another GPT-X in the other company. Even the requirements…

> Second, even if that's not the case, the part of doing the communication can be handled by non-programmers.

It can’t - only programmers will know which follow up questions to ask. GPT will be able to ask those questions before non-programmers will be able to.

Half the work is nitpicking on date formats or where some id comes from or if a certain field is optional, etc.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#428
post #19

As a human programmer I didn't quite understand the problem statement until I read the whole article and the tests. I believe the goal is to find a path with the fewest possible "fire" cells and the minimum cost as a tie breaker. The cost of a path is the sum of its cells' cost and it can't be greater than 5. If I understood the assignment correctly, I don't think the problem statement is equivalent to what's include…

Note that water tiles have cost 2, so the tile-crossing limit cannot be expressed simply as a maximum total cost.

Looking at the two examples in the paragraph after "And there’s a lot of complication to it beyond the simple cases too", I can't figure out how the movement value is defined, as I can only see 10 and 8 moves respectively, not the 14 and 10 movement value claimed in the following text (and only one water tile on each path.)

Re: Testing GPT 4's code-writing capabilities with some real world problems

#429
post #403

How are people generating multiple files for larger applications? I gave it a prompt and asked it to respond with a list of file names required to build the app. Then when I prompted a file name it should print the code for that file along with a list of ungenerated file names. It got through two before it got confused. I’m stuck with having it write one function at a time.

> I’m stuck with having it write one function at a time.

Because thats the most it can do. Claims that it can write code are getting quieter. People made wild claims on reddit but when prompted to share their code they either went mute or the code was hilariously amateurish and limited.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#430

Earlier quoted context omitted.

You know the shoeshine boy indicator story [1]? That's the vibe I've been getting with ChatGPT lately, I have multiple old high school teachers with 0 computational background who hardly ever spoke about ML before posting regularly on LinkedIn about ChatGPT the last couple months. The part that gets me is they aren't just aimlessly making posts, they're getting involved with webinars targeted at educators in their re…

I can see why educators are getting involved. They are seeing it change how their students do work in real time. I grew up and using a pocket calculator was verboten. You just did not do it. You better learn and memorize all of that stuff. Spin on 10 years after me and all kids have them now. If you have the right app on your phone the thing will OCR the problem and auto solve it and show you the steps. ChatGPT and i…

Yeah there is a broader issue here for sure. I don't think that means within a few weeks of the launch of the pocket calculator there should have been dozens of random talks from people that have no understanding of what a pocket calculator does trying to pitch a curriculum based around it.

These posts ooze hype bullshit, not nuanced talk about pros and cons and how it will affect education. ChatGPT should be thought of as a writing tool and perhaps an information source on very surface level topics. Trying to teach an advanced research course at a top school with ChatGPT heavily involved is a terrible idea on the other hand.

I have tested asking it about a number of specific topics in biology research/asking questions about particular papers, and it gives horrible answers the majority of the time. If someone submitted that as a paper to me I'd give it a bad grade because it is dumb, I wouldn't need to know if it were ChatGPT or not. I would be alarmed if my kid's teacher went from organically teaching how to dissect a scientific paper to suggesting that a major part of the curriculum can be replaced with talking to GPT.

I've seen articles about teachers taking the other extreme against ChatGPT too, but I haven't personally seen anything that was a realistic take on what LLMs can do. Maybe it boils down again to disagreement on "where the puck is going" but to me most of the hype is making ridiculous assumptions about what is imminent meanwhile ignoring the things worth discussing now.

Which sounds a lot like bubbles. The dot com crash didn't mean the internet was a bad idea or that it wasn't worth discussing at that time.

Post reply on HN