Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

161–170 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#161

This is one of the best analyses of gpt4 Ive read so far. Besides potentially including the visual aspect, I wonder if part of the reason it has trouble with harder problems is that it’s been tuned/prompted in a suboptimal way. The advertised used case mostly is “write down the solution for this problem”, but for novel problems it does much better when it’s given the chance to reason through it before trying to write…

So what is a software company going to do when people can use their own products to replace them ?

It's a slippery slope for M$. If ChatGPT 15 can just build MS Outlook from looking at photos of the UI, design a hololens, or tell us the secrets of how their Chat bots work, not sure how much future they're going to have as a company?

What I can see being the new thing is "innovation". People building useful solutions that the LLMs don't yet know about.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#162
post #19

As a human programmer I didn't quite understand the problem statement until I read the whole article and the tests. I believe the goal is to find a path with the fewest possible "fire" cells and the minimum cost as a tie breaker. The cost of a path is the sum of its cells' cost and it can't be greater than 5. If I understood the assignment correctly, I don't think the problem statement is equivalent to what's include…

The problem is, indeed, that Mr. Glaiel did not know the category of problem he was dealing with.

A correct statement would be: "Given a solution set containing both the shortest path through fire and the shortest path avoiding fire, select the solution that fits within six tiles of movement, preferring the solution that avoids fire where possible."

It's a constraint optimization problem in disguise: generate a solution set, then filter and rank the set to return a canonical result. That describes most of the interesting problems in gameplay code: collision and physics can use that framing, and so can most things called "AI". They just all have been optimized to the point of obscuring the general case, so when a gamedev first encounters each they seem like unrelated things.

The specific reason why it seems confusing in this case is because while pathfinding algorithms are also a form of constraint optimization, they address the problem with iterative node exploration rather than brute forcing all solutions. And you can, if you are really enterprising, devise a way of beefing up A* to first explore one solution, then backtracking to try the other. And it might be a bit faster, but you are really working for the paycheck that day when the obvious thing is to run the basic A* algorithm twice with different configuration steps. You explore some redundant nodes, but you do it with less code.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#163
A great article with a practical example of a programmer using GPT to solve a problem it hasn't seen in its training data. It gives plausible but incorrect answers and the user isn't able to prompt it to correct them.

It seems likely that a understanding of when NOT to use an LLM is a new skill programmers are going to want to learn in order to use their time efficiently.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#164

Earlier quoted context omitted.

HN echo chamber. Generally the better/best programmers hang around here; your day 2 was probably better than many coders hope to achieve in their whole lives. It is replacing people already; I personally know about 10 people who have been fired or assigned to a completely different position because gpt did a faster and better or equal job. So ‘not anyone’ is simply nonsense; I am only one person, there will be many m…

> I personally know about 10 people who have been fired or assigned to a completely different position because gpt did a faster and better or equal job Please elaborate. And, if true, this would be a major news story that Vox or any number of major newspapers would love to write about - so have you approached the media about this? If not, why not?

> Please elaborate.

I simply know firsthand (i'm old, i have manager, cto, ceo friends who I go golf and play squash with) that people in data entry and programming have been let go in the past weeks because 1 person could take over their work using the gtp/chatgpt api's and do their work faster with less errors. I am recommending the same in my company as a lot of my colleagues are doing nothing anymore as the skilled seniors are doing it themselves with gpt now as it's faster, less communication etc. We feed jira issues into gpt and it generates code; we review and refine or fix ourselves. It works much much faster and with better results. Most things most of us do all day is integrating ancient API's of partners and so mapping xml/soap/... api's to our json schema's. With chatgpt that's really fast and mostly painless; it even renames the properties that need to be changed to our enums properly. With humans this is a painful and slow process, especially with people who are fast and loose (broken education seems to made many of those graduate just by cheer production speed & volume instead of quality; gpt can do that better too...).

> so have you approached the media about this? If not, why not?

Why would I do that? Even anonymous, it doesn't seem to make much sense for me to do that. Anyway; that'll come soon enough as it will be common soon.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#165

Earlier quoted context omitted.

Personally I think it's 99% hype. The current iteration and architecture of these systems means they will never be at the level where they can actually replace a programmer. The best they will ever get is barfing up snippets for a programmer (who still needs the industry-specific knowledge to VERIFY the snippets). Additionally, "the rate of increase in capabilities" is very much a false flag. Past performance (especi…

But obviously increasing avg and sr programmer productivity by many x implies many fewer jobs for programmers. That’s the point we’re interested in.

Is that obvious? The history of programming has been full of things that have at least claimed to increase programmer productivity, everything from high level languages to smart refactoring IDEs, debuggers, PaaS, etc and in all that time the trend has been towards more jobs for programmers not fewer.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#166
post #99

Earlier quoted context omitted.

Also if all the company is doing is boring crud app as a service then the whole company is going to disappear once anyone can ask ChatGPT to create a version of the service for themselves

Most CRUD apps in enterprise settings relate to something in the real world. E.g. ordering systems, warehouse management and so on. The hard part here isn’t really the coding - but understanding the business logic and getting it out of the customer what they need. Often the customer hasn’t even got a good enough grasp on their own business to be able to formulate what they need. In the short term this ai tools will m…

‘Understanding the business logic and getting it out of the customer’ is precisely what a lot of programmers are bad at doing. Many would rather talk to a compiler than a human being. For them, ChatGPT is a real existential threat.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#167

Earlier quoted context omitted.

the big thing to me is programming isn't hard I've worked on adtech, crypto, fintech, gamedev, startup founder, BigCo. Not once was programming something that was a time sink. Makes me feel like GPT is marketing to the incompetent or something.

> Makes me feel like GPT is marketing to the incompetent or something. Absolutely. The common constant I can see in people who are really blown away by GPT's performance at [task] is that they are bad at [task]. Programmers who describe their job as copying from StackOverflow think it's great at coding. People who don't read fiction think it's great at writing fiction, and so on.

That's not accurate at all for me. I'm not impressed with the output when rating it against humans who are good at said task. It's obviously not as good as those who are good in that relevant field.

But it's certainly far better than humans who are not skilled at those tasks, and that is what I find to be very impressive. I just didn't realise that these models could be this good, and they're not even as good as they will be.

I guess if you were expecting something that's going to be as good as those who are skilled in a particular field, you'll be unimpressed -- but I wasn't even expecting mediocrity.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#168
Blah Blah Blah. I use ChatGPT for this every day to write code to save my own efforts and it is doing just fine thanks. I also use it for creative content in my apps, although I edit this work to get the tone in its writing correct. It is excellent for this.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#169
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

  > I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating.
Equally fascinating is all of the "this is fine" posts from programmers suddenly realizing they are not the gods they once thought.

But fret not, programming is not the first industry that has been automated into a shell of itself. Yes, the industry is going to shrink massively, but this is what new skills are for. Just as farmers had to learn industrial jobs and then miners and autoworkers had to "learn to code", most programmers will have to learn to do something else. Humans are resilient and will adapt.

And there will still be jobs in development for the most talented and in niche areas, but when the largest tech companies can layoff hundreds of thousands of employees without skipping a beat that should tell you all you need to know about the value of most "programming" jobs.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#170
The biggest thing here is that it's semi capable and improving. I feel safe about my job right now but it is worrying to invest time to compete with a machine that will continue to get better over the years where previously I felt safe that the effort of my labour would bear fruit for decades to come. Now I'm not so sure.
Post reply on HN