Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

41–50 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#41
I tried out gpt4 today with the task of “take some html files made by a non technical person using various versions of microsoft word over a decade ago and put the contents into a csv” and it hasn’t done great. Not terrible, but not great.

That being said, I don’t know anybody talented enough to handle it that would even look at this project for $20 so ¯\_(ツ)_/¯

Re: Testing GPT 4's code-writing capabilities with some real world problems

#42
post #10
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

Have you seen this video? https://www.youtube.com/watch?v=9QZlQMpNk-M I think the author is onto something – while AI might not be able to program per se, it can certainly be handed a code snippet and then use its huge corpus of Internet Learning™ to tell you things about it, code that looks like it, and ways (people on the Internet think) it might be solved better. In that sense, it isn't replacing the programmer; i…

> ways (people on the Internet think)

It reminds me of arguments that it's not the computer that plays chess, but its programmers.

You can describe a GPT's response as a statistical average of responses on the internet (for quite a contrived definition of average), but at some point it will be easier to describe it as analyzing a snippet and forming an opinion (based on what people on the Internet think). Are we past that point? I'm not sure yet, but we are close.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#43
post #25

Earlier quoted context omitted.

I think most people see where the puck is going, and even where it is right now is very impressive. It's not hard to see that it will be likely less than 5 years before it will be able to do what you did on day 2, and much more, at a tiny fraction of the cost, with no downtime, no attitude problems, sick days, etc. The things you mentioned (taking on board feedback, testing across devices, iterating on solutions) doe…

That's what they said about Watson and Wolfram Alpha. You only hear this nonsense on HN.

Except Watson and Alpha weren’t useful out of the box to everyone. They weren’t the fastest growing app of all time and couldn’t even remotely play that way. ChatGPT isn’t hype because it is so useful — it’s already on the “Plateau of Productivity”.

But it looks like an exponentially steep climb from here. indefinitely

Re: Testing GPT 4's code-writing capabilities with some real world problems

#44
post #37
post #2

So, for a bit of fun, I signed up to GPT-4 thingy plus and I picked a fairly common web application and built it from scratch, only by talking to GPT-4 and copy pasting the code bits. I'm actually taken back by how well it's doing; including providing me some refreshers on stuff I forgot how it should work. I can see it failing at solving complex problems, but like the blog post mentions, most programming isn't new o…

I don't really have anyone to ask questions I get sometimes about building software, and chatGPT has been helping fill the gaps. Basically I'm thinking of it like a combination of a rubber duck and a dialogue-enabled google search. But it's been really helpful along those lines when I'm for example not sure a good way to change a bunch of stuff across a bunch of files, and am pretty sure it's something that can be au…

Yes, thanks, that's exactly how I feel as well.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#45
post #25

Earlier quoted context omitted.

That's what they said about Watson and Wolfram Alpha. You only hear this nonsense on HN.

Demonstrably untrue, it’s taken up semi-residence in everything from oped pages to boardrooms since the New Year. Welcome to the mid singularity, my friend.

You can’t say “singularity” on HN without a ton of downvotes. It’s not because people disagree (or they’d say so), but because they literally shit their pants when they think about it. Hard to blame them.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#47
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I feel it's the average programmer who is enamoured with GPT. The programmers working on hard stuff know that AI writing programs is still far away.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#48

So it continues to reaffirm what we’ve known: generative LLM does not have a model of the world, can not reason and can not plan. It generates text by mix-matching remembered texts and thus it can not generate truly new content. No surprise because GPT-4 is built upon the same model as GPT-3. Clever Engineering will bring us far, but breakthrough requires change of the fundamentals. Nevertheless, it’s useful and can…

I copied the opinion of Yann LeCun, one of the authorities on deep learning: (Feb 13,2023) My unwavering opinion on current (auto-regressive) LLMs 1. They are useful as writing aids. 2. They are "reactive" & don't plan nor reason. 3. They make stuff up or retrieve stuff approximately. 4. That can be mitigated but not fixed by human feedback. 5. Better systems will come. 6. Current LLMs should be used as writing aids,…

Fantastic comment, saving this. It's clear that for AI to be AI, it needs what philosophers of language call a Knowledge Base and a few intrinsic axiomatic presuppositions that, no matter what happens, cannot be broken (kind of like the Pauli exclusion principle in real life).

Re: Testing GPT 4's code-writing capabilities with some real world problems

#49
post #19

As a human programmer I didn't quite understand the problem statement until I read the whole article and the tests. I believe the goal is to find a path with the fewest possible "fire" cells and the minimum cost as a tie breaker. The cost of a path is the sum of its cells' cost and it can't be greater than 5. If I understood the assignment correctly, I don't think the problem statement is equivalent to what's include…

I agree. That description of the problem was horrible. Maybe ChatGPT could write a better description and then people could code up the algorithm.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#50
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I don’t think you’ve tried GPT-4.

If you have, I don’t think you are like majority of devs (maybe not on HN, but in real life).

You sound lucky to have true, novel problems to solve each day. I’m with many here commenting that this is quite powerful stuff, especially when my day-to-day is writing simple CRUD apps, or transforming data from one format to another within an API, or configuring some new bit of infra or CI/CD.

I’d love to be challenged in some new way, and have access to truly fascinating problems that require novel solutions. But most enterprises aren’t really like that nor do that need that from majority of engineers.

Post reply on HN