Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

91–100 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#91

Earlier quoted context omitted.

Demonstrably untrue, it’s taken up semi-residence in everything from oped pages to boardrooms since the New Year. Welcome to the mid singularity, my friend.

You can’t say “singularity” on HN without a ton of downvotes. It’s not because people disagree (or they’d say so), but because they literally shit their pants when they think about it. Hard to blame them.

[dead]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#92
post #75

Earlier quoted context omitted.

Eh, it's still impressive that these systems can write such good code despite pretty much just predicting the next word. I guess it's a matter of perspective. You can either be astounded by how much it can do relative to your expectations from 2018 or you can be skeptical relative to the volume of excitement. I think some excitement is also people extrapolating to the future: if predicting the next word gets you this…

> despite pretty much just predicting the next word Is GPT really like the old school NLP trigram stuff?

"predicting the next word" sounds trivial until you realize the value and complexity of "predicting the next word[s] by a world leading expert in a particular domain".

Re: Testing GPT 4's code-writing capabilities with some real world problems

#93
post #50
post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I don’t think you’ve tried GPT-4. If you have, I don’t think you are like majority of devs (maybe not on HN, but in real life). You sound lucky to have true, novel problems to solve each day. I’m with many here commenting that this is quite powerful stuff, especially when my day-to-day is writing simple CRUD apps, or transforming data from one format to another within an API, or configuring some new bit of infra or C…

Also if all the company is doing is boring crud app as a service then the whole company is going to disappear once anyone can ask ChatGPT to create a version of the service for themselves

Re: Testing GPT 4's code-writing capabilities with some real world problems

#94

Earlier quoted context omitted.

Demonstrably untrue, it’s taken up semi-residence in everything from oped pages to boardrooms since the New Year. Welcome to the mid singularity, my friend.

You can’t say “singularity” on HN without a ton of downvotes. It’s not because people disagree (or they’d say so), but because they literally shit their pants when they think about it. Hard to blame them.

Who's not disagreeing? Singularity feels like "man facing existential dread confuses saturation curve for exponential".

Re: Testing GPT 4's code-writing capabilities with some real world problems

#95
Remember that these models generate one token at a time. They do not "think ahead" much more than maybe a few tokens in beam search. So if the problem requires search - actual comparison of approaches, and going back-and-forth between draft and thinking through the implications - the model can't do it (except in a limited sense, if you prompt it to give it's "train of thought"). So it's comparable of you being in front of whiteboard, hit with a question, and you would have to start answering immediately without thinking more than you can while talking through your answer at the same time. Doable if you know the material well. If it's a new problem, that approach is doomed. Given that, I think the language models do remarkably well. A little bit of search, and maybe trying to generate the answer in different order (like short draft -> more detailed draft -> more detailed draft... etc.) will improve things a lot.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#96
post #88
post #67

Earlier quoted context omitted.

> you can coerce it to do surprising things Yeah, I've gotten it to write some pretty decent contracts, but only because I have written said contracts and can ask it all the right questions/prod it into adding what I need.

And the next iteration of ChatGPT will likely generate a better contract because your interaction with it will be used as training data

Huh. There’s something scary about that in my lizard brain. Not sure what.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#97
post #71

Earlier quoted context omitted.

> Not once was programming something that was a time sink. Can you please expound on that? Do you mean not hard as in comparison to something else?

I think all it shows is that GP has never worked on complicated problems that require anything more than gluing libraries together.

If you do nontrivial coding then GPT wont help you, so that isn't relevant for the discussion.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#98
As a designer/non-coder, it feels like I'm just pair programming all the time.

Stuff that usually took me a long time like regexes or Excel/Sheets formulas now take like two minutes. AND I'm learning how they work in the process. I can actually write regexes now that used to be wildly confusing to me a couple of months ago, because Copilot / ChatGPT is walking through the process, making mistakes, and me prodding it along.

I feel like it doesn't matter how "mindblowing" or "a big deal" this tool is — it's a great learning tool for me and helps me do my work 100x faster.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#99
post #50

Earlier quoted context omitted.

I don’t think you’ve tried GPT-4. If you have, I don’t think you are like majority of devs (maybe not on HN, but in real life). You sound lucky to have true, novel problems to solve each day. I’m with many here commenting that this is quite powerful stuff, especially when my day-to-day is writing simple CRUD apps, or transforming data from one format to another within an API, or configuring some new bit of infra or C…

Also if all the company is doing is boring crud app as a service then the whole company is going to disappear once anyone can ask ChatGPT to create a version of the service for themselves

Most CRUD apps in enterprise settings relate to something in the real world. E.g. ordering systems, warehouse management and so on.

The hard part here isn’t really the coding - but understanding the business logic and getting it out of the customer what they need.

Often the customer hasn’t even got a good enough grasp on their own business to be able to formulate what they need.

In the short term this ai tools will make the programming part easier but the planning part is still hard.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#100
post #83
post #73

Earlier quoted context omitted.

On the flip side, I am having a great time charging enterprises exorbitant hourly rates to come in and help them fix their crappy crud programms. But I still don’t know if I am thrilled by the prospect of being called in to make their monkeypatched, GPT generated project work.

"Hey GPT-X: Can you refactor the codebase in gpt@monkeypatched-crap.git for me? Preferably in the style of gpt@crappycrud.git that I worked on last year."

Token cap will probably be the biggest problem here.

After validation.

After getting the changes to disk, documented, actually compiling, etc…

But the biggest problem is that transferring the nuance that is external to the code base is typically really tiresome and lengthy grunt work and again token cap.

Post reply on HN