Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

131–140 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#131

Post by theK »

anonzzzies wrote: post #112
theK wrote: post #100

Earlier quoted context omitted.

Token cap will probably be the biggest problem here. After validation. After getting the changes to disk, documented, actually compiling, etc… But the biggest problem is that transferring the nuance that is external to the code base is typically really tiresome and lengthy grunt work and again token cap.

They went from 4K to 32k in a few months. Wait a few years.

Few years, yes. I’m with you “the change is coming” but we still need to transfer millions of tokens in and out to cater for context and out of repo intricacies.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#132

Post by theK »

anonzzzies wrote: post #110
theK wrote: post #73

Earlier quoted context omitted.

On the flip side, I am having a great time charging enterprises exorbitant hourly rates to come in and help them fix their crappy crud programms. But I still don’t know if I am thrilled by the prospect of being called in to make their monkeypatched, GPT generated project work.

> their monkeypatched You mean, like almost every outsourcing company pops over? So the type of code that infests companies who hired some sweatshop to do ‘some simple crud’? What’s the difference? Can you see the difference? Besides the gpt code will be far better commented as comments come for almost free with gpt while humans hate writing them.

I think you can trust the code of both those worlds exactly the same, not at all.

I’ve seen GPT 3 and 4 hallucinate the most amazing commentary about their own output. Maybe we will get dependable, out of process, guidance at some point about how factual the model thinks it is on an output per output basis but until that point you should trust every LOC and comment exactly the same as code gifted to you by an adversary.

My modest 2¢

Re: Testing GPT 4's code-writing capabilities with some real world problems

#133

Post by saurik »

it_citizen wrote: post #3

To be fair, in 15 years writing code, I have spent the vast majority of my time working on minor variations of problems already solved. I am not a fancy developer coming up with new algorithms. I make sign up flows, on-boarding flows, paginated lists, integrations to other apis. And I definitely feel that my job might be threatened by LLMs.

To the extent to which anything that makes you take less time doing the specific tasks you are doing today (and thereby, presumably, bill fewer hours or fail to defend such a high headcount on your team) threatens your job, we might also say that better programming languages and tooling threaten your job, better error messages and documentation threaten your job, or higher levels of abstraction and higher quality frameworks threaten your job... were you also fretting about the new version of TypeScript that just came out earlier today, or did you think "wow, that makes me more effective, I can't wait to use it"?

I might go so far as to argue that the entire reason software developers exist is to threaten all jobs, including our own: at our best--when we are willing to put in a bit of thought into what we are doing--we don't just make things easier to do for a moment while we are employed (which is the best of what most professions can achieve): we make things persistently and permanently easier to do again and again... forever; and we don't just make other peoples' jobs easier: this same power we have applies to our own tasks, allowing us to automate and replace ourselves so we can move on to ever more rewarding pursuits.

I'm not a fan of GPT for coding for a number of reasons (at least, in its current form, which is all we can ever have a true opinion about); but, it isn't because it will replace anything I've ever done: it would have just unlocked my ability to work on better things. There are so many things I wish I could get done before I die, and I know I'm going to be able to get to almost none of it... I have so many plans for ways to improve both the world and my life that will never happen as I just don't have the capability and bandwidth to do it all. If I had a God I could ask to do all the things I already do... I can only imagine what I'd do then.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#134

Post by matwood »

keyle wrote: post #2

So, for a bit of fun, I signed up to GPT-4 thingy plus and I picked a fairly common web application and built it from scratch, only by talking to GPT-4 and copy pasting the code bits. I'm actually taken back by how well it's doing; including providing me some refreshers on stuff I forgot how it should work. I can see it failing at solving complex problems, but like the blog post mentions, most programming isn't new o…

I think there’s augmenting programmers (which I think will happen) or replacing programmers (which I think will not happen soon). It’s a capable and improving tool that humanity will figure out how to saturate like we do with everything else.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#135

Post by Jensson »

hnfong wrote: post #92
copperx wrote: post #75

Earlier quoted context omitted.

> despite pretty much just predicting the next word Is GPT really like the old school NLP trigram stuff?

"predicting the next word" sounds trivial until you realize the value and complexity of "predicting the next word[s] by a world leading expert in a particular domain".

Quoting famous people sounds smart until you realize they just memorized a ton of trivia. These models have demonstrated that they don't learn logical models, instead they learn to generate text that looks logical at first glance but is nonsense.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#136

Post by theshrike79 »

pixl97 wrote: post #126
theshrike79 wrote: post #30

I just used GPT-4 yesterday to write a Go-parser for a specific JSON input. Within two prompts it could read the JSON data from a stdin stream, unmarshal it to Go structs and print the correct fields to stdout as a human-readable line of text. Then I told it to colour the timestamp and id fields using the fatih/color -package, and it did it correctly. In total it took me about 4-5 prompts to get where I wanted. I jus…

>Then I asked it to write unit tests for the code, and it confidently started writing correct-looking code that would take the same input and expect the correct output, but just stopped in the middle. One of two things. First ask it to continue. Sometimes it just stops half way thru code foe whatever reason. The other possibility is you filled up the token context window. Not much you can do but wait for the 32k mode…

I asked it to continue twice after the first failure. Every time it failed in about the same point. Might've filled up some mysterious limit in the model.

I didn't really need the unit tests anyway, but I wanted to try if it could do it :)

Re: Testing GPT 4's code-writing capabilities with some real world problems

#137

Post by chillfox »

nitwit005 wrote: post #31
chillfox wrote: post #6

I don't think I have ever solved a truly new problem from scratch when programming... It's all been apply algorithm x to y problem or crud stuff. The most difficult problem that I have asked GPT-4 to solve was writing a parser for the Azure AD query language in a niche programming language and it did that just fine (I did have to copy paste some docs into the prompt).

Pathfinding with extra constraints isn't "a new problem" either. There are a bunch of papers on the topic, and I'm sure there are multiple different variations on github. It still didn't succeed (did get close though).

Maybe it could have got there with better prompting, maybe not. But by the time GPT-5 or 6 comes around it would be highly likely to be able to solve it perfectly.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#138

Post by qikInNdOutReply »

nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

The reason is easy to imagine. Most non-programmers, are living like analphabets in a world were reading is valuable super power. They grudgingly accept this power assemtry, but ocassionally rebel - with "easier" visual programming languages made and excel.

This is another one of those rebellions, non-programers hoping to avoid reading the book and closing it for good, while keeping the awesome around. The code-bases we will see, were the commits are basically chatgpt tags and tasks for each document.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#139

Post by stareatgoats »

keyle wrote: post #2

So, for a bit of fun, I signed up to GPT-4 thingy plus and I picked a fairly common web application and built it from scratch, only by talking to GPT-4 and copy pasting the code bits. I'm actually taken back by how well it's doing; including providing me some refreshers on stuff I forgot how it should work. I can see it failing at solving complex problems, but like the blog post mentions, most programming isn't new o…

> silly to ignore it and palm it off. It's a big deal.

Agree, this is a big deal, and has the capacity to revolutionize all the techniques we have been using up to now for compiling, summarizing and reframing existing knowledge as expressed in writing (including code).

Not only does Google get (well deserved) competition, it means pressure on all the businesses that now make a living in that space. In a few years it will even have a serious impact on major such institutions in society like schools and universities.

A lot if not all of the kickback from established institutions will be attempts to smear the competition, and by all means, to carve out new niches where GPT-X is not applicable or as efficient.

There are valid concerns about the veracity of the information it provides which means there are limits to the extent it can be used in automated processes, but I'd loathe to trust the data unconditionally anyway. As for not being able to think creatively: good on us. But it's likely just temporary.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#140

Post by Hedepig »

It brings to my mind the levels of self driving

We're definitely at 2 right now, and picking away at level 3.

I have heard some people skeptical that we can overcome the problems of truthfulness due to the inherent limitations of LLMs. But, at least on the face of it, it appears we can make incremental improvements.

If only they would actually be OpenAI

I have seen

Post reply on HN