Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

151–160 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#151

Post by tomduncalf »

I didn’t have much luck with ChatGPT trying to solve a novel problem (sorry can’t share details), it gave answers that kind of sounded plausible if you didn’t really understand the problem but in reality were no help. It also hallucinated a bunch of research papers that sounded really useful haha.

Will have to try GPT-4 for the same thing and see if it’s any better, I suspect though that this kind of genuinely novel problem solving may be beyond its current abilities (unless you work through to step by step in a very granular way, at which point you’re solving the problem and it’s writing the code - which could be a glimpse of the future!)

Re: Testing GPT 4's code-writing capabilities with some real world problems

#152

Post by dragonwriter »

bsaul wrote: post #148

there's a bit of confusion when people say it's not going replace programmers because they all have tricky things to do in their work week. This is not how it's going to happen : if your boring time-consuming tasks take virtually 0 time thanks to gpt, and let you focus on the 1% that's hard, you've suddenly become 100x more efficient, and can thus accomplish the same job as 100 you. That means the company can now fir…

> if your boring time-consuming tasks take virtually 0 time thanks to gpt, and let you focus on the 1% that’s hard, you’ve suddenly become 100x more efficient, and can thus accomplish the same job as 100 you. That means the company can now fire 99 coworkers, keeping only you, and end up with the same result.

But it means that tasks where building software would only deliver 1% of the necessary value to pay for the cost of doing it are now suddenly worth paying for, so even if your company, being a stick-in-the-mud non-innovator that is going to stay in exactly the same niche doing the same thing cut 99% of its programming staff and used the cost savings on executive bonuses and stock buybacks, a whole lot of customers (and the new and pivoting companies serving them) are going to be spending money on programmers that weren’t before, so not only will your ex-coworkers still be employed more programmers in total will be, even if their work is now mostly higher-level abstraction and wrangling LLM code generators, with the level we think of as “source code” today being touched as rarely as today’s high-level application developers touch machine code.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#153

Post by Planktonne »

whateveracct wrote: post #53
JimmyRuska wrote: post #22

Earlier quoted context omitted.

This post feels like the people that go into linux forums and say linux sucks because I can't get it to do X but microsoft can, but then get 400 replies and several that show GPT has limited reasoning but given enough knowledge of the problem you can coerce it to do surprising things so long as you can relate it to something in else in the knowledge base. Given how big that knowledge base is, you can get lucky surpri…

the big thing to me is programming isn't hard I've worked on adtech, crypto, fintech, gamedev, startup founder, BigCo. Not once was programming something that was a time sink. Makes me feel like GPT is marketing to the incompetent or something.

> Makes me feel like GPT is marketing to the incompetent or something.

Absolutely. The common constant I can see in people who are really blown away by GPT's performance at [task] is that they are bad at [task].

Programmers who describe their job as copying from StackOverflow think it's great at coding. People who don't read fiction think it's great at writing fiction, and so on.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#154

Post by DaiPlusPlus »

anonzzzies wrote: post #121
whateveracct wrote: post #52

Earlier quoted context omitted.

There's not such thing as actually boring CRUD. I've worked at many companies and founded my own. Even when it felt like CRUD, a year+ in it was clear that tasteful decisions pay off and iteration and cost gradients matter. GPT doesn't sniff that. I agree with GP - day 2 dev me outclasses it, which means it isn't replacing anyone.

HN echo chamber. Generally the better/best programmers hang around here; your day 2 was probably better than many coders hope to achieve in their whole lives. It is replacing people already; I personally know about 10 people who have been fired or assigned to a completely different position because gpt did a faster and better or equal job. So ‘not anyone’ is simply nonsense; I am only one person, there will be many m…

> I personally know about 10 people who have been fired or assigned to a completely different position because gpt did a faster and better or equal job

Please elaborate.

And, if true, this would be a major news story that Vox or any number of major newspapers would love to write about - so have you approached the media about this? If not, why not?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#155

Post by ashwagary »

pixl97 wrote: post #104
jamiek88 wrote: post #96

Earlier quoted context omitted.

Huh. There’s something scary about that in my lizard brain. Not sure what.

Its like feeding a baby polar bear. When it grows up it is going to eat you.

Extending this analogy into a question: Could the polar bear community eat enough expert food providers that their quality of food and eventually overall health declines?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#156

Post by zackees »

noitpmeder wrote: post #76
jacobedawson wrote: post #16

Earlier quoted context omitted.

I think most people see where the puck is going, and even where it is right now is very impressive. It's not hard to see that it will be likely less than 5 years before it will be able to do what you did on day 2, and much more, at a tiny fraction of the cost, with no downtime, no attitude problems, sick days, etc. The things you mentioned (taking on board feedback, testing across devices, iterating on solutions) doe…

Personally I think it's 99% hype. The current iteration and architecture of these systems means they will never be at the level where they can actually replace a programmer. The best they will ever get is barfing up snippets for a programmer (who still needs the industry-specific knowledge to VERIFY the snippets). Additionally, "the rate of increase in capabilities" is very much a false flag. Past performance (especi…

[dead]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#157

Post by jah242 »

Whilst maybe GPT-4 will change this, I think it is important to remember that these general ChatBots are not the way we have generally trained LLMs to write the best code. In fact, coding is one of the few areas where training specifically just using source code and maybe some stack overflow (not all natural language on the internet) leads to better results on the previous iteration of LLMs (GPT-3 wave). So the real test will be whether the GPT-4 wave of specific coding LLMs i.e GPT-4-Codex can 'actually write code' see:

AlphaCode Codex CodeGen

Re: Testing GPT 4's code-writing capabilities with some real world problems

#158

Post by DaiPlusPlus »

copperx wrote: post #119
akamoonknight wrote: post #12

One thought that came to mind when the author mentioned that GPT-4 seemed to want to use A* was that maybe the problem statement was not formed in a way that would get the best response. I'm not entirely sure if this type of back and forth would be useful, but just struck me as interesting to try. I put the following prompt into ChatGPT (no GPT-4 access for me yet): *A problem statement is given as follows: An agent…

This is perhaps one of the most impressive responses I've read. It truly seems like there is some reasoning happening. I don't understand how this can be the output of a generative LLM.

> It truly seems like there is some reasoning happening. I don't understand how this can be the output of a generative LLM

Right - this seeming "cognition" is exactly what's so spooky about the whole thing.

Here's what spooked me out from yesterday: https://news.ycombinator.com/item?id=35167685 - specifically how it determines the divide-by-zero error in this code: https://whatdoesthiscodedo.com/g/6a8f359

...which demonstrates GPT as being capable of at-least C++ "constexpr"-style compile-time computation, which shouldn't even be possible if one presumes GPT is "just" a giant database storing only multidimensional word similarity scores and sequence distribution from text inference.

> a generative LLM

I definitely wanted to believe that GPT was "just predicting the next word" - it was somewhat comforting to think of GPT as still being far from being-human or behaving like a real brain does - but that explanation never sat well with me: it was too simplistic and dismissive, and didn't explain the behaviour I was witnessing in GPT.

...so after having read Wolfram's (surprisingly accessible) article on GPT ( https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-... ) it made a lot of things "click" in my head - and enabled me to start to understand why and how GPT is capable of... the surprising things it does; but it also leads me to believe we're (warning: incoming cliche) barely scratching the surface of what we can do: right-away I do believe we're almost at the point where we could simply ask GPT how to adapt it into some kind of early AGI - and we've all heard of what's supposed to follow next... and it really is quite unsettling.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#159

Post by Maxion »

paganel wrote: post #116
Maxion wrote: post #99

Earlier quoted context omitted.

Most CRUD apps in enterprise settings relate to something in the real world. E.g. ordering systems, warehouse management and so on. The hard part here isn’t really the coding - but understanding the business logic and getting it out of the customer what they need. Often the customer hasn’t even got a good enough grasp on their own business to be able to formulate what they need. In the short term this ai tools will m…

Excellent response. I also wanted to add about the myriad of incoming data formats that need to be processed and the myriad of data exports that one has to implement for most of those "boring" CRUD apps. If one hasn't written code that includes comments like "Special case, California does it this way" or "Alberta needs an .xml export, not an .xsl one", with a link to a .pdf spec that points to somewhere on the intern…

And n+1 things that HAVE to be done against the spec. Or that for some region some field has to be hidden, but only for that region - not any else.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#160

Post by ChatGTP »

Hedepig wrote: post #140

It brings to my mind the levels of self driving We're definitely at 2 right now, and picking away at level 3. I have heard some people skeptical that we can overcome the problems of truthfulness due to the inherent limitations of LLMs. But, at least on the face of it, it appears we can make incremental improvements. If only they would actually be Open AI I have seen

True "Open AI" is coming, there's no way this is going to stay Microsoft alone for very long.

Many, many companies will be looking to integrate with this thing and no one is going to juts sit three and let MS take their lunch forever.

Post reply on HN