Testing GPT 4's code-writing capabilities with some real world problems • Hacker News Board

Re: Testing GPT 4's code-writing capabilities with some real world problems

#211

Post by staminade » Fri, Mar 17, 2023, 12:34 PM UTC

nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills.

What they failed to predict was that some people wouldn't try to automate them like-for-like. Instead they would reconfigure their entire approach to fit with the specific advantages and limitations of the machinery. And this new approach might even be qualitatively worse in various ways, but not so much as to overwhelm the economic advantages that provided by the things machines were good at.

AI likely isn't going to slot into a developer-shaped hole in a software team. But it's possible we'll see new organisation approaches, companies, and development paradigms that say: How far can you get if you put prompt-generated code at the heart of the workflow and make everything else subservient to it. I'm not sure, right now, that that approach is feasible, but I'm not sure it won't be in a year or two.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#212

Post by davemp » Fri, Mar 17, 2023, 12:35 PM UTC

jacobedawson wrote: post #16

nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I think most people see where the puck is going, and even where it is right now is very impressive. It's not hard to see that it will be likely less than 5 years before it will be able to do what you did on day 2, and much more, at a tiny fraction of the cost, with no downtime, no attitude problems, sick days, etc. The things you mentioned (taking on board feedback, testing across devices, iterating on solutions) doe…

> The things you mentioned (taking on board feedback, testing across devices, iterating on solutions) doesn't seem very far away at all.

This is a very bold claim IMO.

Modeling/understanding interactions in a complex system of potential black boxes is much, much more computationally difficult problem that source code to source code operations.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#213

Post by chrismsimpson » Fri, Mar 17, 2023, 12:35 PM UTC

eggsmediumrare wrote: post #197

chrismsimpson wrote: post #46

Prediction based on statistical probabilities != comprehension. So no.

If the outcome is the same, does it matter?

Have you heard of the halting problem? For one, you’ll never be able prove the outcome is the same.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#214

Post by danielbln » Fri, Mar 17, 2023, 12:36 PM UTC

braingenious wrote: post #41

I tried out gpt4 today with the task of “take some html files made by a non technical person using various versions of microsoft word over a decade ago and put the contents into a csv” and it hasn’t done great. Not terrible, but not great. That being said, I don’t know anybody talented enough to handle it that would even look at this project for $20 so ¯\_(ツ)_/¯

An alternative path would be to tell it to write python (or $LANG) code that can parse these HTML files and output the right CSVs.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#215

Post by sgarland » Fri, Mar 17, 2023, 12:36 PM UTC

Jensson wrote: post #135

hnfong wrote: post #92

Earlier quoted context omitted.

"predicting the next word" sounds trivial until you realize the value and complexity of "predicting the next word[s] by a world leading expert in a particular domain".

Quoting famous people sounds smart until you realize they just memorized a ton of trivia. These models have demonstrated that they don't learn logical models, instead they learn to generate text that looks logical at first glance but is nonsense.

I asked GPT-4 something fairly niche that I happen to know a fair amount about: to explain the concept of Xenon poisoning in a nuclear reactor. Other than skipping Te-135 being the initial fission product that starts the decay chain (and tbf, operationally it can be skipped since the half-life is 19 seconds), it got everything correct.

I'm sure if I kept probing on smaller and smaller details it would eventually fail, but I'd argue for _most_ people, on _most_ subjects, it performs incredibly well.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#216

Post by Quarrelsome » Fri, Mar 17, 2023, 12:37 PM UTC

SuoDuanDao wrote: post #182

nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

I think we're seeing the early phases a prediction I made in my first book come true: That computers will be more suitable than humans for most of the functions humans currently use their left brain half for. Best case, that will have a whole lot more humans using their right brain halves on things like defining the problem. I like the thought of that, it's more pleasant work. But a lot of intelligent people define t…

I feel like there's a slight inaccuracy here that this article covers. GPT-like models excel at logical problems that have already been solved. As the article suggests the models (at least at the moment) are extremely keen to utilise existing solutions as opposed to inventing new ones. The existing solutions end up somewhat being far too distracting when problem complexity compounds beyond the common. This implies that those truly talented at solving logical problems will still be spearheading the development of novel solutions.

We might be able to state that GPT will easily trim away all the average workloads for both the left and right. It can perform "creative writing" or draw pictures to an average or even beyond average extent, but it continues to currently struggle to hit the exceptional examples that humanity are capable of.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#217

Post by joenot443 » Fri, Mar 17, 2023, 12:38 PM UTC

theshrike79 wrote: post #106

nitwit005 wrote: post #58

Earlier quoted context omitted.

If all you're doing is very simple crud apps and transforming API responses, your coding is already replaceable with "no code" tools.

I'm still waiting for the no-code tools that can replace me, the API Glue Coder. They've been promising those since the early 2000's, but I'm still employed. Some stuff has been automated by Zapier, IFTTT and the like, but most of the systems I'm integrating with don't have public APIs.

"I'm still waiting for the automobile that can replace me, the Horse Carriage Driver. They've been promising those since the 1880s, but I've still got passengers in my carriage every day!"

Obviously it's a cheeky example, but this would not be the first time in history a previously well-established career was upended in a (relatively) short amount of time. I'm a FAANG dev, I've got skin in the game too and I'm trying to be optimistic, but I can't help but be at least a little worried. From Wikipedia -

"In 1890 there were 13,800 companies in the United States in the business of building carriages pulled by horses. By 1920, only 90 such companies remained."

I don't think we'll be out of the job entirely, but I can definitely imagine the bar being raised and the compensation stagnating as we now have to justify the time and cost of our work compared to the near-free, near-instantaneous output of an LLM.

All that being said, if you've been working since the 2000s, you've got nearly a 20 year head-start on me, so perhaps it makes sense for me to be a bit more worried.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#218

Post by muyuu » Fri, Mar 17, 2023, 12:38 PM UTC

keyle wrote: post #2

So, for a bit of fun, I signed up to GPT-4 thingy plus and I picked a fairly common web application and built it from scratch, only by talking to GPT-4 and copy pasting the code bits. I'm actually taken back by how well it's doing; including providing me some refreshers on stuff I forgot how it should work. I can see it failing at solving complex problems, but like the blog post mentions, most programming isn't new o…

assistants and wizards have been tried before with varying levels of success

clippy tanked because it annoyed more than it helped, although some people did like it

install wizards did their job in a world where a single binary format and OS dominated and stuff ran offline pretty much exclusively, with the odd connection to networks - those installers sorted a laundry list of situations, both underlying situations and user configurations and choices, and for the most part they worked

Siri, Cortana, Alexa etc have been working as expert systems with central curated bases and some AI/ML on top, for a lot of people they've been working quite well - for me personally they've sucked, they've totally failed to answer my questions the few times I've tried them, and they've creeped the hell out of me (they are a lot more centred on extracting my info and selling me stuff than understanding stuff)

generative ML is orders of magnitude more sophisticated, but so are our needs and our computing from a global perspective, it does make sense that those assistants, pilots, etc start taking off

but the incentive issues of the previous generation assistants and recommendation algorithms remains there and I wonder how will that turn out - if they start demanding access to my phone, my email, my contacts etc I will do my best to avoid them and to poison any info I have to give them

Re: Testing GPT 4's code-writing capabilities with some real world problems

#219

Post by superasn » Fri, Mar 17, 2023, 12:39 PM UTC

tarruda wrote: post #198

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Maybe if you train/tune GPT-4 with enough samples of similar interactions, it will learn to do it too.

If anything it will be much better as it won't mind waiting and asking for answers nor get frustrated with incompetence.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#220

Post by davewritescode » Fri, Mar 17, 2023, 12:39 PM UTC

I think most folks (perhaps not here) misunderstand is that writing code is the easiest part of a software engineering job. Anyone can write code in their little piece of the system, write some tests to prove it works and move on. Given enough time I feel most good software engineers can do that part of the job without issue.

Knowing how code might fail and preventing cascading effects, tuning resource usage, troubleshooting incidents are the actual hard parts of software development and it's where even good software engineers tend to fall over. We've created whole specialties like SRE to pickup where application developers fall short. I've seen lots of systems fail for the dumbest reasons. Thread pools misconfigured, connection timeouts with poor configuration, database connection pools are completely incorrect.

Wake me up when ChatGPT can troubleshoot at 1 AM when the SRE and on call engineer are both frantically trying to figure out why logs are clean but the service is missing it's SLO.