Testing GPT 4's code-writing capabilities with some real world problems • Hacker News Board

Re: Testing GPT 4's code-writing capabilities with some real world problems

#451

Post by _fizz_buzz_ » Fri, Mar 17, 2023, 4:05 PM UTC

vages wrote: post #253

IKLOL wrote: post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

If increased productivity equaled job loss, there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950. A lot of projects today are not even greenlit because they would be too expensive to make. For instance, there are a lot processes in almost every country that require you to file paper forms, even though we have had web forms and…

This is true for other domains also. My wife translates for an ecommerce business. A lot of what she does is very automated. The software she uses remembers all phrases she has translated and uses a word bank she manages for special parts and uses DeepL for new translations which are then only proofread and fine tuned. It's amazing how much stuff she can translate that way in a day (no idea how many thousands words). She is kind of managing and overseeing an AI (DeepL) and sanity checking the work. If this was 20 years ago one would probably need a translation department of 10-15 people to do the same work. However, her company would have never been able to justify that kind of cost. So, in my view: Software and AI made translators probably more than 10x more efficient in the last 10 years, however the amount of stuff that gets translated also increased 10 fold during this time.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#452

Post by mcv » Fri, Mar 17, 2023, 4:06 PM UTC

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Yeah, I don't see AI replacing programmers. Or any job that's the slightest bit interesting. I see AI as another tool in our toolbox that will help us do our job better.

People have been working on medical and judicial expert systems for ages, but nobody wants to put those systems in charge; they're just meant to advise people, helping people make better decisions.

And of course chatGPT and GPT-4 are way more flexible than those expert systems, but they're also more likely to be wrong, and they're still not a flexible as people.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#453

Post by nmca » Fri, Mar 17, 2023, 4:08 PM UTC

jokoon wrote: post #426

People need to understand that AI doesn't think, doesn't have insight or intuition. AI just repeat patterns it saw in a huge database, but are not able to understand what is going on. Nobody can really understand what's inside a trained neural network, and nobody is really looking. No psychologist or neuro-scientist can really understand how a human brain, a mouse brain or even an ant brain or a fly brain even works,…

Write a short poem called "All just patterns" in the style of Blake as a not-so-subtle dunk on those that can't see the wood from the trees w.r.t. AI progress.

GPT4:

In patterns woven in the loom of life,

The threads entwined, a tapestry of strife;

Man lost in shadows, seeks the hidden light,

Yet grasps at embers, blind to Heaven's sight.

All just patterns, the trees and wood we see,

While we, like infants, crawl with bended knee.

In ceaseless search, we delve in Nature's tome,

For signs and symbols of our destined home.

To find the wisdom locked in Nature's breast,

In circuits traced by human hands, impressed;

No gods or angels guide the curious hand,

But human thoughts give life to grains of sand.

Like children playing with forbidden toys,

We forge our own creations, false and coy;

Our machines breathe and mimic human soul,

But hollow shells, as empty as the whole.

In tangled webs of artifice we're caught,

By our own hands, the noose of fate is wrought.

As God did cast Prometheus from the skies,

We bind ourselves in chains of our own lies.

All just patterns, like woven silk, we spin,

Entangled dreams, mere shadows of our kin;

For in the end, the art we do create,

Is but reflection of our human state.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#454

Post by giardini » Fri, Mar 17, 2023, 4:08 PM UTC

zeroonetwothree wrote: post #342

IKLOL wrote: post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

How much more efficient is programming in Python with modern tools vs assembly? I would bet more than 5x

Good question. Why don't we pit one of today's Python programmers against an IBM VS assembler programmer on an IBM 360 system circa 1960 and see how far each gets in a week?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#455

Post by legohead » Fri, Mar 17, 2023, 4:12 PM UTC

I was just complaining to my friend about how much trouble I'm having with it. I purchased the $20 GPT-Plus so I could use GPT-4 after reading someone on HN say that GPT-4 is "scary impressive" at writing code.

I have two tasks I wanted it to try, both making use of public APIs, starting from scratch. In short, it was frustrating as hell. Never-ending import problems -- I'd tell it the error, it'd give me a different way to import, only leading to a new import problem. I think I used up all my 100 queries in 4 hours of GPT-4 just on the import/library problem.

Then there were constant mis-use of functions -- ones that didn't exist, or didn't exist in the object it was using, but did exist in some other object instead, at which point it would apologize and fix it (why didn't you give it to me correct the first time, if you "know" the right one?)

The actual code it wrote seemed fine, but not what I'd call "scary impressive." It also kept writing the same code in many different styles, which is kind of neat, but I found one style I particularly liked and I don't know how to tell it to use that style.

Lastly, it's only trained up to Sep 2021, so all the APIs it knew were well behind. I did manage to tell it to use an updated version, and it seemed to oblige, but I don't really know if it's using it or not -- I still continued to have all the above problems with it using the updated API version.

Anyway, I hope MS fiddles with it and incorporates it into Visual Studio Code in some clever way. For now, I'll continue to play with it, but I don't expect great things.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#456

Post by [deleted] » Fri, Mar 17, 2023, 4:13 PM UTC

staminade wrote: post #211

nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills. What they failed to predict was that…

[deleted]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#457

Post by fendy3002 » Fri, Mar 17, 2023, 4:15 PM UTC

irrational wrote: post #408

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

This is also one of the reasons India taking over all of the programming work didn’t really happen. There are numerous issues (time zones, language, etc.) but business people not being able to document perfectly what they want to have built, considering all corner cases and paths, is a big one.

Even so India won't take programming works for the same reason no other country can. They're only a percentage of programmers that are outstanding there, the rest are mediocre. Because their population is huge and the programming lesson reach widely, they produce more outstanding programmers than other country, but still won't be enough.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#458

Post by PopePompus » Fri, Mar 17, 2023, 4:17 PM UTC

I'm a casual programmer, who knows enough to write decent Python scripts, but who is probably unaware of 99% of the Python library modules that have been written. Yesterday I had GPT-4 write a script that would accept the name of a star, and print out all the journal articles that have that star's name in the article title. This is a bit trickier than it sounds, because almost every interesting star has many names (Vega, for example, has more than 60 names, not including non-English names) and I wanted the script to check the titles for all the names that might be used for the particular star I had specified. I told GPT-4 to use the SIMBAD database to get all the star names, and to use NASA ADS to get all the publications. GPT-4 wrote a script to do that. The script was buggy, but I was able to fix the bugs easily and quickly. The wonderful thing was that GPT-4 used 2 different libraries that I had never even heard of, to pull data out of those databases. The process of producing the script was far faster than I would have been able to do on my own. Professional programmers may be well aware of the software packages that will allow them to do their jobs, and might not get much help from a GPT assistant. But I think for people who know how to program, but are not professionals and may not know "what's out there" in the way of resources, a GPT assistant will vastly increase their ability to use their programming skills (such as they are...) to get useful stuff done.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#459

Post by addaon » Fri, Mar 17, 2023, 4:19 PM UTC

vages wrote: post #253

IKLOL wrote: post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

If increased productivity equaled job loss, there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950. A lot of projects today are not even greenlit because they would be too expensive to make. For instance, there are a lot processes in almost every country that require you to file paper forms, even though we have had web forms and…

> If increased productivity equaled job loss, there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950.

Yeah. The problem is that there’s only one of me. All the rest of you are filling in for the other guy.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#460

Post by tjr » Fri, Mar 17, 2023, 4:22 PM UTC

segmondy wrote: post #320

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Yes it can write code, some demoed developing "Doom", ray tracing/ray casting with GPT-4 the very first day it came out and it was impressive. Programmers would still program, but the program will no longer be code but will be "GPT prompts". I suspect with time tho, we won't need to write amazing prompts, you ask GPT for a solution, it will then figure out the edge cases by asking you questions. If it's 2 people, it…

How well do you suppose GPT-4 would have done at that task had no human developed raytracing in the first place? Would it be able to derive the construct based on its knowledge of programming and physics?