Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

341–350 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#341

Post by dureuill »

syntheweave wrote: post #162
scg wrote: post #19

As a human programmer I didn't quite understand the problem statement until I read the whole article and the tests. I believe the goal is to find a path with the fewest possible "fire" cells and the minimum cost as a tie breaker. The cost of a path is the sum of its cells' cost and it can't be greater than 5. If I understood the assignment correctly, I don't think the problem statement is equivalent to what's include…

The problem is, indeed, that Mr. Glaiel did not know the category of problem he was dealing with. A correct statement would be: "Given a solution set containing both the shortest path through fire and the shortest path avoiding fire, select the solution that fits within six tiles of movement, preferring the solution that avoids fire where possible." It's a constraint optimization problem in disguise: generate a solut…

> And it might be a bit faster, but you are really working for the paycheck that day when the obvious thing is to run the basic A* algorithm twice with different configuration steps.

Pretty much this. Attempt to find a path to the target destination with a first A* run that disregards fire tiles, and if that fails due to limited movement, then do a second run with the fire tiles. I like that this mirrors the decision making a human would follow, too: I won't cross the fire tile unless I'm absolutely required to.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#342

Post by zeroonetwothree »

IKLOL wrote: post #229
quitit wrote: post #200

Earlier quoted context omitted.

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

How much more efficient is programming in Python with modern tools vs assembly? I would bet more than 5x

Re: Testing GPT 4's code-writing capabilities with some real world problems

#343

Post by dmm »

sequoia wrote: post #290

One other question: can GPT-4 reliably modify code? In A Philosophy of Software Design the author points out that code is written once and modified possibly dozens of times, so ease of maintainability/reading is more important than ease of writing. I wonder whether a) AI can reliably modify code and b) whether AI can reliably write code that is able to be easily modified by humans. If AI starts spitting out machine c…

> so ease of maintainability/reading is more important than ease of writing.

That may be the case now but in a theoretical future where software systems are generated by AI why would I bother modifying the old system? Why not generate a new one with the original prompts modified to meet the new requirements?

In a sense the "source code" of the system could be the AI model + the prompts.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#344

Post by coldtea »

vages wrote: post #253
IKLOL wrote: post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

If increased productivity equaled job loss, there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950. A lot of projects today are not even greenlit because they would be too expensive to make. For instance, there are a lot processes in almost every country that require you to file paper forms, even though we have had web forms and…

>If increased productivity equaled job loss there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950.

The only reason it's not the case in this example is because computers at the time were a tiny early adopter niche, which massively multiplied and expanded to other areas. Like, only 1 in 10,000 businesses would have one in 1950, and only big firms would. Heck, then 1 in 100 million people even had a computer.

Today they've already done that expansion into all businesses and all areas of corporate, commerce, and leisure activities. Now almost everybody has one (or a comparable device in their pocket).

Already cloud based systems have made it so that a fraction of programmers and admins are needed. In some cases eliminating the need for one altogether.

There are tons of other fields, however, more mature, where increased productivity very much equaled job loss...

Re: Testing GPT 4's code-writing capabilities with some real world problems

#345

Post by Jeff_Brown »

Quarrelsome wrote: post #228
staminade wrote: post #211

Earlier quoted context omitted.

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills. What they failed to predict was that…

That's an extremely interesting thought. Perhaps we will see organisations in the future structure themselves more like a suite of unit tests. Instead of getting a developer or software house to plug a specific need in their org and being entirely outside of the development process: they will reflect the development process organisationally to ensure they catch any problems with the current output and just feed the b…

Tests and type signatures. Hopefully soon, dependently typed signatures.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#346

Post by maxwell »

segmondy wrote: post #320
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Yes it can write code, some demoed developing "Doom", ray tracing/ray casting with GPT-4 the very first day it came out and it was impressive. Programmers would still program, but the program will no longer be code but will be "GPT prompts". I suspect with time tho, we won't need to write amazing prompts, you ask GPT for a solution, it will then figure out the edge cases by asking you questions. If it's 2 people, it…

Seems like just a higher level of abstraction: prompts become input for generating high-level language code output.

It's an electric bicycle for the creative mind (how long until the first one-person unicorn?), I don't anticipate much success for those trying to use it as a self-driving car for half baked ideas.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#347

Post by 2-718-281-828 »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

wouldn't it just spit out a solution every time without a care in the world? that sort of messy communication is only bothersome for humans because writing code takes time and effort plus it's mentally taxing to change requirements several times. also boredom is a big issue. none of those challenges are relevant for a computer.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#348

Post by coldtea »

korijn wrote: post #314
IKLOL wrote: post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

If as GP says AI could automate 1% of a single programmer's job (the boring part where you write code), then how on earth can you derive that a single programmer could do the job of 5 with AI? It's completely illogical.

The GP estimate is way off...

If they indeed wait for input from other departments/companies 99% of the time (so they just need to actually program 5 minutes in their 8-hour workday), then they can be already thrown out of a job and have the company do with 1/10 the programmers, no AI required...

Re: Testing GPT 4's code-writing capabilities with some real world problems

#349

Post by justinzollars »

GPT 3.5 helped me debug something very complex. There was a bug related to symlinks in neovim with gopls LSP. The error dialog line was appearing, then disappearing.

Chat GPT walked me through strategies to debug this, confirm everything was set up, tail the RPC log (wasn't aware that was a feature) - and identify the failing path - which was a symlink!

I'm actually blown away by this capability. It was like having a savant next to me. I couldn't have debugged it on my own.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#350

Post by zeroonetwothree »

Jevon23 wrote: post #279
caleb-allen wrote: post #236

Earlier quoted context omitted.

It's astonishing to see the goalposts move so quickly. The cope of "well, okay, it can do that, but that's not even the hard part!" when just a year ago this entire product was almost unimaginable.

I don’t think anyone is claiming to not be impressed. Yes, we’re all impressed. This was unimaginable sci-fi just 5 years ago. You did it, we’re impressed! You won. The next step after that is for people to figure out how to justify their continued relevance. I think that’s a pretty natural reaction.

I’m not concerned. If AI replaces my job then ok I’ll find something else to do. Self driving cars seem to have stagnated so maybe I’ll be an Uber driver
Post reply on HN