Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

641–650 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#641

Post by marstall »

nemo44x wrote: post #254
IKLOL wrote: post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

Wouldn’t this actually create more demand for programmers? More businesses can exist that have a need for software. To date every advance in programming efficiency has resulted in more tech, not less. Additionally there’s the math behind it. If Company A fires 50% of their staff because AI lets the remaining 50% work at twice the productivity then how will they compete with Company B that keeps their staff and now ge…

yes - and in another way, too. a lot of the demand for programmers is driven by successive waves of new technology adoption and investment - and AI is looking to be a motherload that should keep us going for awhile.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#642

Post by runeks »

insomagent wrote: post #617
legohead wrote: post #455

I was just complaining to my friend about how much trouble I'm having with it. I purchased the $20 GPT-Plus so I could use GPT-4 after reading someone on HN say that GPT-4 is "scary impressive" at writing code. I have two tasks I wanted it to try, both making use of public APIs, starting from scratch. In short, it was frustrating as hell. Never-ending import problems -- I'd tell it the error, it'd give me a different…

I think the current train of thought is "keep increasing the size of the language model and you don't need to worry about integrating with LSPs". Perhaps there is some merit to this. If the language model is large enough to contain the entirety of the documentation and the LSP itself, then why bother integrating with the LSP? _Especially_ if you can just paste the entirety of your codebase into the LLM.

> If the language model is large enough to contain the entirety of the documentation and the LSP itself, then why bother integrating with the LSP?

If your goal is to get a response to an LSP query, why on earth would you use an LLM trained on data where >99.9999% of that data has nothing to do with answering an LSP query?

Why would I switch out an LSP server of 100% accuracy for an LLM that’s slower and has lower accuracy?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#643

Post by teaearlgraycold »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

That's a cheap 1% boost!

Re: Testing GPT 4's code-writing capabilities with some real world problems

#644

Post by mejutoco »

anileated wrote: post #384
slfnflctd wrote: post #375

Earlier quoted context omitted.

My intuition here is that it's because people don't always say what they mean, or know how to describe what they want. I've been working on a database migration recently, and I look forward to the rare moments when I get to write queries and analyze actual data. The vast majority of my billable hours are spent trying to tease out the client's needs by going over the same ground multiple times, because their answers k…

> Especially when a higher-ranking person comes back from vacation and says "no, everything you nailed down in the last two weeks is completely wrong". Yes, and at some point this high-ranking person is fed up with this now-inefficient use of time and money enough that they will just sort this out using an LLM tuned to handle this situation better if not today then tomorrow. Maybe they will pay someone to coach them…

Imagine how angry this hypothetical person will be when they get the same problem from the LLM, after all those extra steps.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#645

Post by ThreeToZero »

anonymoushn wrote: post #173
ThreeToZero wrote: post #4

A modified A* that solves the fire routing problem (less efficiently than OP's I think). Each A* location stores where it comes from, how long it takes to get to it, and how many fires it passed through to get there. The algorithm only considers fire cells neighbors if the current number of fires passed through is less than the current fireWillingness global. 1. count fire tiles within movement range 2. run A* from s…

Counterexample: ...XX SF.FD ...XX S = start F = fire X = wall D = destination The cat can to the destination in 6 moves passing through 1 fire. In the fireWillingness=1 pass, the middle tile is reached after passing through fire, so the destination appears unreachable. The proposed algorithm will pass through 2 fires instead of 1.

Haha good counter example. Well played

Re: Testing GPT 4's code-writing capabilities with some real world problems

#646

Post by lamp987 »

yucky wrote: post #639
lamp987 wrote: post #420

Earlier quoted context omitted.

"Yes, the industry is going to shrink massively" You meant "expand massively" i think. Did all the programmers manually making machine code get fired and the job of programmer disappear when compilers were invented and totally replaced these jobs? No, it just changed to use the new tool. There wont be any unprecedented mass layoffs, despite what the jaded STEM-hating twitter crowd wants. Companies will simply make th…

> You meant "expand massively" i think. Nope, I meant what I said. > There wont be any unprecedented mass layoffs Baghdad Bob strikes again.

Then youre wrong. There is no market incentive for layoffs due to GPT-like technology, as I have demonstrated above. Similar breakthroughs "replacing jobs" have happened before in the field of software engineering, this is nothing new or unprecedented. Its merely another tool that will become in widespread use to increase production.

What i think will lead to mass layoffs is the current recession rather.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#648

Post by boringuser1 »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

[dead]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#649

Post by quonn »

ianmcgowan wrote: post #535
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

If the us/they back and forth happens over email, perhaps between two different AI instances, that whole process would happen much faster though? It's not like ChatGPT can't review the contract json and ask relevant questions. Granted, the problem solving part might be delegated to a human, but the purely routine back and forth part seems already possible?

Maybe some day. But I tried it just now on a database design I‘ve been working on for two months and it spits out something superficially close immediately from a two sentence prompt. On one hand that’s impressive, it‘s interesting and somewhat correct but all the interesting parts are missing or wrong and it never get‘s beyond that, not even with my help. No sane person would answer so confidently yet superficially useless.

A sane approach would be to start understanding the requirements and work from there, trying to figure out where the challenges are.

GPT can‘t do this currently.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#650

Post by hungryforcodes »

sgarland wrote: post #215
Jensson wrote: post #135

Earlier quoted context omitted.

Quoting famous people sounds smart until you realize they just memorized a ton of trivia. These models have demonstrated that they don't learn logical models, instead they learn to generate text that looks logical at first glance but is nonsense.

I asked GPT-4 something fairly niche that I happen to know a fair amount about: to explain the concept of Xenon poisoning in a nuclear reactor. Other than skipping Te-135 being the initial fission product that starts the decay chain (and tbf, operationally it can be skipped since the half-life is 19 seconds), it got everything correct. I'm sure if I kept probing on smaller and smaller details it would eventually fail…

Ok, but it has all this information already. So why is this surprising.

A real test of it's abilities would be synthesizing a NEW type of nuclear reactor.

Post reply on HN