Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

611–620 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#611

To be fair, in 15 years writing code, I have spent the vast majority of my time working on minor variations of problems already solved. I am not a fancy developer coming up with new algorithms. I make sign up flows, on-boarding flows, paginated lists, integrations to other apis. And I definitely feel that my job might be threatened by LLMs.

Whenever these kinds of comments are made, the short story “Profession” comes to mind by Isaac Asimov. In this story people’s aptitudes are evaluated and the relevant knowledge and skills are downloaded. The protagonist of the story however keeps being rejected for download and has to struggle to acquire the same skills his peers acquire instantly and magically. It’s a great read with a fantastic ending. The morale i…

The unique hard-won skills are what AI can do on the cheap. The more high-level interesting stuff is just creativity, and creativity is a universal human trait. It’s amazing but low value (monetarily).

As you often hear on HN, ideas are a dime a dozen it’s all about execution.

Well we’re rapidly approaching the time when the execution is essentially free, and done faster and better than humans.

A small team of four, over an afternoon, can literally just speak with the computer to generate a new TV ad, or develop a new sass product. There is no longer any skill required, just imagination. The problem being of course that the skills and specialized knowledge are what people have been traditionally paid for.

With all that “work” out of the way there’s not much value anyone can add . You’re probably not any smarter or creative than whoever’s manning the machine.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#612

Earlier quoted context omitted.

We all need to save up money and think of a plan B. If there is no problem, worst case you'll have a bunch of money saved and a plan B you won't be using.

That’s not really an option for a lot of people outside of the US. The average SWE here makes 50K USD a year. Good luck saving up a safety net that large if AI is ready to take all our jobs. That’s kind of my point. Nobody has a good answer to the question IF AI does take SWE jobs, what then?

It really depends where you live I agree. If you live in Europe there's usually a generous unemployment benefits for quite awhile and the government will probably pay for you to learn a new skill. That's not a great scenario but its much better than if you live in India.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#613
post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

> Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors. That is already possible without AI and has been the case for a long time... the issue is nobody will stay to be that highly paid senior running entire projects because at that point you can just run your own shop and pocket the full profits.

I wanted to say the same thing - except that the senior can't actually perform 10x because they're too busy trying to train the next gen devs while attempting to deliver stories themselves in the fractions of time available to them.

Not to say that this is a bad thing, but the difference between a junior and senior is often much more than the difference in their salary.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#614
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

The counter argument here is building a new codebase to solve a problem that a single engineer can articulate and map out.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#615

> After going in circles a few more times, I decided that was it. It got close . It seemed to understand the problem, but it could not actually properly solve it. I had this same loop issue with Chat-GPT. I had something I wanted to do with asyncio in Python. That's not something I work with much so I thought I'd see if Chat-GPT could help me out. It was actually good at getting me up to speed on ansycio and which pa…

This here is my fear to. I have a buddy right now that is getting his degree in CS and he is using ChatGPT for a lot of his assignements. I worry that the next generation of developers are going to grow up just figuring out how to "program GPT" and when they have an error rather than investigating it (because they can't because they aren't actually familiar with code in the first place) they'll simply tell GPT about…

I mean, we already kinda do that with Stack Overflow and cargo cult programming.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#616
post #606

Earlier quoted context omitted.

But that’s never happened in the past. Why should it happen now? The Industrial Revolution is used as an example because it was the biggest spike in productivity, but the general trend has been the same since the invention of tools. Agriculture, another perfect example, lead to others having time to produce things instead of hunting/gathering. It’s easy to grasp with the agriculture example that “unemployment” wasn’t…

> Why should it happen now? What are S-curves.

Perhaps the quantity of labour utilizing each new technology through time is a (n-shaped) parabola that intersects with the technology it replaced.

The fear that technological advances will cause mass unemployment and destroy labour markets has been common throughout history. Yet here we are at full employment. Maybe this time is different?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#617

I was just complaining to my friend about how much trouble I'm having with it. I purchased the $20 GPT-Plus so I could use GPT-4 after reading someone on HN say that GPT-4 is "scary impressive" at writing code. I have two tasks I wanted it to try, both making use of public APIs, starting from scratch. In short, it was frustrating as hell. Never-ending import problems -- I'd tell it the error, it'd give me a different…

I think the current train of thought is "keep increasing the size of the language model and you don't need to worry about integrating with LSPs".

Perhaps there is some merit to this. If the language model is large enough to contain the entirety of the documentation and the LSP itself, then why bother integrating with the LSP? _Especially_ if you can just paste the entirety of your codebase into the LLM.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#618
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

It can sit there and communicate with people endlessly. In fact, that is what it's good at.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#619
post #534
post #439

Earlier quoted context omitted.

Have no doubt, we will find new places to put computers. In the 80's and even the 90's everyone said the same thing, "Why do I need a computer? I can do everything I do already without a problem?" Well, turns out with computers you could do 12 more things you can never considered. Consider the interoffice memo: it'd take what, 1-2 hours to get a document from one floor to another through the system? Cool, you can wor…

I dunno, I feel like we've been trying to find new places to put computers for a couple decades now, and the effort is kind of losing steam. Every week there are threads about how useless a lot of these efforts are.

You just don't hear about all the computers, that's all. There's one on your car's key fob, and likely 100 more in the car. Your coffee grinder has one. So does your dishwasher. And your electric blanket.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#620
post #611

Earlier quoted context omitted.

Whenever these kinds of comments are made, the short story “Profession” comes to mind by Isaac Asimov. In this story people’s aptitudes are evaluated and the relevant knowledge and skills are downloaded. The protagonist of the story however keeps being rejected for download and has to struggle to acquire the same skills his peers acquire instantly and magically. It’s a great read with a fantastic ending. The morale i…

The unique hard-won skills are what AI can do on the cheap. The more high-level interesting stuff is just creativity, and creativity is a universal human trait. It’s amazing but low value (monetarily). As you often hear on HN, ideas are a dime a dozen it’s all about execution. Well we’re rapidly approaching the time when the execution is essentially free, and done faster and better than humans. A small team of four,…

Care to place a wager where I give your mythical unskilled team the best AI available today against me without AI and we’ll see who can solve a difficult engineering problem? Heck, even a moderately skilled team.

Sorry no. AI is impressive but I’ve given it very precise prompts where I describe exactly what I want it to do because I’ve already solved it and the solution it generates is complete and utter horseshit because it requires context that’s too difficult to communicate and a deep understanding of the business and technical aspects of that business. Similarly, novel ideas are not things it knows how to implement.

If you have a counter example I’d love to see it because my experience seems to line up pretty well with other reporting of where the limits of it lies - ie it can regurgitate solutions to solve problems but struggles to provide solutions and correct implementation. In fact, trying to find the problems is itself even harder sometimes because the way it solves things is simultaneously not a good coder and the approach it takes isn’t one a human would and thus it takes extra effort to figure out what path it’s trying to take and where it made a mistake.

As an example. Try to get ChatGPT to implement the server-side implementation of R2’s ListObjects (or heck - in any language / platform you choose, implement that). It’ll make really bad bugs like reading in the entire dataset into memory, not applying the delimiter properly in really subtle ways etc etc. basically, it can’t even do a usable first draft. Just don’t use go because I suspect it’ll cheat and just regurgitate minio

Post reply on HN