Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

351–360 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#351
I was mock interviewing ChatGPT for a few hours yesterday with application and system design + coding said application. My conclusion was it was a no hire for even the most jr positions because it required considerable amounts of direction to arrive at anything approximating an acceptable solution.

tldr -- this matches my experiences as well.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#352
post #264

Earlier quoted context omitted.

How does this work from an IP perspective? Aren't you feeding OpenAI and Microsoft with your confidential information? Also, what license is the code under when you get it back?

I take care not to feed it secrets ; this is just boring ERP stuff without the magic numbers (they are not needed to create or test; we use test data normally as well, as we cannot give that to outsourcing companies either, so there is no difference in work).

And the output? Are you allowed to use their code for your own projects or company?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#353
I'm sure I'll change my mind as this tech improves, but having AI generate code goes against every instinct I have. Way too easy for there to be a subtle bug in the code among other problems. It makes me wonder though if AI could useful for writing tests of my code. And also AI code review.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#354
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

To paraphrase Harold Abelson: Computer Science is not really very much about computers. And it’s not about computers in the same sense that physics isn’t really about particle accelerators, and biology is not really about microscopes and petri dishes. It is about formalizing intuitions about process: how to do things [0]. [0]: https://www.driverlesscrocodile.com/technology/the-wizard-1-... .

That's irrelevant though, because actual programming is very much about computers, and about writing things the majority of which have already been formalized by computer scientists, and gluing code and programs inti pipelines...

Re: Testing GPT 4's code-writing capabilities with some real world problems

#355

Earlier quoted context omitted.

"I'm still waiting for the automobile that can replace me, the Horse Carriage Driver. They've been promising those since the 1880s, but I've still got passengers in my carriage every day!" Obviously it's a cheeky example, but this would not be the first time in history a previously well-established career was upended in a (relatively) short amount of time. I'm a FAANG dev, I've got skin in the game too and I'm trying…

> All that being said, if you've been working since the 2000s, you've got nearly a 20 year head-start on me, so perhaps it makes sense for me to be a bit more worried. Yea, that's mostly why I get hired. Experience gives people a certain intuition on what kind of solutions work for which cases. And when you've been working long enough, you don't (hopefully) feel the need to do cool bleeding edge shit at work, you jus…

That's a comforting perspective, I hope that you're right.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#356
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

> 99% of the effort is NOT WRITING CODE

Writing code is still part of the job. In my company, I'd say it's still very roughly 50% of the job. If i can be a bit more efficient thanks to GPT, it's great. Actually, I already use it for writing simple things in language I'm not proficient with. Or how to improve a particular piece of code that I know can be rewritten in a more idiomatic way. It's not perfect, but I've found it useful.

It's not going to replace SWEs, but it's going to make us more productive.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#357
post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

How much more efficient is programming in Python with modern tools vs assembly? I would bet more than 5x

That's an interesting case and use of the word efficient.

Developer time reduced by 5X CPU time increased by 500x

When CPU cycles are cheap it's a logical tradeoff but it still grates against my sensibilities.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#358

Earlier quoted context omitted.

"I'm still waiting for the automobile that can replace me, the Horse Carriage Driver. They've been promising those since the 1880s, but I've still got passengers in my carriage every day!" Obviously it's a cheeky example, but this would not be the first time in history a previously well-established career was upended in a (relatively) short amount of time. I'm a FAANG dev, I've got skin in the game too and I'm trying…

> I'm still waiting for the automobile that can replace me, the Horse Carriage Driver. Wow. The hype around "AI"s has entered the crypto stages. Just like crypto was the new car, the new internet etc., so is AI now. Well, at least AI is useful in certain applications.

I got into and out of Bitcoin back in 2010 when I was still in high school. Even then, I thought and still do that crypto as a whole is a complete ponzi scheme with little real world utility.

I feel completely differently about LLMs; I'd say we're closer to 2007 when the first iPhone was released. I believe LLMs will become a part of our day to day lives at a level that crypto never came close to.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#359

Earlier quoted context omitted.

For me, its mostly that I have used GPT-3.5 a little for programming C++, and I wasnt impressed. For one, it made horrible, glaring mistakes (like defining extern functions which dont exist, using functions which are specific to a platform im not using, etc.), stuff beginners would do. It also decided to sneak in little issues, such as off-by-one errors (calling write() with a buffer and a size that is off by one in…

Two points: GPT4 is significantly better in this regard, and you should be concerned about the rate of progress more than it’s actual capabilities today.

> you should be concerned about the rate of progress

Kind of agree?

On the one hand we don't even have a roadmap toward reliable AI.

On the other, if we ever plug an LLM into something that has memory, acquires experiences, does experiments, observes the outcome and adjusts its worldview in response, consciousness might fall out of that. And writing good code might not even require consciousness.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#360

I'm sure I'll change my mind as this tech improves, but having AI generate code goes against every instinct I have. Way too easy for there to be a subtle bug in the code among other problems. It makes me wonder though if AI could useful for writing tests of my code. And also AI code review.

That's what this beta extension for vscode[0] does, generate tests using AI and your code as input.

[0] https://about.sourcegraph.com/blog/release/4.4

Post reply on HN