Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

221–230 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#221

Post by pif »

eggsmediumrare wrote: post #199
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

Company XXQ to LLM once they figure their shit out: here is the endpoint we need. LLM: gives code. You: not involved.

Maybe you did not get the point: "once they figure their shit out" is the pain point, and no ChatGPT can ease that!

Re: Testing GPT 4's code-writing capabilities with some real world problems

#222

Post by shp0ngle »

it_citizen wrote: post #3

To be fair, in 15 years writing code, I have spent the vast majority of my time working on minor variations of problems already solved. I am not a fancy developer coming up with new algorithms. I make sign up flows, on-boarding flows, paginated lists, integrations to other apis. And I definitely feel that my job might be threatened by LLMs.

Your job is already threatened by cheap outsourcing.

However, the risk with cheap outsourcing is exactly the same as with LLMs - you get what you pay for, and you need to constantly check if it's really doing what it's supposed to be doing.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#223

Post by sumtechguy »

caddemon wrote: post #68
dragonwriter wrote: post #40

Earlier quoted context omitted.

> I think most people see where the puck is going Overinflated hype about “where the puck is going” being wrong is…not a new phenomenon. And the non-tech media (traditional and social, and particularly a whole lot of the elite commentariat that spans both more than the “hard” news side of traditional media, though that is influenced too) perspective on this is driven quite disproportionately by the marketing message…

You know the shoeshine boy indicator story [1]? That's the vibe I've been getting with ChatGPT lately, I have multiple old high school teachers with 0 computational background who hardly ever spoke about ML before posting regularly on LinkedIn about ChatGPT the last couple months. The part that gets me is they aren't just aimlessly making posts, they're getting involved with webinars targeted at educators in their re…

I can see why educators are getting involved. They are seeing it change how their students do work in real time.

I grew up and using a pocket calculator was verboten. You just did not do it. You better learn and memorize all of that stuff. Spin on 10 years after me and all kids have them now. If you have the right app on your phone the thing will OCR the problem and auto solve it and show you the steps. ChatGPT and its ilk are here, now. How we learn and create things has dramatically changed in under a year. It is not clear how much though.

Teachers are trying to get ahold of what does it mean to teach if you can just ask some device to summarize something as complex as the interactions of the 6 major countries in WWII and what caused it. Then the thing thinks for 2 seconds and spits out a 1000 word essay on exactly that. Then depending on which one you use it will footnote it all and everything.

This style of learning is going to take some getting used to. This tool is in their classrooms right now. Right or wrong. It will be there. The teachers are going to have to figure out what this means to their class planning. Not 5 years from now, today.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#224

Post by pydry »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

I'm pretty sure you may be right. I'm also worried that what youve just described is the kind of task that leads to burnout in large doses. And I'm not sure humans are so great at it either.

I had one job that involved a small amount of coding and mainly hooking together opaque systems. The people behind those systems were unresponsive and often surly. I had to deal with misleading docs, vague docs, subtle, buried bugs that people would routinely blame on each other or me and I was constantly on a knife edge a balancing political problems (e.g. dont make people look stupid in front of their superiors, dont look or sound unprepared) with technical concerns.

It was horrible. I burned out faster than a match.

I'm sure ChatGPT couldnt do that job but I'm not sure I could either.

If most tech jobs turn into that while the fun, creative stuff is automated by ChatGPT... that would be tragic.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#225

Post by the_af »

The first problem statement that GPT got wrong actually shows a problem with human language.

"Avoid fire" means "do not ever go through fire" to me (and GPT thinks the same, apparently). The author thought it meant "avoid fire if you can, but go through it if there's no other way". This was a problem with informal requirements that could have happened in an entirely human context.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#226

Post by ethanbond »

anonyfox wrote: post #195
guax wrote: post #189

Earlier quoted context omitted.

I have the same feeling, people are very concentrated on the ability of this AI generators to create working code from super specific and well formed prompts. When in reality, figuring out what the prompt should be accounts for 80% of the job.

don't fall into this mental trap. you can get into recursion quite easily here, and figuring out what to prompt can start from simple general questions - and there is no need for a developer at all, aside from the current limitations of copy/paste/run workflow has to be done manually

Have you ever worked at a software firm of greater than say 30 people? Doesn’t resonate with my experience at all, and the 30+ people are there not just to write code.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#227

Post by sgarland »

MrDresden wrote: post #80
pewpew2020 wrote: post #47

Earlier quoted context omitted.

I feel it's the average programmer who is enamoured with GPT. The programmers working on hard stuff know that AI writing programs is still far away.

It is not just those who are working on hard problems who are of that opinion. Good luck having a ML model understand a 20 year old undocumented dataformat developed inhouse at a specific research lab to be used in their proprietary systems which are also undocumented and are a part of a spiderweb of interconnected systems at that lab (I have worked at this particular lab). It will be a long time (if ever) until a ML…

Have you tried giving it a skeleton example of said undocumented data format? I've created some arbitrary ones and asked it to parse them, and it succeeded.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#228

Post by Quarrelsome »

staminade wrote: post #211
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

While I think there's truth to what you say, I'd also point our that workers in many pre-automated industries with an "artisan" approach also considered themselves irreplaceable because they figured, correctly, that nobody could build a machine with the capability of reproducing their workflow, with all its inherent uncertainty, flexibility and diverse physical and mental skills. What they failed to predict was that…

That's an extremely interesting thought. Perhaps we will see organisations in the future structure themselves more like a suite of unit tests. Instead of getting a developer or software house to plug a specific need in their org and being entirely outside of the development process: they will reflect the development process organisationally to ensure they catch any problems with the current output and just feed the box new prompts to increase their efficiency or efficacy.

Their competitive advantage in their field then becomes the range of their tests (borne through experience), efficiency in running their pipeline of testing and ability to generate effective prompts.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#229

Post by IKLOL »

quitit wrote: post #200
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

This is so common in many types of business, and usually a very difficult point to articulate so thank you for that. It's something to be shown to those ringing the death-knell for programmers, artists, and the like. Those death-knell types seemingly aren't aware of what day to day operations looks like and how AI makes a great tool, but doesn't necessarily deal with the very human factors of whims, uncertainty, reac…

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most.

Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#230

Post by theshrike79 »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

I once had to implement a Swedish standard for energy usage reporting.

EVERY FIELD IN THE STANDARD WAS OPTIONAL.

That was one of the most not-fun times I've had at work :D Every single field was either there or it was not, depending whether the data producer wanted to add them or not, so the whole thing was just a collection of special cases.

Post reply on HN