Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I'm tired of fixing customers' AI generated code

medium.com

Post reply on HN
Print view View original

131–140 of 345 posts

Re: I'm tired of fixing customers' AI generated code

#131

Post by Pikamander2 »

> Often this takes the form of trying to access an endpoint that does not exist, or read a property off the API response that does not exist. After probing a bit more, my suspicions are usually confirmed — ChatGPT hallucinated that endpoint or property

In some cases, you might be able to use this to your advantage to improve the product.

When working with third-party APIs, I've often run into situations where my code could be simplified greatly if the API had an extra endpoint or parameter to filter certain data, only to be disappointed when it turns out to have nothing of the sort.

It's possible that ChatGPT is "thinking" the same thing here; that most APIs have an X endpoint to make a task easier, so surely yours does too?

Over time I've sent in a few support tickets with ideas for new endpoints/parameters and on one occasion had the developer add them, which was a great feeling and allowed me to write cleaner code and make fewer redundant API calls.

Re: I'm tired of fixing customers' AI generated code

#132

Post by boredhedgehog »

FanaHOVA wrote: post #83
alexeiz wrote: post #55

I had a related episode at work when my coworker asked me why his seemingly trivial 10 line piece of code was misbehaving inexplicably. It turned out he had two variables `file_name` and `filename` and used one in place of another. I asked him how he ended up with such code, he said he used copilot to create it. Using code from a generative AI without understanding what it does is never a good idea.

Copilot wouldn't make a typo. He just made that up and / or broke the code himself.

What likely happened is that he asked the AI two separate questions and fused the answers himself.

Re: I'm tired of fixing customers' AI generated code

#133

Post by throwaway2037 »

dylan604 wrote: post #90
fragmede wrote: post #70

Earlier quoted context omitted.

not "at large", but there's very much a segment of smart people who's expertise lies elsewhere, and they just haven't taken the time to learn the basics of programming. as someone who's spent a lot of time programming, I love meeting brilliant people who could program but don't just because they've gone a different way.

Some people forget how easy programming can be when you know nothing and just try stuff to see what works. Working in a procedural manner with everything in a global scope is simple to get stuff working. Not everything has to be extrapolated out into namespaces, functions, classes. It's nightmare code to maintain later, but going from blank page to working code is totally possible. I think sometimes we forget not eve…

I agree with you. In the 2000s, what you describe was normal for Excel/VBA. The trick to learn VBA from nothing, was to use the macro recorder, then slowly modify the code. And, arguably, Excel formulas was/is functional programming.

Re: I'm tired of fixing customers' AI generated code

#134

Post by varjag »

brigadier132 wrote: post #109
FanaHOVA wrote: post #83

Earlier quoted context omitted.

Copilot wouldn't make a typo. He just made that up and / or broke the code himself.

You are getting downvoted but you are right, a typo in a variable that already exists in a file like this is not the failure mode for LLMs. The failure mode is logic bugs, making up methods / functions.

No, you can get variable names "mutated" on follow up requests. The thing is like sculpting with toothpaste.

Re: I'm tired of fixing customers' AI generated code

#135

Post by fzeroracer »

FanaHOVA wrote: post #126
creesch wrote: post #113

Earlier quoted context omitted.

Oh for crying out loud, I obviously mean these specific mistakes. If you have worked in any capacity with LLMs like this you would have seen them variables or suddenly switch up the convention of how they're written. Certainly if you are in a conversation mode after a few back and forths this happens from time to time. I am just not going to spend my time digging to previous prompts of code I might not want to share…

[flagged]

Not the OP. I have certainly seen LLM coding tools generate blocks of code with misspelled variables and typos. Trying to shove someone into a box of being a cynic because they have had bad personal experiences with tools is a good way to ensure people filter out your opinions.

Re: I'm tired of fixing customers' AI generated code

#137

Post by n4r9 »

f6v wrote: post #129
frumper wrote: post #60

Earlier quoted context omitted.

I knew a guy that made a good living as a freelance web developer decades ago. He would pretty much just copy and paste code from tutorials or stack overflow and had no real idea how anything worked. Using code without understanding it is never a good idea, it doesn’t need to be from AI for that to be true.

Or maybe you’re just exaggerating. I’ve done my fair share of copy pasting and it never worked to just do it without understanding what’s going on. I think the problem with “AI” code is that many people have almost a religions belief. There’re weirdos on internet who say that AGI is couple years away. And by extension current AI models are seen as something incapable of making a mistake when writing code.

The other downside to AI code vs stackoverflow is that a stackoverflow post can be updated, or a helpful reply might point out the error. With the advent of LLMs we may be losing this communal element of learning and knowledge-sharing.

Re: I'm tired of fixing customers' AI generated code

#139

Post by fzeroracer »

bdcravens wrote: post #22

This seems like a support issue, not an AI issue. AI is how the code was written, but the issue would be the same if it was amateurs writing bad code. If all you want to do is support your API, a support article outlining the issues you see over and over would be something to point your customers to. Warrant your API against errors, but point out that anything more is billable work. If you're not interested, partner…

The difference is scale. I don't know how many times people need to say this, but LLM tools enable people to spam low quality code at a rate that is far faster than ever.

There's been multiple stories and posts here on HN about issues with AI generated PRs for open source repos because of people using them to game numbers or clout. This is a similar problem where you have a bunch of people using your API and then effectively trying to use you for free engineering work.

Re: I'm tired of fixing customers' AI generated code

#140

Post by Mashimo »

creesch wrote: post #113
brigadier132 wrote: post #111

Earlier quoted context omitted.

This is a strawman, he never said it didn't make mistakes.

Oh for crying out loud, I obviously mean these specific mistakes. If you have worked in any capacity with LLMs like this you would have seen them variables or suddenly switch up the convention of how they're written. Certainly if you are in a conversation mode after a few back and forths this happens from time to time. I am just not going to spend my time digging to previous prompts of code I might not want to share…

I, for one, have not have this experience with LLM creating new variable names when they already defined one.

Lots of mistakes, but never this one.

Post reply on HN