I'm tired of fixing customers' AI generated code • Hacker News Board

Re: I'm tired of fixing customers' AI generated code

#181

Post by stavros » Thu, Aug 22, 2024, 9:21 AM UTC

sim7c00 wrote: post #175

IanCal wrote: post #160

Earlier quoted context omitted.

The description here isn't about using LLM hallucinations as code. It's using them as example API users and seeing what they get stuck on. Let's say you've got some rest API. Are they trying to send PATCH requests but you only support POST? Do they keep trying to add pagination or look for a cursor? Do they expect /things to return everything and /things/id to return a single one but you have /things and /thing/1 ? N…

Ah, sorry, totally didn't get that it was using them as users of the API. I would recommend API fuzzers here, which cycle through all available possibilities, and generate 'garbage' inputs as well as structured inputs regarding the specifications the APIs implement (http, and the underlying structures provided to the api endpoints). an LLM would likely not perform an exhaustive test. There are several projects freely…

> Ah, sorry, totally didn't get that it was using them as users of the API.

Originally, it wasn't. From the GP post:

> If there is a common hallucination API call, just add that API. This will make the API more "friendly" and resemble common similar APIs.

If it's just users, you wouldn't be adding the APIs.

Re: I'm tired of fixing customers' AI generated code

#182

Post by OJFord » Thu, Aug 22, 2024, 9:23 AM UTC

I think this is the point to establish a 'community', and in particular to only 'offer' community support (or charge more for spending your own time on it).

Some people will enthusiastically fix other customers' generated crap for free, let them.

Re: I'm tired of fixing customers' AI generated code

#183

Post by stavros » Thu, Aug 22, 2024, 9:23 AM UTC

TillE wrote: post #3

I'm always a little surprised at how many people out there want to develop software yet haven't put in the effort to gain even the most basic computer nerd programming chops. You see this all the time in the more newbie-friendly game engine communities. Maybe you don't want to pursue a career in software, but anyone can spend a week learning Python or JavaScript. I suspect/hope a lot of these people are just kids who…

> I'm always a little surprised at how many people out there want to develop software yet haven't put in the effort to gain even the most basic computer nerd programming chops.

If you're surprised by reality, that says something about your mental model, not about reality. People don't want to "learn programming", they want to "make a thing". Some people learn programming while making a thing, but why learn about how the sausage is made when all you want is to eat it?

Re: I'm tired of fixing customers' AI generated code

#184

Post by IanCal » Thu, Aug 22, 2024, 9:29 AM UTC

sim7c00 wrote: post #175

IanCal wrote: post #160

Earlier quoted context omitted.

The description here isn't about using LLM hallucinations as code. It's using them as example API users and seeing what they get stuck on. Let's say you've got some rest API. Are they trying to send PATCH requests but you only support POST? Do they keep trying to add pagination or look for a cursor? Do they expect /things to return everything and /things/id to return a single one but you have /things and /thing/1 ? N…

Ah, sorry, totally didn't get that it was using them as users of the API. I would recommend API fuzzers here, which cycle through all available possibilities, and generate 'garbage' inputs as well as structured inputs regarding the specifications the APIs implement (http, and the underlying structures provided to the api endpoints). an LLM would likely not perform an exhaustive test. There are several projects freely…

I think you're picturing this differently again.

It's not testing the API for bugs, it's asking whether the API you have follows typical norms and isn't missing things. Is your API confusing or incomplete?

Imagine you have a library dealing with chat rooms, and LLMs keep trying to call "room.getUser(id)" and "room.getUsers()" when your API is actually "room.user(filter:{id:id})[0]" and "room.user(filter:None)". Maybe that's a sign that your API, while totally functional is going to be confusing to other programmers. Maybe you don't have anything that lists all current users, or there's a flag or option somewhere that would make sense or be more typical.

A fuzzer won't tell you that.

For the original post, you could imagine that a user might want the price of a token in a different currency, but maybe it currently returns everything in dollars. An LLM might try and add ?currency=GBP . Maybe an LLM keeps expecting to see in the response data the last updated timestamp to check data freshness, but it isn't something you add right now.

Re: I'm tired of fixing customers' AI generated code

#185

Post by yifanl » Thu, Aug 22, 2024, 9:29 AM UTC

shakna wrote: post #141

FanaHOVA wrote: post #100

Earlier quoted context omitted.

Show me 1 example.

There's more than one mistake in the code given, and I tried just once. "Create Python code for iterating a file with a cache." class CachedFileIterator: def __init__(self, file_path): self.file_path = file_path self.cache = [] # List to store cashed lines self.current_index = 0 # Index for the current line to read def _read_lines_from_file(self): """Read lines from the file and cache them.""" with open(self.file_pat…

Garbage code is bad enough, but it's not like people have never had to walk juniors through mistakes before LLMs.

But this is actually so much worse for that same reason - the type of developer who'd submit Copilot output (I can call it that, as it's definitely not code) for a PR is unable to respond to any comment beyond asking Copilot again and wasting everyone's time with 6 more rounds of reviews. I've literally had to write test cases for someone else and told them "You can't ask for another code review until your code passes these."

Re: I'm tired of fixing customers' AI generated code

#186

Post by IanCal » Thu, Aug 22, 2024, 9:30 AM UTC

stavros wrote: post #181

sim7c00 wrote: post #175

Earlier quoted context omitted.

Ah, sorry, totally didn't get that it was using them as users of the API. I would recommend API fuzzers here, which cycle through all available possibilities, and generate 'garbage' inputs as well as structured inputs regarding the specifications the APIs implement (http, and the underlying structures provided to the api endpoints). an LLM would likely not perform an exhaustive test. There are several projects freely…

> Ah, sorry, totally didn't get that it was using them as users of the API. Originally, it wasn't. From the GP post: > If there is a common hallucination API call, just add that API. This will make the API more "friendly" and resemble common similar APIs. If it's just users, you wouldn't be adding the APIs.

Users were calling their API.

Re: I'm tired of fixing customers' AI generated code

#187

Post by treprinum » Thu, Aug 22, 2024, 9:30 AM UTC

Set firm boundaries and reject/ghost customers that want to build their app for free and are "angry". Those never lead to anything profitable.

Re: I'm tired of fixing customers' AI generated code

#188

Post by nkrisc » Thu, Aug 22, 2024, 9:43 AM UTC

lagniappe wrote: post #18

There's room for all of us in this industry. What someone is unwilling to do is just an opportunity for someone else to pick up the yoke.

I have the sense that most of these people won’t be willing to pay anything to have their code fixed.

Re: I'm tired of fixing customers' AI generated code

#189

Post by sfn42 » Thu, Aug 22, 2024, 9:47 AM UTC

stavros wrote: post #183

TillE wrote: post #3

I'm always a little surprised at how many people out there want to develop software yet haven't put in the effort to gain even the most basic computer nerd programming chops. You see this all the time in the more newbie-friendly game engine communities. Maybe you don't want to pursue a career in software, but anyone can spend a week learning Python or JavaScript. I suspect/hope a lot of these people are just kids who…

> I'm always a little surprised at how many people out there want to develop software yet haven't put in the effort to gain even the most basic computer nerd programming chops. If you're surprised by reality, that says something about your mental model, not about reality. People don't want to "learn programming", they want to "make a thing". Some people learn programming while making a thing, but why learn about how…

I like learning how to do stuff. It's strange to me that people think they can do stuff without learning how

Re: I'm tired of fixing customers' AI generated code

#190

Post by swah » Thu, Aug 22, 2024, 9:49 AM UTC

woctordho wrote: post #143

delusional wrote: post #116

Earlier quoted context omitted.

We hired a new guy at work. In one of his first tasks he had chosen to write some bash, and it was pure nonsense. I mean it contained things like: if [ -z "${Var}+x" ] Where I can see what the author was trying to do, but the code is just wrong. I dont mind people not knowing stuff, especially when it's essentially Bash trivia. But what broke my heart was when I pointed out the problem, linked to the documentation, b…

Everyone working with shell scripts should know shellcheck

And Python...