Live data from Hacker News

Study finds AI assistants help developers produce code likely to be buggy

theregister.com

61–70 of 153 posts

Re: Study finds AI assistants help developers produce code likely to be buggy

#61
P¹ All people write buggy code. [1]

P² AI assistants are trained on code written by people

∴ People using AI assistants will write buggy code

[1] Some might object to this, either in their own case or that of some of their professional associates. I would not trust the judgement of these people.

Re: Study finds AI assistants help developers produce code likely to be buggy

#62
post #32

Isn't this like trouble shooting a pong console to assess the future of video gaming?

No, they aren't judging the future, they are judging the present. Many developers are using language models to assist them today so it is worth checking if that actually helps them or not. When we are in the future we can start judging future language models, but until then we can only test what we have now.

It's all short sighted given what we already know about the future. We know that these models will soon and continuously be more complex. We know they will only trend toward greater accuracy.

Computer engineers won't be impressed with the output for some time. Front end devs like me just got an amazing search tool. I work faster now. I told my boss I work faster now. No going back.

Understand, it's now faster for me to ask ChatGPT to find an answer and then Google to confirm it than it is for me to only google things. For me the AI revolution already came and will only be getting better.

With design and process know-how I'm not really worried about losing my job. AIs will eventually be able to make ad hoc decisions about people's brand identity. But not anytime soon.

Re: Study finds AI assistants help developers produce code likely to be buggy

#64
post #21

Surprising. If only there were a way that we could have foreseen that an AI trained to write code in part by looking at people who, self-admittedly, don’t know how to write code, and people who write code for others with minimal context (Stack Overflow), would produce buggy code. It is a case of GIGO. Most developers do not learn much from Stack Overflow. Why do we expect AI to fare better? In my experience, one in t…

If I had a little robot riding in the passenger seat that could tell me whether to go left, straight, or right, and it was correct 90% of the time, I'd think that was pretty great. I'd get where I needed to be, even with a couple mishaps. ML code suggestions are the same thing to me. If I don't know where I am going, I can just ask it for suggestions. And it's probably going to be what I want. In both cases, I am ann…

One in ten turns being told to do the wrong thing sounds incredibly annoying. Maybe if you’re driving on the highway for long stretches, but otherwise that would be awful.

Re: Study finds AI assistants help developers produce code likely to be buggy

#66
I have been using GitHub copilot for several months now, nearly half the code in a side gig is written with copilot. then came chatgpt, I like it better on what I communicate with it and the code it generates and explains it and I can ask it to modify the generated code. I wish copilot gets chatgpt as the core soon as I prefer that style and the quality of code I get.

yes there are occasional bugs in code but you can find and debug it when you run the code.

Re: Study finds AI assistants help developers produce code likely to be buggy

#67
post #9

I've been producing buggy code for over 20 years. Don't need no AI to help me do that.

But what if you could write more LoC/h and make Elon Musk happier?

You’ll need to pair it up with a super fast laser printer to be able to get it all properly code reviewed.

Re: Study finds AI assistants help developers produce code likely to be buggy

#68
post #21

Surprising. If only there were a way that we could have foreseen that an AI trained to write code in part by looking at people who, self-admittedly, don’t know how to write code, and people who write code for others with minimal context (Stack Overflow), would produce buggy code. It is a case of GIGO. Most developers do not learn much from Stack Overflow. Why do we expect AI to fare better? In my experience, one in t…

If I had a little robot riding in the passenger seat that could tell me whether to go left, straight, or right, and it was correct 90% of the time, I'd think that was pretty great. I'd get where I needed to be, even with a couple mishaps. ML code suggestions are the same thing to me. If I don't know where I am going, I can just ask it for suggestions. And it's probably going to be what I want. In both cases, I am ann…

90% right is superficially impressive but in actual practice is abysmal.

Voice recognition software needed to get to 99.9% accuracy to be actually useable.

Re: Study finds AI assistants help developers produce code likely to be buggy

#69
post #4

I made a site using the OpenAI APIs new models (similar to ChatGPT) for creating simple apps or demos using natural language: https://aidev.codes . If anyone is interested please try it out. I tried to promote the site by submitting to HN and did not receive a single vote. I feel that although many people may not be impressed, it is better than 0. So I am just trying to promote it in related threads. Anyway if you ha…

Hi ilaksh, I was able to view your video on an android device. That said, I did have to watch it several times before I understand what was going on because a lot is happening everywhere in the screen. With the video and the minimalist intro on the site, I was still left guessing what was being offered and how it works.

Usually before signing up for any website, I want to check out all of the information I can find on it on the website, perhaps online forums if the site isnt clear, and competitors so make sure what I'm signing up for is something I actually want. I also read any terms and conditions and the privacy policy if they ask for anything from me, such as an email address. Pretty much none of this was available.

The first thing I noticed/recommend is changing the lack of tabs. It reminded me of scam sites that barely have enough to make a site appear legit so they can get at least someone to sign up for something. Also just for the persuation aspect of the site, there should be helpful tabs, like one with an example of a person using it (I know thats what the video is, but it's not intuitive for a non-user, which I'll get to in a second). One of the other guys gave the idea of having images with commentary of whats going on between each pic, which I think is a great idea if implimented properly. Maybe that page could have a longer video under the photo tutorial with audio of someone explaining a basic how to, in the form of them showing what they want to do and how they did it. Then building a simple button with a counter would be enough. Maybe a tab could be a contact page with the T&C, PP, would be nice. And another explaining how it works - like your little comment mentioning the APIs is more information than what is on the website and makes me more willing to try it out than your site did.

So onto the site itself, after log in. Because of the previous bullets, I would not have been willing to sign up, but to give hopefully a helpful response to you, and knowing I'd need to see more of the site, was the only reason I made an account. Btw the UI of your little program is very nice. It's stylish, mostly quick to grapple, and simplistic.

Anyway when I first saw it, again due to the front page lacking any info, I was surprised that it was exactly the video. The video wasn't clear whether we were looking at a feature, the whole program, or what really. It was just visual noise without explanation on the main page. Giving it a title like "this is exactly what you'll see" will not help it though, again because it isn't clear what is going on and the video by itself is more unclear in use than the actual program itself. In fact, if you had a tab where users can actually use the program without making an account, but it just doesn't save their work and is a demo, that would already be a huge help because people would instantly see how easy and fun it is to use and it would rid like 90% of the questions caused by not having any information.

Anyway, I saw the program and thought, oh its just the front page video thing. I can use this! And I proceeded to try to use the top right portion and wondered why it didn't work. Then I kinda clicked randomly until I realized the in-prompt area is actually the top-left part. Not exactly a fatal flaw, but a clear tutorial would have led me to understanding the program faster. Like the fact that every input is its own section of code, and so my input needs to follow an orderly pseudocode format. As a person not familiar with how good AI devs are, I am not already in the know if I can have inputs be rearranged, have a correcting input (say, if I didn't like the previous input) be a new input after the prev one or if I have to edit the previous input directly. The rules of using the AI are just unclear. Speaking of edits, and I recognize that since I was using a phone to use the program, so some features are more difficult to use, I couldn't figure out how the editing works. I simultaneously was getting a glitch where the green loading bar would almost reach the end then restart, and then was going back and forth like a spring after I tried editing, and wouldn't submit anything. It was unclear if I was trying to edit wrong or there was a different bug I was running into. The already lack of clarity had ate up my patience so I didn't bother to figure out what I was doing wrong.

TL;DR:

In summary, you need 3 tabs on the front page explaining what it does, how it works and how a user uses it, and maybe another tab for T&C, PP, and contact. Since the landing page is already a attempt at that, it could be changed to why a person wants to use it. I can think of 3 reasons off the top of my head for a potential target audience and I don't even know the full uses of the program so I'm sure you can think of more.

After log in, there needs to be an additional tab with detailed tutorials, like how to think when giving info to an AI dev, how to make corrections, etc. Since you'll have a contact page, people will find more things that aren't unclear that you can use to either improve the tutorial or have a common Q&A section (which depending on the content would go on the page before logging in or after, or accessible to both). Lastly you'll want a bug submission area as well, since in the few minutes of use, I ran into bugs, and you'll probably want to separate general contact from bug submissions.

Post reply on HN