Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: AI-generated spam pull requests?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 27 posts

Re: Ask HN: AI-generated spam pull requests?

#11

Post by sudo_navendu »

fishtoaster wrote: post #7

Relatedly, the Rails codebase recently received a (clearly marked) AI-generated pull request: https://github.com/rails/rails/pull/47708 At a glance, it looks like it's been mostly well-received and has not yet been immediately closed as spam.

This is how you use AI tools to write code.

In our project, the user just copy-pasted the output from the AI tool and called it a day. They did not even bother to build the project and test it.

I have also started using AI tools and it has made me much more efficient. I could have done a task without AI but with it, it is much more faster.

Re: Ask HN: AI-generated spam pull requests?

#12

Post by sudo_navendu »

enumjorge wrote: post #4

> We do not know what the end goal of the person was but we confronted the person and closed the pull requests. Maybe this was a naïve attempt at inflating their GitHub numbers? Some people use those as a credibility measure when applying to jobs or getting clients.

Now they clearly won't because we will be reporting them to GitHub.

Re: Ask HN: AI-generated spam pull requests?

#13

Post by sudo_navendu »

georgel wrote: post #2

Don't other projects have automated tests that check that the PR will actually compile/work? And if the tests fail, the PR is rejected.

Actually, we were sure that it was spam. GitHub gives the option to "approve workflow run for first-time contributors". I guess none of the maintainers thought to approve it because they thought it might be spam. Still, a lot of time and effort spent to review it.

Re: Ask HN: AI-generated spam pull requests?

#15

Post by sudo_navendu »

I wrote more about what actually happened here: https://navendu.me/posts/ai-generated-spam-prs/

It can help set some context to the discussions.

TLDR:

Recently, a person has been using AI tools to generate code and open pull requests to open source projects I contribute to.

The code is entirely wrong and doesn’t work, and it is evident that the person making these pull requests doesn’t understand the code.

The person also copied explanations (which was an obvious giveaway as it sounded like a typical response) into the pull request and attempted to explain the code and answer questions from the reviewers.

We were polite and when it didn’t work, reported the person to GitHub.

I don’t want to shame the person publicly. But I want to make other open source maintainers aware that this is a thing and prevent them from wasting time and effort chasing such people down.

Re: Ask HN: AI-generated spam pull requests?

#16

Post by sudo_navendu »

crop_rotation wrote: post #14

It is the new reality though. Soon AI generated PRs will look human enough to require a good amount of time of the repo maintainers to review.

It is perfectly fine if the code actually works. At least for now, getting the exact code you want from an AI is a legit skill. This was pure spam.

I wrote more about what actually happened here: https://navendu.me/posts/ai-generated-spam-prs/

It can help set some context to the discussions.

Re: Ask HN: AI-generated spam pull requests?

#18

Post by crop_rotation »

debesyla wrote: post #17
crop_rotation wrote: post #14

It is the new reality though. Soon AI generated PRs will look human enough to require a good amount of time of the repo maintainers to review.

I doubt that AI generated PRs will ever be as bad as (some) human generated ones.

Correct. But they will be a. much higher in volume and b. would not look bad at a glance.

Re: Ask HN: AI-generated spam pull requests?

#19

Post by badloginagain »

sudo_navendu wrote: post #11
fishtoaster wrote: post #7

Relatedly, the Rails codebase recently received a (clearly marked) AI-generated pull request: https://github.com/rails/rails/pull/47708 At a glance, it looks like it's been mostly well-received and has not yet been immediately closed as spam.

This is how you use AI tools to write code. In our project, the user just copy-pasted the output from the AI tool and called it a day. They did not even bother to build the project and test it. I have also started using AI tools and it has made me much more efficient. I could have done a task without AI but with it, it is much more faster.

That makes no sense. The one thing chatgpt does _really_ well is setup unit tests, which is the part of unit tests I hate.

My ChatGPT workflow is give requirements -> have it create unit tests -> give it test results until it passes.

Been playing around with a generated-code-only project: https://github.com/JerkyTreats/scrivr/

In that workflow I don't really look closely at the code. In most cases I've found it isn't really necessary.

Re: Ask HN: AI-generated spam pull requests?

#20

Post by ynik »

sudo_navendu wrote: post #13
georgel wrote: post #2

Don't other projects have automated tests that check that the PR will actually compile/work? And if the tests fail, the PR is rejected.

Actually, we were sure that it was spam. GitHub gives the option to "approve workflow run for first-time contributors". I guess none of the maintainers thought to approve it because they thought it might be spam. Still, a lot of time and effort spent to review it.

That button was added to GitHub to protect against new bot accounts creating PRs against random projects, adding a CI step that runs a cryptominer. Now that the CI doesn't run automatically for new users without a button click, these attackers have a much harder time.

So tell your maintainers to use that button more liberally -- it mostly just exists to save GitHub money / discourage these attacks. It doesn't hurt to click it for these "CV improvement" spam PRs, and it makes rejecting the PR a lot simpler if there's a red X.

I usually just scan file list changed by the PR, and if it isn't changing CI stuff, I just let the actions run prior to the actual code review.

Post reply on HN