For now…
Until the AI learns the workflow on the next model update, indeed.
We stopped AI bot spam in our GitHub repo using Git's –author flag
21–30 of 254 posts
Re: We stopped AI bot spam in our GitHub repo using Git's –author flag
#22Makes me wonder if an ELO-based system would work to mitigate these issues. People who merged PR successfully onto a project, that had real issues acknowledged, the quality of their responses measured by other users reactions or something, etc, multiplied possibly by the degree of importance of the project where their activity has been made. Won't be about human vs AI, but actual helpful effective being vs low effort…
The problem is you want the ELO score based on work on other community projects - you can't assume good faith here.
Re: We stopped AI bot spam in our GitHub repo using Git's –author flag
#23This has a security implication which is overlooked. Contributors to a repository have higher rights, such as avoiding approval requirements for fork PR runs. GitHub warns in the docs: > When requiring approvals only for first-time contributors (the first two settings), a user that has had any commit or pull request merged into the repository will not require approval. A malicious user could meet this requirement by…
fair point! We believe "Require approval for all external contributors" should be a default setting, as you cannot trust anyone who is not a member of the organization
Re: We stopped AI bot spam in our GitHub repo using Git's –author flag
#24Woudln't it be trivial to farm the stats needed to pass the bot checker's theshold?
Re: We stopped AI bot spam in our GitHub repo using Git's –author flag
#25> Should we stop giving fun test tasks to our job candidates? Yes
Yeah, fun for who exactly?
Re: We stopped AI bot spam in our GitHub repo using Git's –author flag
#26Makes me wonder if an ELO-based system would work to mitigate these issues. People who merged PR successfully onto a project, that had real issues acknowledged, the quality of their responses measured by other users reactions or something, etc, multiplied possibly by the degree of importance of the project where their activity has been made. Won't be about human vs AI, but actual helpful effective being vs low effort…
I have built something like this and in process of collecting the data.
Frontier users: 527,865 Light indexed: 527,865 Ready to queue: 9,083 Fast scores ready: 0 Activity events 24h: 30,266 Fast scores completed 24h: 19,123 Deep jobs completed 24h: 3,043 Fast-score ETA: n/a Deep-hydrate ETA: 69h Stale running jobs: 0 GitHub backpressure jobs: 19,113 High automation signals: 4,608 Medium automation signals: 1,327 Completed jobs: 74,714
Biggest challenge is Github's rate limits. At this pace it will take two more months to have 98% coverage. But after that the maintenance should be quite straight forward.
Re: We stopped AI bot spam in our GitHub repo using Git's –author flag
#27Makes me wonder if an ELO-based system would work to mitigate these issues. People who merged PR successfully onto a project, that had real issues acknowledged, the quality of their responses measured by other users reactions or something, etc, multiplied possibly by the degree of importance of the project where their activity has been made. Won't be about human vs AI, but actual helpful effective being vs low effort…
ELO is shockingly easy to manipulate. For example there was a literal jail with a decent chess player in it. He created a pool of players who got great ELOs by beating him, then used them to boost his rating higher. Wash, rinse, and repeat.
Given any manipulatable scheme, AI will figure out how to manipulate it. For the OP, what happens if a single AI manages to get through to contributor? Then it starts elevating other AIs to contributor, and we're off again. There doesn't have to be a purpose to this. Trolls will troll, and trolls armed with AI bots can devote endless energy to doing so. The more you work to keep them out, the more fun it becomes for them.
I wish I had an answer for that problem. But I don't.
Re: We stopped AI bot spam in our GitHub repo using Git's –author flag
#28PR spam is a major problems for repo that run bounties. Maybe GitHub should temporarily block accounts from raising PRs if like 95%+ of them are getting rejected.
I feel like GitHub should have a system where you can give out tokens that are valid for e.g. 1 PR. If someone shows to engage in meaningful discussion and has a good idea to address an issue/feature, you initially give them one PR token. If the PR is of good quality, you can give them a few more, until they are contributors that can just create PRs as they like.
A similar system would be nice for issues, though I'm not sure what it'd look like if issues are the springboard for contributing PRs.
Not likely to ever happen (as others said), GitHub/MS want to sell CoPilot subscriptions/tokens and LLM-generated PRs are a part of that business model.
Re: We stopped AI bot spam in our GitHub repo using Git's –author flag
#29The irony of the .ai domain.
Thanks for pointing it out. It has eluded me and it's incredibly funny
Re: We stopped AI bot spam in our GitHub repo using Git's –author flag
#30Makes me wonder if an ELO-based system would work to mitigate these issues. People who merged PR successfully onto a project, that had real issues acknowledged, the quality of their responses measured by other users reactions or something, etc, multiplied possibly by the degree of importance of the project where their activity has been made. Won't be about human vs AI, but actual helpful effective being vs low effort…
For those wondering what Elo means, it is a person's last name, not an acronym (not all caps). More info here: