[flagged]
[deleted]
A standard protocol to handle and discard low-effort, AI-Generated pull requests
91–100 of 129 posts
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#92Everyone is missing the obvious solution. Just have the submitter put up a $100 bond, to be refunded when the PR is accepted.
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#93I prefer this policy: https://github.com/ghostty-org/ghostty/blob/main/AI_POLICY.m... > If you can't explain what your changes do and how they interact with the greater system without the aid of AI tools, do not contribute to this project. edit: added that quote
Good idea, though I'm not sure how to enforce it. You can ask an AI for that and then rewrite it in your own words.
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#94The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted exactly as how much we do not want to review your generated submission. I know it is in jest, but I really hate that so many documents include “shall”. The interpretation of which has had official legal rulings going both ways. You MUST use less ambiguo…
On (possibly weak) counterpoint that I can offer is that in some languages, “must not” is a false friend, easily misinterpreted as “is not required to” (“it is not the case that they must”).
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#95[dead]
[dead]
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#96Amazing. I hope this gets tons of use shaming zero-effort drive by time wasters. The FAQ is blissfully blunt and appropriately impolite, I love it.
While I am with you on hoping, someone shamelessly PRing slop just is not going to feel shame when one of their efforts fail. It’s like being mean to a phone scammer, they just hang up and do it again
Cheap, nearly free voice phone calls killed old fashioned phone service. Once the incoming spam exceeded 95% I shut off the ringer and no longer use voice phone calls.
Once the cost of generating push media drops low enough (close enough to zero) the media is dead.
Pull requests are (ironically) a push media, and infinite zero effort PRs can be generated, therefore PRs are dead.
The proper way to handle the situation is to no longer accept PRs.
In github, enter a repo, "settings" "General" scroll down to Features, then uncheck "Pull requests". Or at least set to collaborators only. Probably need to shut off issues.
It gitlab, (I'm not as certain about this) enter a repo, "Settings", Visibility, "Merge Requests" change to "Only project members"
Its a post AI world, those features cannot be enabled on the internet anymore. Anything that accepts push from the public will get spammed into inability to use it. As a social activity PRs are dead. They were nice, but they are dangerous to leave enabled on the internet now. Oh well thats the cost of AI.
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#97I'd love to hear some commentary about my idea surrounding this problem of AI PRs.
Why not restrict the agents to writing tests only?
If the tickets are written concisely, any feature request or fix could be reduced to necessary spec files.
This way, any maintainer would be tasked with reviewing the spec files and writing the implementation.
CI is pretty good at gatekeeping based on test suites passing...
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#98I'd love to hear some commentary about my idea surrounding this problem of AI PRs. Why not restrict the agents to writing tests only? If the tickets are written concisely, any feature request or fix could be reduced to necessary spec files. This way, any maintainer would be tasked with reviewing the spec files and writing the implementation. CI is pretty good at gatekeeping based on test suites passing...
Quis custodiet ipsos custodes?
If the problem is that we don't trust people who use AI without understanding its output, and we base the gate-keeping on tests that are written on AI, then how can we trust that output?
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#99I think part of the deeper issue is that contributing to an OSS project has become a rite of passage, a way to strengthen your profile. If you need to have contributed to look good but you don't really care about the contribution itself then you resort to this kind of trick.
We had a similar plague for vulnerability disclosures, with people reporting that they had "discovered" vulnerabilities like "if you call this function with null you get a NullPointerException". D'uh.
There is also the fact that we're measuring the wrong thing like speed of development. In my previous employer people had jumped in fully into the AI bandwagon, everyone was marvelled at how fast they were. Once I was reviewing the PR and I had to tell the author "dude, all your tests are failing". He just laughed it out. Everyone can produce software very fast if it's not required to work.
AI-assisted gamification.
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#100I'd love to hear some commentary about my idea surrounding this problem of AI PRs. Why not restrict the agents to writing tests only? If the tickets are written concisely, any feature request or fix could be reduced to necessary spec files. This way, any maintainer would be tasked with reviewing the spec files and writing the implementation. CI is pretty good at gatekeeping based on test suites passing...
Quis custodiet ipsos custodes? If the problem is that we don't trust people who use AI without understanding its output, and we base the gate-keeping on tests that are written on AI, then how can we trust that output?
Isn't that the purpose of red/green refactoring though? To establish working software that expresses regression, and builds trust (in the software)?
If your premise is that people would shift to using AI to write tests they don't understand, then that's not necessarily a failing of the contributor.
The contributor might not understand the output, but the maintainer would be able to critique a spec file and determine pretty quickly if implementation would be worthwhile.
This would necessitate a need for small tickets, thereby creating small spec files, and easier review by maintainers.
Also, any PR that included a non spec file could be dismissed patently.
It is possible for users of AI to learn from reading specs.
But if agents are doing the entire thing (reading the ticket, generating the PR, submitting the PR)...then the point of people not understanding is moot.