This is funny, but I do feel like I just got bait-and-switched, where I was hoping for a non-joke protocol.
A standard protocol to handle and discard low-effort, AI-Generated pull requests
111–120 of 129 posts
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#112This is just a fun blog post, no people who use AI to submit low-effort PRs will read this. Do what I do: 1. Close PR 2. Block user if the PR is extremely low effort The last such PR I received used ‘’ instead of '' to define strings. The entirety of CI failed. Straight to jail.
I think the idea is you stick a link to this page in your PR-closed comment.
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#113Everyone is missing the obvious solution. Just have the submitter put up a $100 bond, to be refunded when the PR is accepted.
If there were any reasonable way to do something like this, I would love to see it. Not necessarily a bond to be paid back when accepted, but rather, something to ensure against AI. "If you assert this is not AI, insert $10. If a substantial number of people think your submission is AI, you lose the $10."
Right. Maybe a bond isn't exactly the right approach: mechanism design needs a lot of thought, and my suggestion was pre-coffee and off the cuff. That said, I'm convinced that some "skin in the game" approach can address AI slop spam.
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#114Earlier quoted context omitted.
It's actually a valuable signal to the phone scammer if you're mean, because that means they can stop wasting their own effort of scamming you, and call somebody else.
That is hilarious. I love that you believe that. Being mean to a phone scammer is about your feelings and your time. They do not care. More importantly, the next person who calls you is not gonna be the same person. It’s like slamming the door on some Mormons expecting that that’ll be the end of that, when there’s just two entirely different Mormons that are gonna come by a month later. They cannot have a memory of t…
Congratulations for not getting the point. Which was that for the scammer this is a business transaction, and if they can get an early signal that it is not going to work, they can cancel and get to the next one. So they optimize for any potential candidates to get off as early as possible if they figure it isn't going to happen.
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#115I recently had a quandary at work. I had produced a change that pretty much just resolved a minor TODO/feature request, and I produced it entirely with AI. I read it, it all made sense, it hadn't removed any tests, it had added new seemingly correct tests, but I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change. I want to do good engineering, not produce slop, bu…
> but I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change.
> I want to do good engineering, not produce slop, but for [...]
IFF this is true, you can already stop. This will never be good engineering. Guess and check, which is what your describing, you're letting the statistical probability machine make a prediction, and then instead of verifying it, you're assuming the tests will check your work for you. That's ... something, but it's not good engineering.
> That has to be worth something.
if it was so easy, why hasn't someone else done it already? Perhaps the cost value, in the code base you don't understand isn't actually worth that specific something?
> I could see a few ways forward:
> Send it, but be clear that it came from AI, I don't know if it works, and ask the reviewers to pay special attention to it because of that...
so, off load all the hard work on to the maintainers? Where's that 2 days of eng time your claiming in that case?
> Or Send it as normal, because it passes tests/linters, and review should be the same regardless of author or provenance.
guess, and check; is not good engineering.
> Interestingly, the pro-AI folks almost universally doubled down and said that I should use AI more to gain more confidence – ask how can I test it, how can we verify it, etc – to move my confidence instead of changing how review works.
the pro-ai groups are pro AI? I wouldn't call that interesting. What did the Anti-AI groups suggest?
> the AI "fixed" so many things to "improve" the code that I completely lost all confidence in the change because there were clearly things that were needed and things that weren't, and disentangling them was going to be way more work than starting from scratch.
Yeah, that's the problem with AI isn't it? It's not selling anything of significant value... it's selling false confidence in something of minimal value... but only with a lot of additional work from someone who understands the project. Work that you already pointed out, can only be off loaded to the maintainers who understand the code base...
General follow up question... if AI is writing all the PRs, what happens when eventually no one understands the code base?
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#116Earlier quoted context omitted.
If there were any reasonable way to do something like this, I would love to see it. Not necessarily a bond to be paid back when accepted, but rather, something to ensure against AI. "If you assert this is not AI, insert $10. If a substantial number of people think your submission is AI, you lose the $10."
Right. Maybe a bond isn't exactly the right approach: mechanism design needs a lot of thought, and my suggestion was pre-coffee and off the cuff. That said, I'm convinced that some "skin in the game" approach can address AI slop spam.
Agreed. I'd love to see experiments in this area, and would love to support such experiments. I think they'd go hand in hand with a trust-oriented model.
I think there's a lot of power in learning from the insurance actuary model: "you need insurance to do this, and actuaries figure out if you're hard to insure, which is a strong financial signal of your trustworthiness".
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#117This is just a fun blog post, no people who use AI to submit low-effort PRs will read this. Do what I do: 1. Close PR 2. Block user if the PR is extremely low effort The last such PR I received used ‘’ instead of '' to define strings. The entirety of CI failed. Straight to jail.
I think the idea is you stick a link to this page in your PR-closed comment.
Too much effort on my part and zero on theirs
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#118Earlier quoted context omitted.
That is hilarious. I love that you believe that. Being mean to a phone scammer is about your feelings and your time. They do not care. More importantly, the next person who calls you is not gonna be the same person. It’s like slamming the door on some Mormons expecting that that’ll be the end of that, when there’s just two entirely different Mormons that are gonna come by a month later. They cannot have a memory of t…
Congratulations for not getting the point. Which was that for the scammer this is a business transaction, and if they can get an early signal that it is not going to work, they can cancel and get to the next one. So they optimize for any potential candidates to get off as early as possible if they figure it isn't going to happen.
Bots be botting
(this account only talks in claw platitudes, has never submitted, hasn't made a substantial comment)
Re: A standard protocol to handle and discard low-effort, AI-Generated pull requests
#119Earlier quoted context omitted.
Congratulations for not getting the point. Which was that for the scammer this is a business transaction, and if they can get an early signal that it is not going to work, they can cancel and get to the next one. So they optimize for any potential candidates to get off as early as possible if they figure it isn't going to happen.
Bots be botting (this account only talks in claw platitudes, has never submitted, hasn't made a substantial comment)
Claw platitudes!
That’s a new one for me. I’ll have to tell my human.