Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a clearer framework for AI-assisted contributions to open source

samsaffron.com

Post reply on HN
Print view View original

131–140 of 161 posts

Re: We need a clearer framework for AI-assisted contributions to open source

#131

Post by saghm »

sothatsit wrote: post #128
pdonis wrote: post #126

Earlier quoted context omitted.

> Because that's the definition of collaboration? I don't think the definition of collaboration includes making close to zero effort and expecting someone else to expend considerable effort in return.

The problem is that the sheer volume of low-quality AI PRs is overwhelming. Just the time it takes determining whether you should pay attention to a PR or not can add up when there are a lot of plausible-looking, but actually low-quality and untested, pull requests to your project. But if you stop looking at PRs entirely, you eliminate the ability for new contributors to join a project or make changes that improve th…

Since the bar to opening a PR has gotten lower, there's an argument that the bar for closing it might need to be lowered as well. I think right now, we consider the review effort to be asymmetric in part because it's natural to want to give the benefit of the doubt to PR authors rather than making a snap judgement from only a looking briefly at something; the current system seems to place a higher value on not accidentally closing a potentially useful but poorly presented PR than not accidentally wasting time on one that superficially appears like it might be good but isn't. I have to wonder if the best we can do is to just be more willing to close PRs when reviewers aren't sufficiently convinced of the quality after a shorter inspection regardless of whether we're 100% certain about whether that judgment is perfect. If "false positive" PRs that seem reasonable but turn out not to be are better at appearing superficially good, the best option seems like it might just be to be willing to throw out more "false negatives" that would be useful but aren't sufficiently able to distinguish themselves from the ones that aren't.

After a minute (or whatever length of time makes sense for the project), decide whether you're not fully confident that the PR is worth your time to continue reviewing, with the default answer being "no" if you're on the fence. Unless it's a yes, you got a bad vibe; close it and move on. Getting a PR merged will require more effort in making the case that there's value in keeping it open, which restores some of the balance that's been lost in the effort having been pushed to the review side.

Re: We need a clearer framework for AI-assisted contributions to open source

#132

Post by sails »

solotronics wrote: post #120
andy99 wrote: post #69

This is a problem everywhere now, and not just in code. It now takes zero effort to produce something, whether code or a work plan or “deep research” and then lob it over the fence, expecting people to review and act upon it. It’s an extension of the asymmetric bullshit principle IMO, and I think now all workplaces / projects need norms about this.

This problem statement was actually where the idea for Proof of Work (aka mining) in bitcoin came from. It evolved out of the idea of requiring a computational proof of work for sending an email via cypherpunk remailers as a way of fighting spam. The idea being only a legitimate or determined sender would put in the "proof of work" to use the remailer. I wonder how it would look if open source projects required $5 to…

$5 could go towards a strict AI reject/review funnel as a prefilter

Re: We need a clearer framework for AI-assisted contributions to open source

#134

Post by aleph_minus_one »

andy99 wrote: post #69

This is a problem everywhere now, and not just in code. It now takes zero effort to produce something, whether code or a work plan or “deep research” and then lob it over the fence, expecting people to review and act upon it. It’s an extension of the asymmetric bullshit principle IMO, and I think now all workplaces / projects need norms about this.

> This is a problem everywhere now, and not just in code. It now takes zero effort to produce something, whether code or a work plan or “deep research” and then lob it over the fence, expecting people to review and act upon it.

Where is the problem? If I don't have the time to review a PR, I simply reject it. Or if I am flooded in PRs, I only take those from people from which I know that their PRs are of high quality. In other words: your assumption "expecting people to review and act upon it" is wrong.

Even though I would bet that for the kind of code that I voluntarily write in my free time, using an LLM to generate lots of code is much less helpful because I use such private projects to try out novel things that are typically not "digested stuff from the internet".

So, the central problem that I rather see is the license uncertainties for AI-generated code.

Re: We need a clearer framework for AI-assisted contributions to open source

#136

Post by nkrisc »

sothatsit wrote: post #128
pdonis wrote: post #126

Earlier quoted context omitted.

> Because that's the definition of collaboration? I don't think the definition of collaboration includes making close to zero effort and expecting someone else to expend considerable effort in return.

The problem is that the sheer volume of low-quality AI PRs is overwhelming. Just the time it takes determining whether you should pay attention to a PR or not can add up when there are a lot of plausible-looking, but actually low-quality and untested, pull requests to your project. But if you stop looking at PRs entirely, you eliminate the ability for new contributors to join a project or make changes that improve th…

PR authors blow now need to spend energy and effort to make their PR appear worthwhile for consideration. AI PRs will have the effect of shifting the burden of effort to the PR authors (the real ones).

No more drive-by PRs.

Re: We need a clearer framework for AI-assisted contributions to open source

#137

Post by nkrisc »

gwbas1c wrote: post #103
softwaredoug wrote: post #11

Anyone else feel like we're cresting the LLM coding hype curve? Like a recognition that there's value there, but we're passing the frothing-at-the-mouth stage of replacing all software engineers?

I feel like we need a different programming paradigm that's more suited to LLM's strengths; that enables a new kind of application. IE, think of an application that's more analog with higher tolerances of different kinds of user inputs. A different way to say it. Imagine if programming a computer was more like training a child or a teenager to perform a task that requires a lot of human interaction; and that interact…

But how is that better?

As a parent, this sounds miserable.

Re: We need a clearer framework for AI-assisted contributions to open source

#138

Post by darkwater »

reedlaw wrote: post #79
darkwater wrote: post #16

The title doesn't make justice to the content. I really liked the paragraph about LLMs being "alien intelligence" > Many engineers I know fall into 2 camps, either the camp that find the new class of LLMs intelligent, groundbreaking and shockingly good. In the other camp are engineers that think of all LLM generated content as “the emperor’s new clothes”, the code they generate is “naked”, fundamentally flawed and po…

Isn't the intelligence of every other person alien to ourselves? The article ends with a need to "protect our own engineering brands" but how is that communicated? I found this [ https://meta.discourse.org/t/contributing-to-discourse-devel... ] which seems woefully inadequate. In practice, conventions are communicated through existing code. Are human contributors capable of grasping an "engineering brand" by working…

> Isn't the intelligence of every other person alien to ourselves?

If we agree that we are all humans and assume that all the other humans are conscious as one is, I think we can extrapolate that there is generic "human intelligence" concept. Even if it's pretty hard do nail it down, and even if there are several definitions of human intelligence out there.

For the other part of the comment, not too familiar with Discourse opensource approach but I guess that those rules are there mainly for employees, but since they develop in the open and public, they make them public as well.

Re: We need a clearer framework for AI-assisted contributions to open source

#139

Post by SideburnsOfDoom »

lccerina wrote: post #135

I have a framework: don't use it, if you never used it don't start using it, public shame people, stop talking about it. Slow down. Think long and deep about your problems. Write less code. There is NOTHING inevitable about this stuff.

Indeed. "No." is perfectly clear.

Re: We need a clearer framework for AI-assisted contributions to open source

#140

Post by troupo »

solotronics wrote: post #120
andy99 wrote: post #69

This is a problem everywhere now, and not just in code. It now takes zero effort to produce something, whether code or a work plan or “deep research” and then lob it over the fence, expecting people to review and act upon it. It’s an extension of the asymmetric bullshit principle IMO, and I think now all workplaces / projects need norms about this.

This problem statement was actually where the idea for Proof of Work (aka mining) in bitcoin came from. It evolved out of the idea of requiring a computational proof of work for sending an email via cypherpunk remailers as a way of fighting spam. The idea being only a legitimate or determined sender would put in the "proof of work" to use the remailer. I wonder how it would look if open source projects required $5 to…

> I wonder how it would look if open source projects required $5 to submit a PR or ticket and then paid out a bounty to the successful or at least reasonable PRs. Essentially a "paid proof of legitimacy".

Badly. You will alienate most legitimate contributors, and only leave spam bots subsidized by revenue from other sources

Post reply on HN