Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

github.com

Post reply on HN
Print view View original

71–80 of 117 posts

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#71

Post by cdcarter »

gcanyon wrote: post #44

Your demo video https://www.youtube.com/watch?v=WBVna_ow8vo is ridiculously compelling. You need to make a better version of the video, and maybe a few more of them.

I do think it's a pretty cool demo, but I have to say I didn't love that the PR claims Sweep did "manual testing" of the fix. Additionally, sweep reviews the PR and claims that the function is correctly implemented. A sibling points out that there's actually no testing added or done, and there's also issues with the implementation itself. This appears to be a general issue with GPT4 based products, they are extremely self-confident in their language. Presumably this stems from the overall training to work well as a chatbot.

It's very cool that it inferred the right place to make the change and the steps of finding relevant code, making a plan, then doing it are things I wish all my junior developers would do! This is certainly moving in the right direction.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#72

Post by kevinlu1248 »

cdcarter wrote: post #71
gcanyon wrote: post #44

Your demo video https://www.youtube.com/watch?v=WBVna_ow8vo is ridiculously compelling. You need to make a better version of the video, and maybe a few more of them.

I do think it's a pretty cool demo, but I have to say I didn't love that the PR claims Sweep did "manual testing" of the fix. Additionally, sweep reviews the PR and claims that the function is correctly implemented. A sibling points out that there's actually no testing added or done, and there's also issues with the implementation itself. This appears to be a general issue with GPT4 based products, they are extremely…

Yup, it's a bit frustrating since it's a problem with LLM, RLHF and fine-tuning for chat. In fact, we also added in the prompt to not say that it did testing. I find that in general it seems really difficult to tell a language model (especially 3.5) to not do something.

The self-review generally catches stuff like this since we tell it that this code is written by an inexperienced developer, so that Sweep becomes more critical.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#73

Post by eldavido »

csmpltn wrote: post #54
gcanyon wrote: post #44

Your demo video https://www.youtube.com/watch?v=WBVna_ow8vo is ridiculously compelling. You need to make a better version of the video, and maybe a few more of them.

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR. All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with. Good luck!

I think we're going to see a lot of this. I worked in self-driving and the stuff always 95% worked. Never 100.

This is useful in some ways. Thinking about situations like pre-release software testing, there are exploratory test cases that are simply too numerous to ever have a human perform economically. A lot of AI is going to do this kind of very low-valued grunt work where it doesn't matter if it's 90% or 99% correct, it's the fact that it can get done at all. A lot of this work is "additive" in the sense that, it's just too expensive to do today (with a human).

The work product of these systems is best seen as a "rough draft" or "suggestion". It's a first cut, not the last word.

On the other hand, a lot (most?) of the meat-and-potatoes coding done today, is situations where things have to WORK. Stuff where correctness absolutely matters--billing/money/settlement (calculating tax, handling returns, moving money between accounts), a lot of OS code for things like memory management / locking / resource management, drug dosing, reservation management, etc.

Granted, this stuff is a lot more complex and nuanced than the code of an average CRUD app, but then, I also don't spend my days implementing bcrypt, quicksort, or self-synchronizing Unicode parsing. We have libraries for that. The question is whether we're better off relying on agents to write a bunch of grunty code, or come up with better top-level organization / code structures, that doing it "by hand" is the better approach.

I'm actually optimistic that we can do better code-wise. But I'd love to see how things develop. Maybe we wouldn't need AI if we just had better programming languages.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#74

Post by santiagobasulto »

This is a very cool project, and definitively something we'll see more and more often for the next few years.

Now, a bit off topic. I've seen a proliferation of "open source" startups that just use "open source" as a way of promotion. The repo feels more like a landing page than a README. And the Installation shows anything BUT install instructions.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#75

Post by kevinlu1248 »

eldavido wrote: post #73
csmpltn wrote: post #54

Earlier quoted context omitted.

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR. All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with. Good luck!

I think we're going to see a lot of this. I worked in self-driving and the stuff always 95% worked. Never 100. This is useful in some ways. Thinking about situations like pre-release software testing, there are exploratory test cases that are simply too numerous to ever have a human perform economically. A lot of AI is going to do this kind of very low-valued grunt work where it doesn't matter if it's 90% or 99% corr…

I think for teams that want to move fast in a non-critical environment (health, finance etc.) something that works 90% of the time is fine. Getting to 95% takes twice the amount of time but does not provide twice the value. When the 5% difference becomes the difference maker we can fix it later.

Further, we're adding better test systems to Sweep. For now, you can just comment to get Sweep to cover the edge cases and write tests. Happy to take any other feedback.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#76

Post by kevinlu1248 »

santiagobasulto wrote: post #74

This is a very cool project, and definitively something we'll see more and more often for the next few years. Now, a bit off topic. I've seen a proliferation of "open source" startups that just use "open source" as a way of promotion. The repo feels more like a landing page than a README. And the Installation shows anything BUT install instructions.

Are you referring to the installation.md? It's a bit of a misnomer, we intended for it to be for post-installation instructions for redirect after you install.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#77

Post by twelve40 »

csmpltn wrote: post #54
gcanyon wrote: post #44

Your demo video https://www.youtube.com/watch?v=WBVna_ow8vo is ridiculously compelling. You need to make a better version of the video, and maybe a few more of them.

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR. All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with. Good luck!

Ok i get the skepticism but what i liked about their description is that it's not the overblown hysterical "AI superhuman programmer" pitch, but a more modest "junior" angle. If they keep looking for something that clicks, there are lots of "junior" niches that could be filled - for example, I can see that thing automatically working to beef up the test coverage. It's kind of difficult to screw up tests, the potential fallout is low and there is a unambiguous number (coverage) as the success criteria. If we look around our daily developer lives, there might be more cases that could be automated with this, even if it doesn't ever become good enough for any general programming.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#78

Post by gcanyon »

csmpltn wrote: post #54
gcanyon wrote: post #44

Your demo video https://www.youtube.com/watch?v=WBVna_ow8vo is ridiculously compelling. You need to make a better version of the video, and maybe a few more of them.

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR. All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with. Good luck!

Hence why:

1. They refer to as an "AI junior developer" 2. It creates pull requests, not commits

From (2), your problem is with the person who commits this code without modification, not with the AI.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#79

Post by williamzeng0 »

twelve40 wrote: post #77
csmpltn wrote: post #54

Earlier quoted context omitted.

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR. All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with. Good luck!

Ok i get the skepticism but what i liked about their description is that it's not the overblown hysterical "AI superhuman programmer" pitch, but a more modest "junior" angle. If they keep looking for something that clicks, there are lots of "junior" niches that could be filled - for example, I can see that thing automatically working to beef up the test coverage. It's kind of difficult to screw up tests, the potentia…

Something I'm a big fan of is making a small successful ticket (for example migrate the functions in one file) and then applying it map-reduce style across the entire codebase. This could help a lot, and by definition addresses repetitive work.

We have this (to some degree) with issue templates, where you can pre-populate some text and fill in the rest. We're also thinking about good ways to offload that work to Sweep.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#80

Post by williamzeng0 »

gcanyon wrote: post #78
csmpltn wrote: post #54

Earlier quoted context omitted.

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR. All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with. Good luck!

Hence why: 1. They refer to as an "AI junior developer" 2. It creates pull requests, not commits From (2), your problem is with the person who commits this code without modification, not with the AI.

That would be me, completely happy to take the blame here :) We manually update the content here so the code works just fine for now.
Post reply on HN