Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

github.com

Post reply on HN
Print view View original

51–60 of 117 posts

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#51

Post by williamzeng0 »

z3t4 wrote: post #49
williamzeng0 wrote: post #6

Earlier quoted context omitted.

Thats a good point, I really dislike when Sweep fails. That's why we're so focused on PR validation like self-review and GitHub actions, which brings it even closer to a junior dev. We wrote another blog on it here: https://docs.sweep.dev/blogs/giving-dev-tools There's still a long way to go on automated testing, building, and running code, but I don't see any reason it's not possible!

Another buisness idea is to make repos look more active by giving Sweep different personas.

Good point, we may allow open source repos to do this in order to credit contributors that wrote the original issues (those contributions are really valuable).

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#52

Post by thomasrockhu »

IshKebab wrote: post #50
vhanda wrote: post #31

Earlier quoted context omitted.

If I may ask - why? Why is increasing your code coverage to 100% matter? Would that reduce bugs or speed up development in any way? Wouldn't it just add lots more code to maintain and make refactors more time consuming?

100% code coverage doesn't guarantee there are no bugs, but less than 100% code coverage does mean that there is code that you definitely aren't testing. To put it another way, code coverage isn't a direct measure of how good your testing is, but it is still a useful metric to try and improve. In most cases 100% is too hardcore a target, but you should probably aim for at least 80%.

Tom from Codecov here. This is so true, 80% is usually a much more reasonable approach. It’s better to write good tests than all the tests.

(Shameless plug) I wrote a short post about this here: https://about.codecov.io/blog/the-case-against-100-code-cove...

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#53

Post by williamzeng0 »

thomasrockhu wrote: post #52
IshKebab wrote: post #50

Earlier quoted context omitted.

100% code coverage doesn't guarantee there are no bugs, but less than 100% code coverage does mean that there is code that you definitely aren't testing. To put it another way, code coverage isn't a direct measure of how good your testing is, but it is still a useful metric to try and improve. In most cases 100% is too hardcore a target, but you should probably aim for at least 80%.

Tom from Codecov here. This is so true, 80% is usually a much more reasonable approach. It’s better to write good tests than all the tests. (Shameless plug) I wrote a short post about this here: https://about.codecov.io/blog/the-case-against-100-code-cove...

I really liked the blogpost! We're hoping to change point 2 (Engineering Time is Finite) with Sweep, so hopefully we don't have to make a tradeoff between a high quantity/quality of tests.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#54

Post by csmpltn »

gcanyon wrote: post #44

Your demo video https://www.youtube.com/watch?v=WBVna_ow8vo is ridiculously compelling. You need to make a better version of the video, and maybe a few more of them.

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR.

All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with.

Good luck!

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#55

Post by padolsey »

Love it!! The chunking stuff especially is really impressive. Hitting those token limits often is the annoying bit of working with LLMs.

A weird question: How do you feel about possibly ~wasted efforts of these techniques when gpt in a year or so is probably gonna be 100k+ in context length? I've felt this a bit. E.g. I really want to create a 'massive document' conversational agent but I'm doing around 90% of work just juggling and preempting token constraints with super hueristic indexing. I just feel it's all a bit.. wasted, in terms of effort. At some point the LLM apis (openai, claude, ..) will just accept massive zips of code and use them as entire prompts without need for these creative trickeries. Thoughts?

Oh! And have you tried out the function-calling APIs? I see you've found that XML is far more reliable as it's semantically enriched. I have found this to be the case as well, which is a shame because I really want the function-calling stuff to work equally well.

I'm loving stuff like this that starts to pseudo-expand the token limit.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#56

Post by williamzeng0 »

csmpltn wrote: post #54
gcanyon wrote: post #44

Your demo video https://www.youtube.com/watch?v=WBVna_ow8vo is ridiculously compelling. You need to make a better version of the video, and maybe a few more of them.

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR. All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with. Good luck!

That's completely right, the testimonials will look really strange if the names have 3+ words in them. That's why we're targeting really strong developers to review Sweep's PRs. An experienced dev(like you) will be able to read the code, think "hey this needs tests and edge cases" and then request changes instead of merging it.

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#57

Post by willsmith72 »

Is it possible to provide feedback to a PR? One of the best parts about these AIs is their ability to adapt based on feedback.

E.g. in the demo video, the code doesn't cover if splitName.length === 0. I would want to prompt it to cover that case as well

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#58

Post by williamzeng0 »

padolsey wrote: post #55

Love it!! The chunking stuff especially is really impressive. Hitting those token limits often is the annoying bit of working with LLMs. A weird question: How do you feel about possibly ~wasted efforts of these techniques when gpt in a year or so is probably gonna be 100k+ in context length? I've felt this a bit. E.g. I really want to create a 'massive document' conversational agent but I'm doing around 90% of work j…

That's a good question! We tried using Anthropic 100k before (Claude 1.3 was a lot worse), and I think that it's really important to figure out how to be context efficient, at least for GPT4.

My stance is with models ignoring long contexts(https://arxiv.org/pdf/2307.03172.pdf), we'll have this problem for a long time. I could be wrong though.

Also we did try function calling, but it doesn't allow for a chain of thought step. This made the plan/code way worse. Cool to see you found the same!

Re: Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase

#59

Post by williamzeng0 »

willsmith72 wrote: post #57

Is it possible to provide feedback to a PR? One of the best parts about these AIs is their ability to adapt based on feedback. E.g. in the demo video, the code doesn't cover if splitName.length === 0. I would want to prompt it to cover that case as well

Yep! You can leave a comment on the file just like you would review a PR. There's an example here: https://github.com/sweepai/landing-page/pull/226
Post reply on HN