Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

news.ycombinator.com

Post reply on HN
Print view View original

231–240 of 248 posts

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#231

Post by senko »

reportgunner wrote: post #220
haxton wrote: post #24

The demos I see for these types of tools are always some toy project and doesn't reflect day to day work I do at all. Do you have any example PRs on larger more complex projects that have been written with codebuff and how much of that was human interactive? The real problem I want someone to solve is helping me with the real niche/challenging portion of a PR, ex: new tiptap extension that can do notebook code eval,…

Watching the demo it seems like it would be more effective to learn the skills you need rather than using this for a decade. It takes 5+ seconds just to change one field to dark mode, I don't even want to imaigne a situation where I have two fields and I want to explain that I need to change this field and not that field. I'm not sure who is the target audience for this, people who want to be programmers without lear…

My 2c as someone who worked on a similar product:

> it seems like it would be more effective to learn the skills you need rather than using this for a decade.

Think of it as a calculator. You do want to be able to do addition, but not neccessarily to manually add 4-digit numbers in your head.

> It takes 5+ seconds just to change one field to dark mode

Our current LLMs are way too slow for this. I am chuckling every time someone says "we don't need LLMs to be faster because people can't read faster". Imagine this using Groq with a future model with similar capability level, and taking 0.5 seconds to do this small change.

People need to remember we're at the very beginning of using AI for coding. Of course it's suboptimal for majority of cases. Unless you believe we're way past half the sigmoid curve on AI improvements (which I don't), consider that this is the worst the AI is ever going to be for coding.

A year ago people were incredulous when told that AI could code. A year before that people would laugh you out of the room. Now we're at the stage where it kinda works, barely, sometimes. I'm bullish on the future.

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#232

Post by zh2408 »

jeswin wrote: post #186
haxton wrote: post #24

The demos I see for these types of tools are always some toy project and doesn't reflect day to day work I do at all. Do you have any example PRs on larger more complex projects that have been written with codebuff and how much of that was human interactive? The real problem I want someone to solve is helping me with the real niche/challenging portion of a PR, ex: new tiptap extension that can do notebook code eval,…

> Do you have any example PRs on larger more complex projects that have been written with codebuff and how much of that was human interactive? We have a lot of code in production which are AI written. The important thing is that you need to consciously make a module or project AI-ready. This means that things like modularity and smaller files are even more important than they usually are. I can't share those PRs, but…

This is so REAL: LLMs suck probably because your modularity sucks LOL

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#233

Post by michaelmior »

brandonchen wrote: post #95
michaelmior wrote: post #78

Earlier quoted context omitted.

It sounds like it's arguably still a form of RAG, just where the retrieval is very different. I'm not saying that to knock your approach, just saying that it sounds like it's still the case where you're retrieving some context and then using that context to augment further generation. (I get that's definitely not what people think of when you say RAG though.)

Genuine question: at what point does the term RAG lose its meaning? Seems like LLMs work best when they have the right context, and that context must be pulled from somewhere for the LLM. But if that's RAG, then what isn't? Do you have a take on this? Been struggling to frame all this in my head, so would love some insight.

I think parsimo2010 gave a good definition. If you're pulling context from somewhere using some search process to include as input to the LLM, I would call that RAG.

So I would not consider something like using a system prompt (which does add context, but does not involve search) would not be RAG. Also, using an LLM to generate search terms before returning query results would not be RAG because the output of the search is not input to the LLM.

I would also probably not categorize a system similar to Codebuff that just adds the entire repository as context to be RAG since there's not really a search process involved. I could see that being a bit of a grey area though.

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#234

Post by zh2408 »

How does it work if I'm not adding features, but want to refactor my code bases? E.g., the OOD is poor, and I want to totally change it and split the codes into new files? Would it work properly as it requires extensive reads + create new files + writes ...

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#235

Post by jahooma »

abossy wrote: post #230

> I fine-tuned GPT-4o to turn Claude's sketch of changes into a git patch, which would add and remove lines to make the edits. I only finished generating the training data late at night, and the fine-tuning job ran as I slept Could you say more about this? What was the entirety of your training data, exactly, and how did the sketch of changes and git patch play into that?

Sure! I git cloned some open source projects, and wrote a script (with Codebuff) to pick commits and individual diffs of files. For each of those, I had Claude write a sketch of what changed from the old file to the new.

This is all the data I need: the old file, the sketch of how Claude would update it, and the ground truth diff that should be produced. I compiled this into the ideal conversation where the assistant responds with the perfect patch, and that became the training set. I think I had on the order of ~300 of these conversations for the first run, and it worked pretty well.

I came up with more improvements too, like replacing all the variant placeholder comments like "// ... existing code ..." or "# ... (keep the rest of the function)" with one [[*REPLACE_WITH_EXISITNG_CODE*]] symbol, and that made it more accurate

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#236

Post by taskforcegemini »

redman25 wrote: post #225
jahooma wrote: post #192

Earlier quoted context omitted.

Thanks for your reply! I started Codebuff without being aware of Aider. I actually have not yet tried Aider (though I plan to try it soon!). It's totally true that a lot of the development of Codebuff is merely me (and Brandon) working through a lot of the problems that Aider already solved! That makes sense. Partly, my thesis is that if you start after Sonnet 3.5 is out, that you design things differently. For examp…

> With respect to privacy, we have pledged not to store your codebase [...] It isn't necessarily a strong guarantee to have "pledged", although it is appreciated.

Amber Heard ruined that word for me.

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#237

Post by toisanji »

FergusArgyll wrote: post #137
toisanji wrote: post #76

Earlier quoted context omitted.

That sounds cool and I like the idea, but definitely won't pay 5x. Maybe charge $30/month plus bring your own key. Let me know when you lower the price :)

https://aider.chat/

I've played with aider and didn't like it, is it essentially the same?

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#238

Post by kajecounterhack »

dartos wrote: post #226
kajecounterhack wrote: post #154

Earlier quoted context omitted.

> There’s still a skill floor required to accurately judge something. Sure but it's not high at all. Your typical sysadmin is doing a lot of Googling. If perplexity can tell you exactly what to do 90% of the time without error, that's a pretty good sysadmin. Your typical programmer is doing a lot of googling and write-eval loops. If you are doing many flawless write-eval loops with the help of cline, cline is a prett…

> Sure but it's not high at all. It depends. For a sysadmin maybe not, but for data scientists, the bar would be pretty high just to understand the math jargon. > If perplexity can tell you exactly what to do 90% of the time without error That “if” is carrying a lot of weight. Anecdotally I haven’t seen any llm be correct 90% of the time. IIRC SOTA on swebench (which tbf isn’t a great benchmark) is around 30%. > flaw…

> A lot of things AI is helping with also have good, easy to observe / generate, real-time metrics you can use to judge excellence.

Exactly what I illustrated earlier: your developer productivity metrics. If you're turning code around faster, setting up your network better, turning around insights faster, the AI is working.

> It depends. For a sysadmin maybe not, but for data scientists, the bar would be pretty high just to understand the math jargon.

Why does an AI coding agent need to understand math jargon -- it just helps you write better code. Are you even familiar with what data scientists do? Seems not because if you were, you'd see clearly where the tool would be applied and do a good/bad job.

Reminder: we're talking about evaluating whether Codebuff / alternatives are "pretty good" at X. Just go play with the tools. tgtweak expressed their opinion on how good the tool rates at some tasks {sysadmin, data engineering, cloud architecture} and your response was to question how someone could have an opinion about it. The obvious answer is that they used the tools and found it useful for those tasks. It may only be _subjectively_ good at what they're using for but it's also a rando's opinion on the internet. As another rando I very much agree with what the person you responded to is saying. You're not going to get more rigor from this discourse - go form a real opinion of your own.

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#239

Post by abossy »

jahooma wrote: post #235
abossy wrote: post #230

> I fine-tuned GPT-4o to turn Claude's sketch of changes into a git patch, which would add and remove lines to make the edits. I only finished generating the training data late at night, and the fine-tuning job ran as I slept Could you say more about this? What was the entirety of your training data, exactly, and how did the sketch of changes and git patch play into that?

Sure! I git cloned some open source projects, and wrote a script (with Codebuff) to pick commits and individual diffs of files. For each of those, I had Claude write a sketch of what changed from the old file to the new. This is all the data I need: the old file, the sketch of how Claude would update it, and the ground truth diff that should be produced. I compiled this into the ideal conversation where the assistant…

Very interesting, thanks!

Re: Launch HN: Codebuff (YC F24) – CLI tool that writes code for you

#240

Post by dartos »

kajecounterhack wrote: post #238
dartos wrote: post #226

Earlier quoted context omitted.

> Sure but it's not high at all. It depends. For a sysadmin maybe not, but for data scientists, the bar would be pretty high just to understand the math jargon. > If perplexity can tell you exactly what to do 90% of the time without error That “if” is carrying a lot of weight. Anecdotally I haven’t seen any llm be correct 90% of the time. IIRC SOTA on swebench (which tbf isn’t a great benchmark) is around 30%. > flaw…

> A lot of things AI is helping with also have good, easy to observe / generate, real-time metrics you can use to judge excellence. Exactly what I illustrated earlier: your developer productivity metrics. If you're turning code around faster, setting up your network better, turning around insights faster, the AI is working. > It depends. For a sysadmin maybe not, but for data scientists, the bar would be pretty high…

Wow… why’d you get so defensive and presumptuous?

I have my opinion, it’s just not the same as yours.

Post reply on HN