Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

finecodex.com

Post reply on HN
Print view View original

61–70 of 79 posts

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#61

Post by samatdav »

tucnak wrote: post #56
uxhacker wrote: post #41

Earlier quoted context omitted.

It’s also weird that there is no about page naming the founders.

There's a reason why you don't see frontier-grade AI researchers throwing around meaningless numbers to go with the most layman idea of a product in the field imaginable. The whole thing stinks. I reckon this is some kind of extortion scam intended to trick people into compromising IP.

I understand the concern but we don't need anyone's IP. Unfortunately, it is hard to provide fine-tuning solution without access to the codebase. We just think that using a large general-purpose model for a highly specific codebase with a lot of internal frameworks is not the best solution and want to try to improve it.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#62

Post by samatdav »

dr_kiszonka wrote: post #5
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

I wish you posted more evaluation details on your page as text. What exactly was your accuracy vs. Sonnet? (Right now, we can only tell that Sonnet's was ≤ 1/4.3.) Why the Discourse repo? Providing more detailed information would help folks trust your claims more.

I agree, we need to post more data. Since we are very early (<1 month) we just shared the initial results. Discourse repo was just a good option since it is a big public repo that could benefit from fine-tuning. We plan to add more benchmarks to the website as we progress.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#63

Post by samatdav »

doubtfuluser wrote: post #13
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

Any plans on distilling it down to an 8b model to enable it for pure local usage on most consumer hardware?

Could be done in the future. Our current focus is highest accuracy. But there are no limitations on the models - just would depend on user preference of size/performance tradeoff.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#64

Post by samatdav »

rtfeldman wrote: post #54
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

Interested! Our large Rust code base at https://zed.dev is open-source at https://github.com/zed-industries/zed and I'd be curious to try this out on it. My email is richard at our website's domain if you'd like to get in touch!

Looks like a great repo to try the fine-tuning! I will email you, thanks!

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#65

Post by samatdav »

ryao wrote: post #37
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

I would be interested in a fine tune on OpenZFS: https://github.com/openzfs/zfs

Thank you for the suggestion, we will take a look!

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#66

Post by samatdav »

redman25 wrote: post #43
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

What is the metric for LLMs? Shouldn't more than just accuracy be measured? If something has high accuracy but low recall, won't it be overfit and fail to generalize? Your metrics would give you false confidence in how effective your model is. Just wondering because the announcement only seems to mention accuracy.

Good point, we should provide more detailed metrics. Since we are very early, we focus on the main metric in our view: higher accuracy of changes to be more practically usable. We will do more testing on overfitting and how the model performance on different types of tasks. On high level we believe in the idea of "a well fine-tuned model should be much better than a large general model". But we need more metrics, I agree.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#67

Post by samatdav »

pdimitar wrote: post #9
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

Some more details that programmers can inspect would be very useful.

I agree, we plan to publish more benchmarks and metrics. We also want to publicly host our fine-tuned model for one of the open-source repos so that people can try themselves agains SOTA models.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#68

Post by samatdav »

futureshock wrote: post #42
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

It seems an interesting fine-tuning idea. Drawing from reasoning models, I wonder if it’s effective to 10x or 100x the fine-tune dataset by having a larger reasoning model create documentation and reasoning COTs about the code base’s current state and speculation about future state updates. Maybe have it output some verbose execution flow analysis.

Thank you for the idea! We are also considering upsampling and distillation. But on high level, correctly setting up the data for simple fine-tuning can already produce great results.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#69

Post by samatdav »

siscia wrote: post #11
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

I am getting quite deep into coding with AI and cost of tokens is a bit of an issue indeed. Trivial issue because it saves me A LOT of time, but it could be an issue for new people testing it. I would love to test this approach. Are you guys fine tuning for each codebase?

Yes, we fine-tune for each codebase. Now we are focusing on larger enterprise codebases that would: 1. benefit from the fine-tuning the most. 2. have the budget to pay us for the service. For smaller projects that are price-sensitive we are probably not a good fit at this point.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#70

Post by samatdav »

jbellis wrote: post #48
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

I'm interested. I submitted my email to your landing page form.

Thank you! Will email you within a couple of days:)
Post reply on HN