Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

finecodex.com

Post reply on HN
Print view View original

71–79 of 79 posts

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#71

Post by samatdav »

WhatsName wrote: post #7
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

Talk is cheap, benchmarks please. Also why did you decide for LLama? AFAIK deepseek always had a slight edge over llama when it comes to coding performance, or is this no longer the case?

Good point, we plan to publish more benchmarks and also publicly host a model for anyone to try. We think Llama is a good option but as we progress we will test other open source models too like deepseek.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#72

Post by samatdav »

prmoustache wrote: post #49

How do you measure code generation accuracy? Are there some base tests and if so how can I ensure the models aren't tuned for those tests only the same way vw cheated the emissions tests on their diesels?

We run a set of change requests on the discourse repo. Good point, we plan to publish more detailed testing benchmarks and metrics on the website.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#74

Post by samatdav »

Imnimo wrote: post #52

4.2x doesn't mean anything if you don't tell me what "accuracy" Sonnet 3.5 had.

I agree. Our local early results were promising were a higher percentage of code change requests produced a functionally correct output. We will post more metrics and data in the future.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#75

Post by samatdav »

mathgeek wrote: post #15
aussieguy1234 wrote: post #12

You make a bold marketing claim, 4.2x Sonnet, but viewing your website, I can see no data or test results to back this up.

Thanks for calling this out, even if it just gets OP to comment with some details/data. Was hoping this would be a shallow or deep dive into the results, but looks like it’s just a marketing post to a marketing page to support a PH launch.

Good point, I agree, we haven't shared enough details. Since we are very early, we only got high level results and want to get feedback on what direction would be most applicable and useful. We plan to add more metrics and data to the website in the future and also want to publicly host a fine-tuned model for anyone to try and see.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#76

Post by samatdav »

jeswin wrote: post #18
pcwelder wrote: post #17

Was the comparison done with or without code context (as obtained using RAG or letting Sonnet ask for files)?

In the absense of other information, looks like a cherry-picked example to me.

We used a single file for the context. It is a cherry-picked example, you are right. I wanted to demonstrate a simple visual change that our model did correctly unlike Sonnet-3.5. Since we are just getting started, we don't have many features like making changes across multiple files in the code editor so it would be harder to demo. Our premise is that a smaller fine-tuned works better than a large, general-purpose SOTA model. We plan to share more metrics and data in the future.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#77

Post by samatdav »

DataDaemon wrote: post #19

Maybe ask this model to create a better landing page?

Thank you, we will!:) This was a quick landing page for us to start the conversation and gather feedback. We are trying to make sure we are not building something that nobody needs.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#78

Post by samatdav »

mentalgear wrote: post #33

I like it and it makes sense, but from a business perspective I wonder what keeps the upstream LLM providers (all trying to generate profits) from offering the same fine-tuning service quickly ? Edit: OK, right it's olama, so I assume you can download your own model. (Assuming it's downloadable?) I think openAI already offers fine-tuning with custom data for some of their models, but maybe not specific to coding task…

Yes, you can download and host the fine-tuned open-source model like Llama. The fine-tuning is easy once you have the data, but gathering and cleaning data is challenging. There are also optimizations like upsampling and distillation that could improve the quality of the resulting model. We had 40 engineers at the Asana AI org and never did the fine-tuning because it is not easy.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#79

Post by samatdav »

mrfinn wrote: post #20

2023: Our tiny model blah blah blah beats GPT4! 2024: Our tiny model blah blah blah beats Claude! 2025: Our tiny model blah blah blah beats ???

Haha, yes it is a pattern. However, the claim here is that "our tiny model beats best model" is applicable for highly specific tasks.
Post reply on HN