We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
1–10 of 79 posts
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#2Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#3Makes a ton of sense!
Is this for completions, patches, or new files?
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#4Makes a ton of sense! Is this for completions, patches, or new files?
Hi! Currently we generate a whole diff (like cmd+shift+k in Cursor). But plan to add there rest soon!:)
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#5Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.
I wish you posted more evaluation details on your page as text. What exactly was your accuracy vs. Sonnet? (Right now, we can only tell that Sonnet's was ≤ 1/4.3.) Why the Discourse repo? Providing more detailed information would help folks trust your claims more.
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#6That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#7Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.
Talk is cheap, benchmarks please. Also why did you decide for LLama? AFAIK deepseek always had a slight edge over llama when it comes to coding performance, or is this no longer the case?
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#8Is the source code available for inspection somewhere? It's not really clear from the landing page.
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#9Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.
Some more details that programmers can inspect would be very useful.
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#10[deleted]