Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

finecodex.com

Post reply on HN
Print view View original

11–20 of 79 posts

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#11

Post by siscia »

banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

I am getting quite deep into coding with AI and cost of tokens is a bit of an issue indeed.

Trivial issue because it saves me A LOT of time, but it could be an issue for new people testing it.

I would love to test this approach. Are you guys fine tuning for each codebase?

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#13

Post by doubtfuluser »

banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

Any plans on distilling it down to an 8b model to enable it for pure local usage on most consumer hardware?

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#14

Post by KTibow »

WhatsName wrote: post #7
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

Talk is cheap, benchmarks please. Also why did you decide for LLama? AFAIK deepseek always had a slight edge over llama when it comes to coding performance, or is this no longer the case?

Training and running fine tunes of Deepseek could get expensive.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#15

Post by mathgeek »

aussieguy1234 wrote: post #12

You make a bold marketing claim, 4.2x Sonnet, but viewing your website, I can see no data or test results to back this up.

Thanks for calling this out, even if it just gets OP to comment with some details/data. Was hoping this would be a shallow or deep dive into the results, but looks like it’s just a marketing post to a marketing page to support a PH launch.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#16

Post by manishsharan »

siscia wrote: post #11
banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

I am getting quite deep into coding with AI and cost of tokens is a bit of an issue indeed. Trivial issue because it saves me A LOT of time, but it could be an issue for new people testing it. I would love to test this approach. Are you guys fine tuning for each codebase?

>>cost of tokens is a bit of an issue indeed

Their cost is $0.7 per 1M token.

DeepSeek is $0.14 / 1M tokens ( cache miss)

Post reply on HN