Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

finecodex.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#21

Post by siscia »

manishsharan wrote: post #16
siscia wrote: post #11

Earlier quoted context omitted.

I am getting quite deep into coding with AI and cost of tokens is a bit of an issue indeed. Trivial issue because it saves me A LOT of time, but it could be an issue for new people testing it. I would love to test this approach. Are you guys fine tuning for each codebase?

>>cost of tokens is a bit of an issue indeed Their cost is $0.7 per 1M token. DeepSeek is $0.14 / 1M tokens ( cache miss)

DeepSeek is an amazing product but has few issues:

1. Data is used for training

2. Context window is rather small and doesn't fit as well large codebase

I keep saying this over and over in all the content I create, the valu of coding with AI will come from working on big, complex, legacy codebases. Not from flashy demo where you create a to-do app.

For that you need solid models with big context and private inference.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#22

Post by MacsHeadroom »

siscia wrote: post #21
manishsharan wrote: post #16

Earlier quoted context omitted.

>>cost of tokens is a bit of an issue indeed Their cost is $0.7 per 1M token. DeepSeek is $0.14 / 1M tokens ( cache miss)

DeepSeek is an amazing product but has few issues: 1. Data is used for training 2. Context window is rather small and doesn't fit as well large codebase I keep saying this over and over in all the content I create, the valu of coding with AI will come from working on big, complex, legacy codebases. Not from flashy demo where you create a to-do app. For that you need solid models with big context and private inference…

DeepSeek is open source and has a context length of 128k tokens.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#24

Post by menaerus »

eurekin wrote: post #6

That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!

> That's exactly what everybody advised me against doing - finetuning on own projects

Why would someone advise against it? IMHO that sounds as the end game to me. If it weren't so darn expensive, I'd try this for myself for sure.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#25

Post by siscia »

MacsHeadroom wrote: post #22
siscia wrote: post #21

Earlier quoted context omitted.

DeepSeek is an amazing product but has few issues: 1. Data is used for training 2. Context window is rather small and doesn't fit as well large codebase I keep saying this over and over in all the content I create, the valu of coding with AI will come from working on big, complex, legacy codebases. Not from flashy demo where you create a to-do app. For that you need solid models with big context and private inference…

DeepSeek is open source and has a context length of 128k tokens.

Commercial service have a context of 64k tokens, which I find quite limiting.

https://api-docs.deepseek.com/quick_start/pricing

Running it locally is quite a bit beyond the scope of being productive while coding with AI.

Beside that 128k is still significantly less than Claude

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#26

Post by freehorse »

menaerus wrote: post #24
eurekin wrote: post #6

That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!

> That's exactly what everybody advised me against doing - finetuning on own projects Why would someone advise against it? IMHO that sounds as the end game to me. If it weren't so darn expensive, I'd try this for myself for sure.

I think that people suggest RAG, also because the models develop so fast that very probably the base model you finetune on will be obsolete in a year or so.

If we are approaching diminishing returns it makes more sense to finetune. As the recent advances seem to happen by throwing more compute to CoT etc maybe the time is close or has already come.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#27

Post by redeux »

eurekin wrote: post #6

That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!

Almost no one knows if a project/business idea will be successful or not, so it's not much use asking. It's more productive to ask smart, experienced people how to best validate and execute an idea. People generally give useful and actionable feedback based on their experiences. Just make sure you understand who you're talking to when evaluating someone's advice.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#28

Post by QuesnayJr »

eurekin wrote: post #6

That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!

Other people have spent a lot of time on it and gotten nowhere, so I suspect there is some art to it.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#29

Post by eru »

freehorse wrote: post #26
menaerus wrote: post #24

Earlier quoted context omitted.

> That's exactly what everybody advised me against doing - finetuning on own projects Why would someone advise against it? IMHO that sounds as the end game to me. If it weren't so darn expensive, I'd try this for myself for sure.

I think that people suggest RAG, also because the models develop so fast that very probably the base model you finetune on will be obsolete in a year or so. If we are approaching diminishing returns it makes more sense to finetune. As the recent advances seem to happen by throwing more compute to CoT etc maybe the time is close or has already come.

What's CoT?

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#30

Post by vlabakje90 »

eru wrote: post #29
freehorse wrote: post #26

Earlier quoted context omitted.

I think that people suggest RAG, also because the models develop so fast that very probably the base model you finetune on will be obsolete in a year or so. If we are approaching diminishing returns it makes more sense to finetune. As the recent advances seem to happen by throwing more compute to CoT etc maybe the time is close or has already come.

What's CoT?

Chain of Thought. When I see people using abbreviations like this I sometimes jokingly wonder what they do with all this time they're saving.
Post reply on HN