Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

finecodex.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#31

Post by mentalgear »

redeux wrote: post #27
eurekin wrote: post #6

That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!

Almost no one knows if a project/business idea will be successful or not, so it's not much use asking. It's more productive to ask smart, experienced people how to best validate and execute an idea. People generally give useful and actionable feedback based on their experiences. Just make sure you understand who you're talking to when evaluating someone's advice.

"understand who you're talking to when evaluating someone's advice." Good you mentioned this, found out to this is a crucial part as well: Always perceive the advice you get depending on that person's background and interests (e.g. your target group, or domain-foreign expert).

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#32

Post by elashri »

siscia wrote: post #25
MacsHeadroom wrote: post #22

Earlier quoted context omitted.

DeepSeek is open source and has a context length of 128k tokens.

Commercial service have a context of 64k tokens, which I find quite limiting. https://api-docs.deepseek.com/quick_start/pricing Running it locally is quite a bit beyond the scope of being productive while coding with AI. Beside that 128k is still significantly less than Claude

Shouldn't we be comparing with other open source model? In particular since this is about llama3.3 then they have the exact context limit which is 128k [1]. Also

[1] https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#33

Post by mentalgear »

I like it and it makes sense, but from a business perspective I wonder what keeps the upstream LLM providers (all trying to generate profits) from offering the same fine-tuning service quickly ?

Edit: OK, right it's olama, so I assume you can download your own model. (Assuming it's downloadable?)

I think openAI already offers fine-tuning with custom data for some of their models, but maybe not specific to coding tasks.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#35

Post by plagiarist »

vlabakje90 wrote: post #30
eru wrote: post #29

Earlier quoted context omitted.

What's CoT?

Chain of Thought. When I see people using abbreviations like this I sometimes jokingly wonder what they do with all this time they're saving.

Perhaps they're preemptively reducing several tokens into one, for the machines' benefit.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#36

Post by tucnak »

banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

I'm not saying you're an imposter... but you're making it really easy to assume that; it doesn't seem you have learnt much while you guys were there. Are you sure you weren't hired by mistake?

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#37

Post by ryao »

banddk wrote: post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

I would be interested in a fine tune on OpenZFS:

https://github.com/openzfs/zfs

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#38

Post by tucnak »

tkgally wrote: post #34

> Our team is ex-OpenAI, Anthropic, and Asana research scientists and AI engineers The page includes the logos of those companies. Is it normal to do that for companies one used to work for?

No it's not! These are probably imposters anyway. Apparently, you can buy HN upvotes... I find it hard to believe that honest researchers from frontier labs would behave like crypto scammers.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#39

Post by siscia »

elashri wrote: post #32
siscia wrote: post #25

Earlier quoted context omitted.

Commercial service have a context of 64k tokens, which I find quite limiting. https://api-docs.deepseek.com/quick_start/pricing Running it locally is quite a bit beyond the scope of being productive while coding with AI. Beside that 128k is still significantly less than Claude

Shouldn't we be comparing with other open source model? In particular since this is about llama3.3 then they have the exact context limit which is 128k [1]. Also [1] https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct

Why?

Whenever using a model to be more effective as a developer I don't particularly care if the model is open source or closed source.

I would love to use open source models as well, but the convenience to just plug an API against some endpoints in unbeatable.

Post reply on HN