Live data from Hacker News

We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

finecodex.com

51–60 of 79 posts

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#51
Here's a Hugging Face blog post where they walk through how to fine tune a model on your code base: https://huggingface.co/blog/personal-copilot .

Kudos to the founders for shipping. I do think this kind of functionality will become very rapidly commoditized though. But then, I suppose people said the same thing about Dropbox.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#53
post #6

That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!

Fine tuning to a specific codebase is a bit strange. It's going to learn some style/tool guidance which is good (but there are other ways of getting), at the risk of unlearning some generalization it learned from looking at 1,000,000x more code samples of varied styles.

In general I'd suggest trying this first:

- Large context: use large context models to load relevant files. It can pickup your style/tool choices fine this way without fine tuning. I'm usually manually inserting files into context, but a great RAG solution would be ideal.

- Project specific instructions (like .cursorrules): tell it specific things you want. I tell it preferred test tools/strategies/styles.

I am curious to see more detailed evals here, but the claims are too high level to really dive into.

In generally: I love fine tuning for more specific/repeatable tasks. I even have my own fine-tuning platform (https://github.com/Kiln-AI/Kiln). However coding is very broad. Good use case for foundation large models with smart use of context.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#54
post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

Interested! Our large Rust code base at https://zed.dev is open-source at https://github.com/zed-industries/zed and I'd be curious to try this out on it.

My email is richard at our website's domain if you'd like to get in touch!

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#55
post #29

Earlier quoted context omitted.

What's CoT?

Chain of Thought. When I see people using abbreviations like this I sometimes jokingly wonder what they do with all this time they're saving.

I post in twitter and invest in crypto

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#56
post #38

Earlier quoted context omitted.

No it's not! These are probably imposters anyway. Apparently, you can buy HN upvotes... I find it hard to believe that honest researchers from frontier labs would behave like crypto scammers.

It’s also weird that there is no about page naming the founders.

There's a reason why you don't see frontier-grade AI researchers throwing around meaningless numbers to go with the most layman idea of a product in the field imaginable. The whole thing stinks. I reckon this is some kind of extortion scam intended to trick people into compromising IP.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#57
post #6

That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!

Other people have spent a lot of time on it and gotten nowhere, so I suspect there is some art to it.

They have? Is there a write up about that?

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#58
Hi HN! I'm Samat, the co-founder from the video. Thank you for the critical feedback, great points.

0. Is this a scam? No. We're very early (started 1. Company logos. You're right. We're using our work experience as a credibility signal because at this stage that is our main selling point. We'll replace logos with concrete results as we develop.

2. Team. We're 2 software engineers and 1 AI researcher: - I was an AI product engineer at Asana. https://linkedin.com/in/samatd - Denis was a tech lead at a unicorn startup. https://x.com/karpenoid - Our third co-founder works at Anthropic and was previously at OpenAI. Since he is still at Anthropic and planning to leave soon for the startup, I can share his details privately.

3. Claims and transparency. Our "4.2x Sonnet-3.5 accuracy" is an initial estimate from a locally fine-tuned model. Actual results may vary - a small app might not see big improvements, but we believe larger, private enterprise projects could see significant gains. We plan to publish our fine-tuned model so others can verify the results.

4. Competition from LLM providers. Fine-tuning requires complex data cleanup and setup. Enterprise projects have fragmented data, making automation challenging for big providers like OpenAI.

Appreciate the feedback! If you want to chat more 1-1, happy to discuss at hi@finecodex.com Samat.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#59

Earlier quoted context omitted.

I've seen plenty of startups who's single-page includes the pitch, and the founders section which has big logos of their alma mater (almost always Ivy leaguers + Stanford) and whatever FAANG or consulting job gig they had previously. Easier to land investors and customers if you have the "correct" pedigree.

Private pitch decks aren’t the same as public product websites. It seems this team is using the old scammy marketing trick of using logos of every company you can claim any possible relationship with as a way of building trust. Plastering giant logos on a product website implies some endorsement or affiliation to most casual readers. It’s not until you read all of the text that you realize this is just a list of comp…

Good point! We are just very early and our experience is our main selling point. We plan to remove it.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#60
post #50
post #46

Earlier quoted context omitted.

https://x.com/karpenoid/status/1670723794544263170 https://x.com/karpenoid/status/1873281722613400002 This might be linked.

It says they WILL fine tune a model. Sounds fishy

At Asana we did not do any fine-tuning because it was too complicated even for our AI org of 40 engineers. We believe we can do it by setting up and cleaning data correctly.
Post reply on HN