Kudos to the founders for shipping. I do think this kind of functionality will become very rapidly commoditized though. But then, I suppose people said the same thing about Dropbox.
We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
51–60 of 79 posts
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#52Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#53That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!
In general I'd suggest trying this first:
- Large context: use large context models to load relevant files. It can pickup your style/tool choices fine this way without fine tuning. I'm usually manually inserting files into context, but a great RAG solution would be ideal.
- Project specific instructions (like .cursorrules): tell it specific things you want. I tell it preferred test tools/strategies/styles.
I am curious to see more detailed evals here, but the claims are too high level to really dive into.
In generally: I love fine tuning for more specific/repeatable tasks. I even have my own fine-tuning platform (https://github.com/Kiln-AI/Kiln). However coding is very broad. Good use case for foundation large models with smart use of context.
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#54Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.
My email is richard at our website's domain if you'd like to get in touch!
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#55Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#56Earlier quoted context omitted.
No it's not! These are probably imposters anyway. Apparently, you can buy HN upvotes... I find it hard to believe that honest researchers from frontier labs would behave like crypto scammers.
It’s also weird that there is no about page naming the founders.
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#57That's exactly what everybody advised me against doing - finetuning on own projects. Got really discouraged and stopped. So glad someone has done it!
Other people have spent a lot of time on it and gotten nowhere, so I suspect there is some art to it.
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#580. Is this a scam? No. We're very early (started 1. Company logos. You're right. We're using our work experience as a credibility signal because at this stage that is our main selling point. We'll replace logos with concrete results as we develop.
2. Team. We're 2 software engineers and 1 AI researcher: - I was an AI product engineer at Asana. https://linkedin.com/in/samatd - Denis was a tech lead at a unicorn startup. https://x.com/karpenoid - Our third co-founder works at Anthropic and was previously at OpenAI. Since he is still at Anthropic and planning to leave soon for the startup, I can share his details privately.
3. Claims and transparency. Our "4.2x Sonnet-3.5 accuracy" is an initial estimate from a locally fine-tuned model. Actual results may vary - a small app might not see big improvements, but we believe larger, private enterprise projects could see significant gains. We plan to publish our fine-tuned model so others can verify the results.
4. Competition from LLM providers. Fine-tuning requires complex data cleanup and setup. Enterprise projects have fragmented data, making automation challenging for big providers like OpenAI.
Appreciate the feedback! If you want to chat more 1-1, happy to discuss at hi@finecodex.com Samat.
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#59Earlier quoted context omitted.
I've seen plenty of startups who's single-page includes the pitch, and the founders section which has big logos of their alma mater (almost always Ivy leaguers + Stanford) and whatever FAANG or consulting job gig they had previously. Easier to land investors and customers if you have the "correct" pedigree.
Private pitch decks aren’t the same as public product websites. It seems this team is using the old scammy marketing trick of using logos of every company you can claim any possible relationship with as a way of building trust. Plastering giant logos on a product website implies some endorsement or affiliation to most casual readers. It’s not until you read all of the text that you realize this is just a list of comp…
Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
#60Earlier quoted context omitted.
https://x.com/karpenoid/status/1670723794544263170 https://x.com/karpenoid/status/1873281722613400002 This might be linked.
It says they WILL fine tune a model. Sounds fishy