Live data from Hacker News

We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

finecodex.com

41–50 of 79 posts

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#41
post #38
post #34

> Our team is ex-OpenAI, Anthropic, and Asana research scientists and AI engineers The page includes the logos of those companies. Is it normal to do that for companies one used to work for?

No it's not! These are probably imposters anyway. Apparently, you can buy HN upvotes... I find it hard to believe that honest researchers from frontier labs would behave like crypto scammers.

It’s also weird that there is no about page naming the founders.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#42
post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

It seems an interesting fine-tuning idea. Drawing from reasoning models, I wonder if it’s effective to 10x or 100x the fine-tune dataset by having a larger reasoning model create documentation and reasoning COTs about the code base’s current state and speculation about future state updates. Maybe have it output some verbose execution flow analysis.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#43
post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

What is the metric for LLMs? Shouldn't more than just accuracy be measured? If something has high accuracy but low recall, won't it be overfit and fail to generalize? Your metrics would give you false confidence in how effective your model is. Just wondering because the announcement only seems to mention accuracy.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#44
post #34

> Our team is ex-OpenAI, Anthropic, and Asana research scientists and AI engineers The page includes the logos of those companies. Is it normal to do that for companies one used to work for?

I've seen plenty of startups who's single-page includes the pitch, and the founders section which has big logos of their alma mater (almost always Ivy leaguers + Stanford) and whatever FAANG or consulting job gig they had previously.

Easier to land investors and customers if you have the "correct" pedigree.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#45
post #29

Earlier quoted context omitted.

What's CoT?

Chain of Thought. When I see people using abbreviations like this I sometimes jokingly wonder what they do with all this time they're saving.

There are so many chain types it is easier to do the abbreviations. Basically extend a RAG to have a graph to influence how to either critisize itself or perform different actions. It has gotten to the point where there are libraries for define them. https://langchain-ai.github.io/langgraph/tutorials/introduct...

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#46
post #38

Earlier quoted context omitted.

No it's not! These are probably imposters anyway. Apparently, you can buy HN upvotes... I find it hard to believe that honest researchers from frontier labs would behave like crypto scammers.

It’s also weird that there is no about page naming the founders.

https://x.com/karpenoid/status/1670723794544263170

https://x.com/karpenoid/status/1873281722613400002

This might be linked.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#47
post #34

> Our team is ex-OpenAI, Anthropic, and Asana research scientists and AI engineers The page includes the logos of those companies. Is it normal to do that for companies one used to work for?

I've seen plenty of startups who's single-page includes the pitch, and the founders section which has big logos of their alma mater (almost always Ivy leaguers + Stanford) and whatever FAANG or consulting job gig they had previously. Easier to land investors and customers if you have the "correct" pedigree.

Private pitch decks aren’t the same as public product websites.

It seems this team is using the old scammy marketing trick of using logos of every company you can claim any possible relationship with as a way of building trust. Plastering giant logos on a product website implies some endorsement or affiliation to most casual readers. It’s not until you read all of the text that you realize this is just a list of companies they worked at.

This is the kind of behavior that earns a sternly worded letter from corporate council. You shouldn’t expect to be able to leave a company and then put their logo on your product page.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#48
post #2

Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.

I'm interested. I submitted my email to your landing page form.

Re: We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation

#50
post #46

Earlier quoted context omitted.

It’s also weird that there is no about page naming the founders.

https://x.com/karpenoid/status/1670723794544263170 https://x.com/karpenoid/status/1873281722613400002 This might be linked.

It says they WILL fine tune a model.

Sounds fishy

Post reply on HN