How to train your own large language models
blog.replit.com
How to train your own large language models
1–10 of 63 posts
Re: How to train your own large language models
#2Did we ever get any resolution about what happened after this company threatened to sue their intern for making a side project that supposedly stole all their great ideas? I would like to know before I ever consider anything from them again.
Re: How to train your own large language models
#3Did we ever get any resolution about what happened after this company threatened to sue their intern for making a side project that supposedly stole all their great ideas? I would like to know before I ever consider anything from them again.
The founder admitted his mistake and the ex-intern's site is back up and running https://riju.codes/. I'm personally a fan of both Amjad's (CEO) and Radon's (intern) and realize that everyone makes mistakes. It's not a reason to discount the hard work of the people at replit.
Re: How to train your own large language models
#4How expensive is it? My understanding is that it's not reasonable to train an LLM from scratch by yourself, and that if you want one that isn't just very stupid then you need to spend between hundreds of thousands and hundreds of millions of dollars. But if you don't want to train from scratch then you can fine-tune existing models for cheaper.
Re: How to train your own large language models
#5Did we ever get any resolution about what happened after this company threatened to sue their intern for making a side project that supposedly stole all their great ideas? I would like to know before I ever consider anything from them again.
Story for those who didn't see it: https://intuitiveexplanations.com/tech/replit/
Re: How to train your own large language models
#6How expensive is it? My understanding is that it's not reasonable to train an LLM from scratch by yourself, and that if you want one that isn't just very stupid then you need to spend between hundreds of thousands and hundreds of millions of dollars. But if you don't want to train from scratch then you can fine-tune existing models for cheaper.
Looking at what they're doing here probably not as much as you think.
As you note, with the plethora of open/open-ish LLMs today and LoRA + PEFT you can fine tune with low VRAM and pretty quickly so even a single A100 or whatever cloud GPUs are just fine. I've even seen people pull it off in reasonable time on super cheap T4s, A10s, etc.
I doubt anyone reading a blog post is attempting to train a "true" multi-billion param LLM from scratch.
Re: How to train your own large language models
#7Ghostwriter is notably worse than GPT-4, so while it may be true in a sense that "Training a custom model allows us to tailor it to our specific needs and requirements", the reality is they'd be getting better results just using OpenAI right now. Probably true for almost every other use case.
That said, I am patiently waiting and champing at the bit for the day this isn't true anymore. Cool to see the groundwork being laid for it.
Re: How to train your own large language models
#8How expensive is it? My understanding is that it's not reasonable to train an LLM from scratch by yourself, and that if you want one that isn't just very stupid then you need to spend between hundreds of thousands and hundreds of millions of dollars. But if you don't want to train from scratch then you can fine-tune existing models for cheaper.
Looking at what they're doing here probably not as much as you think. As you note, with the plethora of open/open-ish LLMs today and LoRA + PEFT you can fine tune with low VRAM and pretty quickly so even a single A100 or whatever cloud GPUs are just fine. I've even seen people pull it off in reasonable time on super cheap T4s, A10s, etc. I doubt anyone reading a blog post is attempting to train a "true" multi-billion…
[flagged]
Re: How to train your own large language models
#9Did we ever get any resolution about what happened after this company threatened to sue their intern for making a side project that supposedly stole all their great ideas? I would like to know before I ever consider anything from them again.
The founder admitted his mistake and the ex-intern's site is back up and running https://riju.codes/ . I'm personally a fan of both Amjad's (CEO) and Radon's (intern) and realize that everyone makes mistakes. It's not a reason to discount the hard work of the people at replit.
A badly behaved CEO is absolutely a reason to avoid using a whole company.
Reading through the entire story leaves me with a bad taste in my mouth, especially this bit: "still refused to list any specific part of Replit he thought I had copied, even when I asked him for such details multiple times during the phone call, despite his continuing to claim both privately and publicly that I copied Replit unethically".
I haven't used Replit, but reading about it and looking at riju.codes, I have a hard time believing that there was any secret sauce that was inappropriately used, and the sketchy refusal to give details makes me think it's more about a CEO establishing dominance over the little people than any serious IP concern.
Re: How to train your own large language models
#10Did we ever get any resolution about what happened after this company threatened to sue their intern for making a side project that supposedly stole all their great ideas? I would like to know before I ever consider anything from them again.
The founder admitted his mistake and the ex-intern's site is back up and running https://riju.codes/ . I'm personally a fan of both Amjad's (CEO) and Radon's (intern) and realize that everyone makes mistakes. It's not a reason to discount the hard work of the people at replit.
If you read the blog post from Radon, Amjad's apology comes across as "sorry I got caught" rather than any actual remorse.