How to train your own large language models • Hacker News Board

Re: How to train your own large language models

#41

Post by nemoniac » Thu, Apr 20, 2023, 6:08 AM UTC

koboll wrote: post #7

Ghostwriter is notably worse than GPT-4, so while it may be true in a sense that "Training a custom model allows us to tailor it to our specific needs and requirements", the reality is they'd be getting better results just using OpenAI right now. Probably true for almost every other use case. That said, I am patiently waiting and champing at the bit for the day this isn't true anymore. Cool to see the groundwork bein…

If one of your specific needs and requirements is that you do not share data with OpenAI then this is a viable option.

Re: How to train your own large language models

#42

Post by detrites » Thu, Apr 20, 2023, 6:50 AM UTC

jxi wrote: post #22

Is this just a high level description of the process? There isn’t really any actual code to run or tutorial to follow.

Yes, they mention they plan to publish articles with specific details later on as well as to open source some models.

Re: How to train your own large language models

#43

Post by detrites » Thu, Apr 20, 2023, 6:59 AM UTC

plondon514 wrote: post #3

VWWHFSfQ wrote: post #2

Did we ever get any resolution about what happened after this company threatened to sue their intern for making a side project that supposedly stole all their great ideas? I would like to know before I ever consider anything from them again.

The founder admitted his mistake and the ex-intern's site is back up and running https://riju.codes/ . I'm personally a fan of both Amjad's (CEO) and Radon's (intern) and realize that everyone makes mistakes. It's not a reason to discount the hard work of the people at replit.

Not at all familiar with the details of this, but just to generally observe the bigger the mistake - the harder the walk-back and the higher the chance a lesson was learned. No guarantees, but it's rare to get past the first stage.

Re: How to train your own large language models

#44

Post by lxe » Thu, Apr 20, 2023, 7:06 AM UTC

Shameless plug: I built a toy tool to really quickly get into llm finetuning by just pasting your training samples into a textbox:

https://github.com/lxe/simple-llm-finetuner

Re: How to train your own large language models

#45

Post by ftxbro » Thu, Apr 20, 2023, 7:18 AM UTC

dkhudia wrote: post #38

ftxbro wrote: post #4

How expensive is it? My understanding is that it's not reasonable to train an LLM from scratch by yourself, and that if you want one that isn't just very stupid then you need to spend between hundreds of thousands and hundreds of millions of dollars. But if you don't want to train from scratch then you can fine-tune existing models for cheaper.

Disclaimer: I work for MosaicML (MosaicML is the creator of the training platform used by Replit). Training these models from scratch on your domain specific data is not as expensive as one might think. We have provided some cost estimates in our blogs. https://www.mosaicml.com/blog/mosaicbert https://www.mosaicml.com/blog/training-stable-diffusion-from... https://www.mosaicml.com/blog/gpt-3-quality-for-500k

Thank you this is very interesting!

Re: How to train your own large language models

#46

Post by moltar » Thu, Apr 20, 2023, 7:51 AM UTC

dkhudia wrote: post #38

ftxbro wrote: post #4

How expensive is it? My understanding is that it's not reasonable to train an LLM from scratch by yourself, and that if you want one that isn't just very stupid then you need to spend between hundreds of thousands and hundreds of millions of dollars. But if you don't want to train from scratch then you can fine-tune existing models for cheaper.

Disclaimer: I work for MosaicML (MosaicML is the creator of the training platform used by Replit). Training these models from scratch on your domain specific data is not as expensive as one might think. We have provided some cost estimates in our blogs. https://www.mosaicml.com/blog/mosaicbert https://www.mosaicml.com/blog/training-stable-diffusion-from... https://www.mosaicml.com/blog/gpt-3-quality-for-500k

Do you have any examples on how to train a model that can write code but in a specific domain? Eg I only want to train it on a specific set of code. Eg let’s say functional React components in TypeScript.

Re: How to train your own large language models

#47

Post by ZephyrBlu » Thu, Apr 20, 2023, 9:20 AM UTC

berkle4455 wrote: post #34

plondon514 wrote: post #3

Earlier quoted context omitted.

The founder admitted his mistake and the ex-intern's site is back up and running https://riju.codes/ . I'm personally a fan of both Amjad's (CEO) and Radon's (intern) and realize that everyone makes mistakes. It's not a reason to discount the hard work of the people at replit.

Thanks for linking this. This is actually a superior offering to replit. They recently removed the ability to access a simple repl without logging in. Now you a) have to login and b) have to deal with this obtuse IDE-in-a-browser project creation shit. It's so many extra steps before I can run code. I just want a URL in which I can run some code. https://riju.codes/ is literally that. Thanks!

I ran into the same thing and finally made a Replit account. I'm just gonna use Riju from now on though. Using Replit with an account is way more janky than it was without needing to login.

Re: How to train your own large language models

#48

Post by rmbyrro » Thu, Apr 20, 2023, 10:15 AM UTC

> "a student coding on their phone in India should have access to the same AI as a professional developer in Silicon Valley. To make this possible, we train custom models with reduced cost."

In principle, that's great. But the reality is: whoever has the resources and benefits from something better will look for ways to get it. What they're communicating here is: the most resourceful developers on the planet aren't our ideal customer.

Re: How to train your own large language models

#49

Post by Cypher » Thu, Apr 20, 2023, 10:52 AM UTC

nextworddev wrote: post #23

johnfn wrote: post #20

Earlier quoted context omitted.

The CEO refused to apologize, and instead doubled down, taking advantage of a massive power differential between himself and a random college grad. He only apologized when the differential evaporated after the post hit the top of HN with something like 3000 points. I don't know about you, but I don't find that to be particularly acceptable, nor a "mistake", and I'm happy to continue to punish a CEO's unethical behavi…

Sounds pretty toxic

Why is he toxic? reminds me of those media influencers calling for others to lose their jobs, freedoms and lives because they wouldn't take the vaccine.

I'll never forgive them for their toxicity and I don't think that makes me toxic.

Re: How to train your own large language models

#50

Post by alpaca128 » Thu, Apr 20, 2023, 11:08 AM UTC

koboll wrote: post #7

Ghostwriter is notably worse than GPT-4, so while it may be true in a sense that "Training a custom model allows us to tailor it to our specific needs and requirements", the reality is they'd be getting better results just using OpenAI right now. Probably true for almost every other use case. That said, I am patiently waiting and champing at the bit for the day this isn't true anymore. Cool to see the groundwork bein…

Not everyone wants to depend on and trust a cloud service, and not everyone needs GPT-4 quality.

If there's a viable way to tune and run models locally they could still be useful if you don't need it to play chess and imitate a Python interpreter at the same time.