Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to train your own large language models

blog.replit.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: How to train your own large language models

#51

Post by alpaca128 »

birdplant wrote: post #18
plondon514 wrote: post #3

Earlier quoted context omitted.

The founder admitted his mistake and the ex-intern's site is back up and running https://riju.codes/ . I'm personally a fan of both Amjad's (CEO) and Radon's (intern) and realize that everyone makes mistakes. It's not a reason to discount the hard work of the people at replit.

That’s a very generous interpretation of what happened because it wasn’t a “mistake” when he threatened the intern, it was something he purposefully and intentionally did, and doubled down on, even after having significant time to reconsider. Only when there was widespread public criticism of his actions did he backpedal. I’m curious what he’s said or done to make you a fan?

Mistakes are not necessarily accidental or guilt-free. The CEO absolutely made a mistake.

Re: How to train your own large language models

#52

Post by PlantsRpeople2 »

alpaca128 wrote: post #50
koboll wrote: post #7

Ghostwriter is notably worse than GPT-4, so while it may be true in a sense that "Training a custom model allows us to tailor it to our specific needs and requirements", the reality is they'd be getting better results just using OpenAI right now. Probably true for almost every other use case. That said, I am patiently waiting and champing at the bit for the day this isn't true anymore. Cool to see the groundwork bein…

Not everyone wants to depend on and trust a cloud service, and not everyone needs GPT-4 quality. If there's a viable way to tune and run models locally they could still be useful if you don't need it to play chess and imitate a Python interpreter at the same time.

Is it possible to add to an LLM without re-training it, my understanding was no.

Re: How to train your own large language models

#53

Post by jackmott42 »

koboll wrote: post #7

Ghostwriter is notably worse than GPT-4, so while it may be true in a sense that "Training a custom model allows us to tailor it to our specific needs and requirements", the reality is they'd be getting better results just using OpenAI right now. Probably true for almost every other use case. That said, I am patiently waiting and champing at the bit for the day this isn't true anymore. Cool to see the groundwork bein…

That could all change in a few months. We saw locally runnable, open source image generation catch up quick.

Re: How to train your own large language models

#54

Post by ssivark »

PlantsRpeople2 wrote: post #52
alpaca128 wrote: post #50

Earlier quoted context omitted.

Not everyone wants to depend on and trust a cloud service, and not everyone needs GPT-4 quality. If there's a viable way to tune and run models locally they could still be useful if you don't need it to play chess and imitate a Python interpreter at the same time.

Is it possible to add to an LLM without re-training it, my understanding was no.

The original “pre-training” Is what’s expensive. The “fine-tuning” (also training that it modifies network weights) for instruction following or other tasks costs the thousand dollar range.

Re: How to train your own large language models

#55

Post by birdplant »

minhazm wrote: post #39
birdplant wrote: post #18

Earlier quoted context omitted.

That’s a very generous interpretation of what happened because it wasn’t a “mistake” when he threatened the intern, it was something he purposefully and intentionally did, and doubled down on, even after having significant time to reconsider. Only when there was widespread public criticism of his actions did he backpedal. I’m curious what he’s said or done to make you a fan?

Seems like you're just arguing about the definition of the word "mistake". Intent has nothing to do with it. From Google (Oxford Dictionary): > an action or judgment that is misguided or wrong. So to admit that you made a mistake just means you were "misguided or wrong" which he definitely made clear he was. You're claiming that there was significant time to reconsider but the reality is that this all went down in a…

No, whole thing didn’t go down over a few hours. It was weeks. Not sure why you’d lie about this to protect someone you don’t know.

Furthermore, not only did did he refuse to acknowledge his lies, he continued to lie, doubled down on the previous ones, and to this day still continues to make deceptive statements.

So yes, as his slimy behavior has continued it is relevant to bring this up every single time replit is mentioned here.

Re: How to train your own large language models

#56

Post by birdplant »

koboll wrote: post #7

Ghostwriter is notably worse than GPT-4, so while it may be true in a sense that "Training a custom model allows us to tailor it to our specific needs and requirements", the reality is they'd be getting better results just using OpenAI right now. Probably true for almost every other use case. That said, I am patiently waiting and champing at the bit for the day this isn't true anymore. Cool to see the groundwork bein…

They’re competing directly with Microsoft (and getting crushed) because GitHub is their biggest competitor, so it makes sense that they wouldn’t want to use OpenAI products.

Agree that Ghostwriter is subpar though.

Re: How to train your own large language models

#57

Post by berkle4455 »

ZephyrBlu wrote: post #47
berkle4455 wrote: post #34

Earlier quoted context omitted.

Thanks for linking this. This is actually a superior offering to replit. They recently removed the ability to access a simple repl without logging in. Now you a) have to login and b) have to deal with this obtuse IDE-in-a-browser project creation shit. It's so many extra steps before I can run code. I just want a URL in which I can run some code. https://riju.codes/ is literally that. Thanks!

I ran into the same thing and finally made a Replit account. I'm just gonna use Riju from now on though. Using Replit with an account is way more janky than it was without needing to login.

> Using Replit with an account is way more janky than it was without needing to login.

It’s such a massive miss by their product teams. I don’t need this half-baked IDE. I want an interface that lets me run code as quickly as possible without any intermediate steps.

Re: How to train your own large language models

#58

Post by boredemployee »

thejosh wrote: post #36
boredemployee wrote: post #13

Earlier quoted context omitted.

That's weird. I would never do anything even remotely similar to what my (ex) employer does. CEO sounds like a douchebag tho.

I've seen terms/clauses here in AU for full time employment, depending on the industry/niche, where you can't jump to the same industry within X months.

That's what happens when you have a society worried about money and not interested in true human development.

Re: How to train your own large language models

#60

Post by dskhudia »

moltar wrote: post #46
dkhudia wrote: post #38

Earlier quoted context omitted.

Disclaimer: I work for MosaicML (MosaicML is the creator of the training platform used by Replit). Training these models from scratch on your domain specific data is not as expensive as one might think. We have provided some cost estimates in our blogs. https://www.mosaicml.com/blog/mosaicbert https://www.mosaicml.com/blog/training-stable-diffusion-from... https://www.mosaicml.com/blog/gpt-3-quality-for-500k

Do you have any examples on how to train a model that can write code but in a specific domain? Eg I only want to train it on a specific set of code. Eg let’s say functional React components in TypeScript.

We recently released 1B parameter model trained on a mix of data.[1] If you got your domain-specific data, our platform can cover the rest.

[1]: https://twitter.com/jefrankle/status/1649060478910357504?s=4...

Post reply on HN