Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hello Dolly: Democratizing the magic of ChatGPT with open models

databricks.com

Post reply on HN
Print view View original

51–60 of 194 posts

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#51

Post by joshhart »

_9hey wrote: post #35

I’d like some clarification of terms - when they say it takes 3 hours to train, they’re not saying from scratch are they? There’s already a huge amount of training to get to that point, isn’t that correct? If so, then it’s pretty audacious to claim they’ve democratized an LLM because the original training likely cost an epic amount of money. Then who knows how much guidance their training has incorporated, and it cou…

The 3 hours is the instruction fine-tuning. The base foundational model is GPT-J which was already provided by Eleuther-AI and has been around for a couple of years.

Note: I work at Databricks and am familiar with this project but didn't work on it.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#52

Post by computerex »

feanaro wrote: post #47
typon wrote: post #45

Earlier quoted context omitted.

Key point is vaguely good. Scale is still important and that manifests in the difference between gpt3.5 and gpt4 based chatgpts. It's qualitatively and quantitatively so much better in pretty much every benchmark. There is no way around the bitter lesson.

Isn't it the case that we literally have no clue how GPT4 and GPT3.5 are different in terms of training, given OpenAI doesn't want to disclose anything at all?

We don' have the details, it is true. But empirically and based on their report gpt-4 is notably better than chatgpt.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#54

Post by UncleEntity »

typon wrote: post #45
htrp wrote: post #37

TLDR: Download GPT-J-6B from Eleuther Download Alpaca Fine Tuning Code + Alpaca Examples Train for 6 hours or so. Get vaguely good RLHF model

Key point is vaguely good. Scale is still important and that manifests in the difference between gpt3.5 and gpt4 based chatgpts. It's qualitatively and quantitatively so much better in pretty much every benchmark. There is no way around the bitter lesson.

Free is better than $$/token imho.

If you have a use case or a bunch of disposable income then go with the “bitter” one.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#55

Post by nico »

woeirua wrote: post #15

This is the real risk to OpenAI's business model. If it turns out that you can get most of the same outcome with drastically smaller and cheaper models, then OpenAI is going to have a hell of a time keeping customers around as it will just be a race to the bottom on price and bigger, more expensive models will lose just from a hardware cost standpoint.

That’s why they are moving so fast and trying to get as much press/media attention as possible.

They want to stay top of mind.

Think about CocaCola, anyone can make a drink just as good. But it’s almost impossible to build their brand and distribution from scratch.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#56

Post by typon »

feanaro wrote: post #47
typon wrote: post #45

Earlier quoted context omitted.

Key point is vaguely good. Scale is still important and that manifests in the difference between gpt3.5 and gpt4 based chatgpts. It's qualitatively and quantitatively so much better in pretty much every benchmark. There is no way around the bitter lesson.

Isn't it the case that we literally have no clue how GPT4 and GPT3.5 are different in terms of training, given OpenAI doesn't want to disclose anything at all?

It's not true we know nothing. We know a little bit by using the two models from their API. Given the time per inference and the limit on messages per day for GPT4, I'm willing to bet it's doing around 10x more compute than GPT3.5. If that's because it has 10x more weights, I don't know. But it wouldn't be a terrible guess.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#57

Post by feanaro »

computerex wrote: post #52
feanaro wrote: post #47

Earlier quoted context omitted.

Isn't it the case that we literally have no clue how GPT4 and GPT3.5 are different in terms of training, given OpenAI doesn't want to disclose anything at all?

We don' have the details, it is true. But empirically and based on their report gpt-4 is notably better than chatgpt.

Better, yes, and for that we have evidence. But is the improvement stemming simply from even more data? That's what I'm questioning.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#58

Post by rockzom »

renewiltord wrote: post #29
sillysaurusx wrote: post #9

Interesting. DALL-E, Dalai ( https://cocktailpeanut.github.io/dalai/ ), and now Dolly are all pronounced the same way. It feels like there should be an xkcd for this.

Wow, just discovered that the American pronunciation for Dalai Lama is Da-lee. Well, that's a discovery. This is like when Khan Academy came out and there was a guy online saying it's a terrible brand because it sounds like Con Academy which it doesn't in my dialect. Took a while to get it.

How do you say Khan?

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#59

Post by dopidopHN »

thewataccount wrote: post #27

I might be having a moment - but I can't find any links to a git repo, huggingface, or anything about the models/weights/checkpoints directly from the article. I just see a zip download that AFAIK also doesn't contain the weights/checkpoints. I find this a bit odd, the contents of the zip (from the gdrive preview) look like they should be in a git repo, and I assume they download the model from somewhere? GDrive usua…

Thanks I missed that email while skimming

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#60

Post by ToValueFunfetti »

sillysaurusx wrote: post #22
chatmasta wrote: post #19

Earlier quoted context omitted.

I figured it was a reference to the Dalai Lama (which doesn't invalidate your comment, since that's also pronounced like Dalí). LLM -> Llama -> Dalai Lama

Hmm. Is Salvador Dalí pronounced differently than Dolly or Dalai? The wikipedia page has "dah-lee" as the phonetic, and https://www.google.com/search?q=pronounce+salvador+dali sounds the same as https://www.google.com/search?q=pronounce+dalai+lama . So it seems like all three are identical.

The emphasis in Dalí is on the second syllable, which is at least different from Dolly. I've always pronounced Dalai Lama the same as I would Dolly Lama, but Cambridge dictionary is saying it should be Da-lay in both US and UK pronunciations.

Tangentially, it seems like most of the results for both searches were autogenerated with TTS programs. I wonder if our pronunciations will shift towards TTS mistakes over time. Probably not, these videos only have a few thousand views, but neat if true.

Post reply on HN