Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hello Dolly: Democratizing the magic of ChatGPT with open models

databricks.com

Post reply on HN
Print view View original

11–20 of 194 posts

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#11

Post by rnosov »

Mizza wrote: post #8
rnosov wrote: post #5

Earlier quoted context omitted.

This has nothing to do with facebook. The foundational model here is GPT-J which is opensource and safe to use. Sadly, it is inferior to state-of-the-art models such as LLaMA.

But they're "using data from Alpaca". I don't know what that means, isn't Alpaca using data generated by ChatGPT, which isn't "clean" to use? Or data from Facebook, which isn't "clean" to use? I'm drowning.

They are instruction tuning it using the dataset released by stanford-alpaca team. The dataset itself is synthetic (created using GPT-3) and somewhat noisy and in my view can be easily recreated if OpenAI ever tries to go after it (which is very unlikely). Anyway, facebook has nothing to do with anything used by this project.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#12

Post by Mizza »

rnosov wrote: post #11
Mizza wrote: post #8

Earlier quoted context omitted.

But they're "using data from Alpaca". I don't know what that means, isn't Alpaca using data generated by ChatGPT, which isn't "clean" to use? Or data from Facebook, which isn't "clean" to use? I'm drowning.

They are instruction tuning it using the dataset released by stanford-alpaca team. The dataset itself is synthetic (created using GPT-3) and somewhat noisy and in my view can be easily recreated if OpenAI ever tries to go after it (which is very unlikely). Anyway, facebook has nothing to do with anything used by this project.

So, this is a "dirty" model, in that is was created by data which violated OpenAI ToS. Obviously, this kind of violation is basically fine if you're a massive corporation who the rules don't apply to, but it's a huge risk if you're a small fish.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#14

Post by sebzim4500 »

Mizza wrote: post #12
rnosov wrote: post #11

Earlier quoted context omitted.

They are instruction tuning it using the dataset released by stanford-alpaca team. The dataset itself is synthetic (created using GPT-3) and somewhat noisy and in my view can be easily recreated if OpenAI ever tries to go after it (which is very unlikely). Anyway, facebook has nothing to do with anything used by this project.

So, this is a "dirty" model, in that is was created by data which violated OpenAI ToS. Obviously, this kind of violation is basically fine if you're a massive corporation who the rules don't apply to, but it's a huge risk if you're a small fish.

That's between OpenAI and the people that recorded the data. No one else needs to care.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#15

Post by woeirua »

This is the real risk to OpenAI's business model. If it turns out that you can get most of the same outcome with drastically smaller and cheaper models, then OpenAI is going to have a hell of a time keeping customers around as it will just be a race to the bottom on price and bigger, more expensive models will lose just from a hardware cost standpoint.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#16

Post by rashkov »

woeirua wrote: post #15

This is the real risk to OpenAI's business model. If it turns out that you can get most of the same outcome with drastically smaller and cheaper models, then OpenAI is going to have a hell of a time keeping customers around as it will just be a race to the bottom on price and bigger, more expensive models will lose just from a hardware cost standpoint.

What about the high quality training data that OpenAI has encoded into ChatGPT? Do these other models come close to that?

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#17

Post by rnosov »

Mizza wrote: post #12
rnosov wrote: post #11

Earlier quoted context omitted.

They are instruction tuning it using the dataset released by stanford-alpaca team. The dataset itself is synthetic (created using GPT-3) and somewhat noisy and in my view can be easily recreated if OpenAI ever tries to go after it (which is very unlikely). Anyway, facebook has nothing to do with anything used by this project.

So, this is a "dirty" model, in that is was created by data which violated OpenAI ToS. Obviously, this kind of violation is basically fine if you're a massive corporation who the rules don't apply to, but it's a huge risk if you're a small fish.

ToS are not the law. It would be similar to your power company claiming copyright over the code written using "their" electricity. Not going to happen. I wouldn't be too concerned.

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#18

Post by mejutoco »

sillysaurusx wrote: post #9

Interesting. DALL-E, Dalai ( https://cocktailpeanut.github.io/dalai/ ), and now Dolly are all pronounced the same way. It feels like there should be an xkcd for this.

AFAIK DALL-E is pronounced as Dalí, as in Salvador Dalí.

https://en.wikipedia.org/wiki/Salvador_Dal%C3%AD

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#19

Post by chatmasta »

mejutoco wrote: post #18
sillysaurusx wrote: post #9

Interesting. DALL-E, Dalai ( https://cocktailpeanut.github.io/dalai/ ), and now Dolly are all pronounced the same way. It feels like there should be an xkcd for this.

AFAIK DALL-E is pronounced as Dalí, as in Salvador Dalí. https://en.wikipedia.org/wiki/Salvador_Dal%C3%AD

I figured it was a reference to the Dalai Lama (which doesn't invalidate your comment, since that's also pronounced like Dalí). LLM -> Llama -> Dalai Lama

Re: Hello Dolly: Democratizing the magic of ChatGPT with open models

#20

Post by stametseater »

leobg wrote: post #7
Mizza wrote: post #3

It's immediately become difficult to untangle the licensing here. Is this safe for production use - I have no idea if I can expect a DMCA from Mark if I step out of bounds with this or other post-Alpaca models, unless I'm missing something important. Meta really botched the Llama release.

Why? Dolly had nothing to do with Llama or its weights. Besides: How would anyone ever know which model generated the output you are serving? AFAIK there is no fingerprint in any model’s output. And even if there was, it would probably be destroyed by fine tuning “over it”.

> AFAIK there is no fingerprint in any model’s output.

It seems like there easily could be. What if some of the data they trained it on didn't exist anywhere else except in the training set, and was put there specifically for this purpose? For instance they could have taught it a few poems that don't exist anywhere else. If you can coax the LLM of unknown origin into reciting those poems back to you, you know where it came from.

Post reply on HN