Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The genie escapes: Stanford copies the ChatGPT AI for less than $600

newatlas.com

Post reply on HN
Print view View original

111–120 of 173 posts

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#111

Post by sebzim4500 »

braingenious wrote: post #79
disgruntledphd2 wrote: post #66

Earlier quoted context omitted.

Alpaca wasn't great. The 13b and 30b models are much better, but just for sentence completion. Personally, I think that the RLHF does make a big difference but maybe it's a bug in the quantization code as suggested up thread.

I’m also a bit confused by the quantization thing. Why exactly is everybody running the same program on the same file? Why not just include the quantized weights? It seems like if somebody figured out the “correct” way to quantize the 7b weights it would make way more sense to just torrent the output rather than distribute a fixed program.

There is no reason not to do that, except

i) Distributing large files through torrents is slightly annoying if you don't already happen to have a seedbox

ii) People are still messing around with quantization settings, they might think that they are a few days away from a much better version

iii) No one wants to be sued by Meta. I think the risk is pretty small but not zero.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#112

Post by sebzim4500 »

superkuh wrote: post #11
starik36 wrote: post #9

From the article: Pre-trained on a trillion "tokens"... Doesn't 7B indicates that it was trained on 7 billion tokens? Or am I misunderstanding the nomenclature?

The emerging consensus for larger LLM is you want to train them with at least 2-4x the tokens of the number of parameters (weights between neurons in the layers). A trillion (100x) surprises me.

They probably put most of the effort into the 65B model, the 7B model was just trained so they could get an idea of the scaling behaviour. It makes sense to use the same amount of training steps, then.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#113

Post by codetrotter »

superkuh wrote: post #8

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…

> represent a movie plot as emoji

This sounded like a really cool idea but I asked ChatGPT to do this for the plot of the movie The Shawashank Redemption and there is no way that I would ever have been able to guess that movie from the emojis it gave me. Perhaps GPT-4 does a better job at it.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#114

Post by coeneedell »

onlyrealcuzzo wrote: post #106
superkuh wrote: post #8

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…

It's interesting that when ChatGPT 3.5 came out - everyone said, this is it! It's ready for primetime. And now that there's a few competitors in the same league - 3.5 quality is suddenly garbage and only 4.0 is good enough. Was it good enough before or wasn't it?

The problem is that you’ve identified two distinct and non-overlapping sets of people as “everyone”. Everyone who was applauding 3.5 when it came out were industry hype people. Even the critical voices were industry hype people, paid to assume the AI is powerful and write about the possible negative consequences of that assumption.

Now we’ve all gotten familiar with 3.5, and we’ve come to understand its limitations, so the public knows it’s not a “godlike” AI.

Luckily there’s a fresh new model, not technically different from the earlier one but it cost more money to build. The hype group can start again, citing the publicly known limitations of 3.5. But in 6 months we’ll understand what’s wrong with it, and the public will be talking about the limitations, just in time for 4.5.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#115

Post by nerpderp82 »

oezi wrote: post #87
nerpderp82 wrote: post #7

Earlier quoted context omitted.

The world is becoming more cyberpunk everyday, people making back alley deals for data or models. These weights are shit man, they have been quanted w/o being retrained against the original. I already have this torrent, I want uncut originals. And no water marks this time, the last model wouldn't shutup about investing in tulips.

Shh, I got some unnerfed midjourney v8 for your nsfw needs, my friend...

I will only pay half unless you supply the training data as well.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#116

Post by coeneedell »

superkuh wrote: post #37
thomasahle wrote: post #29

Earlier quoted context omitted.

3 bits? Is that for all weights in the network?

As far as I know, yes. https://arxiv.org/abs/2210.17323 "Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline." This would be 175 billion 3 bit weights instead of 175 billion 16 (or 32!) bit weights. It massively reduces the size of the mod…

I wonder if reducing the bit depth of parameters like we have been acts as a normalization feature in these huge deep models.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#117

Post by [deleted] »

crooked-v wrote: post #23
superkuh wrote: post #8

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…

For me the easiest comparison between models is to give it an absurd but entirely possible request, like "Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain". So far nothing else I've tried has done even as well as GPT 3.5 yet, let alone GPT 4.

[deleted]

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#118

Post by mromanuk »

nico wrote: post #73
xwdv wrote: post #2

Given the high prices of OpenAI offerings it seems it’s better to pirate an AI model before resorting to paying for anything.

How much does it cost to privately fine-tune and run Llama? It’s USD 600 for fine-tuning. Maybe USD 4-5k for a computer that can run it. ChatGPT pro is $20/month. 5k would be 250 months (10+ years) of paid access. Not sure pirating it now adds up.

Most people doesn’t fine tune the models (llama or openAI). A MacBook M1 can run those model ($1000) and in many cases the user already have it. You also need a computer to access openAI, the comparison boils down to $20/m vs $0. At this point in time, LLM are a curiosity for most people

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#119

Post by dang »

Recent and related:

Stanford Alpaca web demo suspended “until further notice” - https://news.ycombinator.com/item?id=35200557 - March 2023 (77 comments)

Stanford Alpaca, and the acceleration of on-device LLM development - https://news.ycombinator.com/item?id=35141531 - March 2023 (66 comments)

Alpaca: An Instruct Tuned LLaMA 7B – Responses on par with txt-DaVinci-3 - https://news.ycombinator.com/item?id=35139450 - March 2023 (11 comments)

Alpaca: A strong open-source instruction-following model - https://news.ycombinator.com/item?id=35136624 - March 2023 (296 comments)

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#120

Post by throwaway1851 »

superkuh wrote: post #8

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…

Hm. I haven’t tried the local installs yet. However, when the Alpaca web demo was live, I did find it to be comparable (though not quite as capable) to davinci-003. It answered arbitrary factual questions about pop culture references, law, medicine, and programming. It generated rhymes and poems. (I didn’t try asking for the emoji thing, so can’t say anything about that.) It performed natural language tasks such as information extraction and summarization. And it did all of it coherently.
Post reply on HN