Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The genie escapes: Stanford copies the ChatGPT AI for less than $600

newatlas.com

Post reply on HN
Print view View original

121–130 of 173 posts

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#121

Post by sitic »

superkuh wrote: post #11
starik36 wrote: post #9

From the article: Pre-trained on a trillion "tokens"... Doesn't 7B indicates that it was trained on 7 billion tokens? Or am I misunderstanding the nomenclature?

The emerging consensus for larger LLM is you want to train them with at least 2-4x the tokens of the number of parameters (weights between neurons in the layers). A trillion (100x) surprises me.

The LLaMA paper contradicts this view: "[...] Although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens." https://arxiv.org/pdf/2302.13971.pdf

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#122

Post by f_devd »

simonw wrote: post #104
f_devd wrote: post #76

Earlier quoted context omitted.

LLama hasn't been fine-tuned with RLHF, so it requires additional prompting, check out the open-assistant[0] project for an open-source ChatGPT equivalent (WIP). [0]: https://github.com/LAION-AI/Open-Assistant

This is why Alpaca is a big deal: it shows what LLaMA can do after it's been fine-tuned to follow instructions like ChatGPT has.

Alpaca uses Self-Instruct[0] which is better than just the pre-training but I wouldn't expect it to be at the level of ChatGPT (RLHF) in terms of human-friendly prompting. OpenAssistant should make it close to ChatGPT (from GPT-3.5 version) if the LLaMA is as powerful as claimed.

[0]: https://arxiv.org/abs/2212.10560

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#123

Post by satvikpendem »

superkuh wrote: post #8

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…

You might need to fix your parameters. From the text-generation-gui guide:

> For a more creative chat, use: temp 0.72, rep pen 1.1, top_k 0, and top_p 0.73

> For a more precise chat, use temp 0.7, repetition_penalty 1.1764705882352942 (1/0.85), top_k 40, and top_p 0.1

https://old.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...

https://old.reddit.com/r/singularity/comments/11vsvro/in_cas...

https://twitter.com/theshawwn/status/1632569215348531201

---

That being said, I found the OpenAssistant model much better: https://huggingface.co/spaces/olivierdehaene/chat-llm-stream...

It's also completely OSS, Apache 2.0, unlike LLaMA and Alpaca which are non-commercial.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#124

Post by satvikpendem »

Alpaca is cool but it's also not technically allowed by OpenAI's TOS, and LLaMA is certainly not allowed to be used for non-commercial purposes. With that in mind, OpenAssistant is an Apache 2.0 licensed fully open source alternative that's pretty good (the model is OpenAssistant/oasst-sft-1-pythia-12b): https://huggingface.co/spaces/olivierdehaene/chat-llm-stream....

I've found OA to be better than Alpaca but I'll wait until the 65B 3-bit quantization efforts for Alpaca are underway to compare them.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#125

Post by satvikpendem »

f_devd wrote: post #76
doctoboggan wrote: post #50

I've used both the 7B and 13B instruction tuned llama weights (quantized using the llama.cpp scripts). Either I am doing something wrong, or these two models are no-where near the level of ChatGPT. Many times they return something totally irrelevant to my question, stop responding, use a different language, or otherwise return the wrong answer. ChatGPT does none of this. (other than the wrong answer due to hallucinat…

LLama hasn't been fine-tuned with RLHF, so it requires additional prompting, check out the open-assistant[0] project for an open-source ChatGPT equivalent (WIP). [0]: https://github.com/LAION-AI/Open-Assistant

Use it here: https://huggingface.co/spaces/olivierdehaene/chat-llm-stream...

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#126

Post by jonplackett »

pram wrote: post #58
crooked-v wrote: post #23

Earlier quoted context omitted.

For me the easiest comparison between models is to give it an absurd but entirely possible request, like "Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain". So far nothing else I've tried has done even as well as GPT 3.5 yet, let alone GPT 4.

How exactly do you get it to keep going? Every time I try a prompt like this in the playground it spits out a couple paragraphs and then refuses to generate anything further, even with tokens maxed out.

Giving it an expected word count works well.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#127

Post by leodriesch »

UncleOxidant wrote: post #97
superkuh wrote: post #8

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…

> the alpaca 7B _4-bit_ [and presumably also 4bit for the 13B, 30B and larger parameter sets] This is the wild card here, though, isn't it? OpenAI's chatGPT likely uses more than 4 bits for it's parameters. IIRC the original LLaMA params were 16bit floats and they were quantitized down to 4bit - considering that large amount of compression, they sill do pretty OK, but not as good as chatGPT. I wonder how the alpaca/L…

Not an expert on the matter so take this with a grain of salt, but I’d say the compression is also about VRAM/RAM, which seems to be the more limiting factor over inference speed.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#128

Post by stavros »

trevyn wrote: post #101
stavros wrote: post #88

Earlier quoted context omitted.

Hmm, you think that's what makes a difference? It might be, though I've gotten different (what I find worse) results on other questions as well.

This is a great (pre-GPT-4) post that goes into the system prompt and its effects: https://minimaxir.com/2023/03/new-chatgpt-overlord/ Are you using any system prompt in your API calls? Note that it does have an explicit domain separation from the user and assistant roles. I suspect it plays a greater role in GPT-4 as well, i.e. was trained on a wider variety of system prompts to guide the output of the model.

I wasn't, no. You're right, that might make a difference. Thanks for the article, it's very useful, I've been curious about the system prompt!

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#129

Post by jonplackett »

onlyrealcuzzo wrote: post #106
superkuh wrote: post #8

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…

It's interesting that when ChatGPT 3.5 came out - everyone said, this is it! It's ready for primetime. And now that there's a few competitors in the same league - 3.5 quality is suddenly garbage and only 4.0 is good enough. Was it good enough before or wasn't it?

I think it’s going to be like movie special effects.

When Jurassic park first came out, or even something like Star Trek next gen. It looked AMAZING. So so realistic. But then…. As time goes on new things showed us what realistic could be.

I think we actually got better at seeing.

Same thing here. The more time you spend with it the more you notice things that don’t quite work. And then the new thing solves those problems, but we’ll find more wrongness

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#130

Post by circuit10 »

satvikpendem wrote: post #123
superkuh wrote: post #8

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…

You might need to fix your parameters. From the text-generation-gui guide: > For a more creative chat, use: temp 0.72, rep pen 1.1, top_k 0, and top_p 0.73 > For a more precise chat, use temp 0.7, repetition_penalty 1.1764705882352942 (1/0.85), top_k 40, and top_p 0.1 https://old.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_... https://old.reddit.com/r/singularity/comments/11vsvro/in_cas... https://twitter.com/the…

I’m impressed by how ChatGPT-like it is but also it’s saying things like

“No, OpenAI does not have an API for dogs. They do, however, have an API for other animals, such as cats. To retrieve an image of a cat, you can use the OpenAI API for Dogs API and select the cat breed or type.”

Post reply on HN