Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

cerebras.net

Post reply on HN
Print view View original

91–100 of 231 posts

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#91

Post by MacsHeadroom »

binarymax wrote: post #46
wsgeorge wrote: post #44

Earlier quoted context omitted.

I guess it's something. It still goes to show how far open models are behind the proprietary SOTA.

Indeed but this is zero-shot performance. Fine-tuning for a task should get you pretty good results. I'm interested in seeing the results of an Alpaca method against this Cerebras 13B model.

>I'm interested in seeing the results of an Alpaca method

You're talking apples to oranges. The "Alpaca method" is a dataset generation method. Nothing about Alpaca's training method is novel, interesting, or efficient. Alpaca used the same standard training method everyone else uses, A100 clusters.

If you mean LoRA/PEFT training which people used to replicate Alpaca then that is also apples to oranges because LoRA/PEFT is a finetuning method not a pre-training method.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#92

Post by Garcia98 »

I've been following open source LLMs for a while and at first glance this doesn't seem too powerful compared to other open models, Flan-Alpaca[0] is licensed under Apache 2.0, and it seems to perform much better. Although I'm not sure about the legalities about that licensing, since it's basically Flan-T5 fine-tuned using the Alpaca dataset (which is under a Non-Commercial license).

Nonetheless, it's exciting to see all these open models popping up, and I hope that a LLM equivalent to Stable Diffusion comes sooner than later.

[0]: https://github.com/declare-lab/flan-alpaca

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#93

Post by [deleted] »

2bitencryption wrote: post #26

This type of article (or press release, or whatever you want to call it) is exactly what makes the future so interesting. The cat is out of the bag, the genie is out of the bottle, the confetti has left the cannon[0]. It's tempting to see a world dominated by Google Bard, ChatGPT, Bing Search, etc. And no doubt, they will be huge players, with services that are far more powerful than anything that can be run on the e…

[deleted]

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#94

Post by espadrine »

simonw wrote: post #31

Does the chinchilla recipe still hold today? I got the impression that the LLaMA paper proposed a different result where throwing far more tokens at the problem had a very meaningful impact, or did I misunderstand that?

I don’t think it holds for two reasons.

First, it only holds for a given architecture and implementation. Obviously, a different architecture will have a different training slope. This is clear when comparing LSTM with Transformers, but is also true between transformers that use prenorm/SwiGLU/rotary-positional, and those that follow Vaswani 2017.

In terms of implementation, some algorithms yield the same result with fewer operations (IO, like FlashAttention and other custom CUDA kernels, and parallelism, like PaLM, which both came after Chinchilla), which unambiguously affect the Tflops side of the Chinchilla equation. Also, faster algorithms and better parallelization will yield a given loss sooner, while less power-hunger setups will do that cheaper.

Second, even in the original Chinchilla paper in figure 2, some lines are stopped early before reaching Pareto (likely because it ran out of tokens, but LLaMA makes it seem that >1 epoch training is fine).

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#95

Post by visarga »

simon83 wrote: post #72
2bitencryption wrote: post #26

This type of article (or press release, or whatever you want to call it) is exactly what makes the future so interesting. The cat is out of the bag, the genie is out of the bottle, the confetti has left the cannon[0]. It's tempting to see a world dominated by Google Bard, ChatGPT, Bing Search, etc. And no doubt, they will be huge players, with services that are far more powerful than anything that can be run on the e…

Google: "confetti has left the cannon" > No results found for "confetti has left the cannon". I'm amazed that a "stochastic parrot" can come up with such a beautiful idiom.

That name aged like milk. First of all, what you said. And second - a parrot can make more parrots without human help, language models can't make GPU chips. Insulting for both LLMs and parrots.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#96

Post by MacsHeadroom »

meghan_rain wrote: post #83
vintermann wrote: post #52

Earlier quoted context omitted.

NVidia has certainly pushing the envelope on image generation. StyleGAN3 was really cool when it came. But it is an issue that their chips are hardly optimized for LLMs.

How can a GPU be optimized for StyleGAN but not LLMs? Serious question.

RAM. GPT-3 is over 600GB, ie just the max RAM of 8xA100s, because that's all the hardware can fit.

StableDiffusion plus a whole chain of imagenets can make any visual imagery imaginable in 2GB of RAM. Meanwhile 2GB of RAM barely runs a basic tiny text completion NN that can't do anything intelligent. Text requires a lot more parameters (and more memory/RAM) than images.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#97

Post by [deleted] »

MacsHeadroom wrote: post #91
binarymax wrote: post #46

Earlier quoted context omitted.

Indeed but this is zero-shot performance. Fine-tuning for a task should get you pretty good results. I'm interested in seeing the results of an Alpaca method against this Cerebras 13B model.

>I'm interested in seeing the results of an Alpaca method You're talking apples to oranges. The "Alpaca method" is a dataset generation method. Nothing about Alpaca's training method is novel, interesting, or efficient. Alpaca used the same standard training method everyone else uses, A100 clusters. If you mean LoRA/PEFT training which people used to replicate Alpaca then that is also apples to oranges because LoRA/P…

[deleted]

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#98

Post by t_minus_2 »

2bitencryption wrote: post #26

This type of article (or press release, or whatever you want to call it) is exactly what makes the future so interesting. The cat is out of the bag, the genie is out of the bottle, the confetti has left the cannon[0]. It's tempting to see a world dominated by Google Bard, ChatGPT, Bing Search, etc. And no doubt, they will be huge players, with services that are far more powerful than anything that can be run on the e…

The cat is out of the bag,The genie is out of the bottle,The confetti has left the cannon,The ship has sailed,The horse has bolted,The toothpaste is out of the tube,The beans have been spilled,The train has left the station,The die is cast,The bell has been run.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#99

Post by groestl »

visarga wrote: post #95
simon83 wrote: post #72

Earlier quoted context omitted.

Google: "confetti has left the cannon" > No results found for "confetti has left the cannon". I'm amazed that a "stochastic parrot" can come up with such a beautiful idiom.

That name aged like milk. First of all, what you said. And second - a parrot can make more parrots without human help, language models can't make GPU chips. Insulting for both LLMs and parrots.

> language models can't make GPU chips

Not yet. I'm sure somebody is taking a hard look already.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#100

Post by alchemist1e9 »

bubblethink wrote: post #61
alchemist1e9 wrote: post #37

Earlier quoted context omitted.

It’s unbelievable stuff. Does anyone know how much a single box costs? They are selling them it looks like.

There is cloud pricing on the website. https://www.cerebras.net/product-cloud/

This is actually really important from my perspective. It looks like an end user can work backwards from available inference hardware, or interference budget, required speed, then figure out a viable model size. Bring their own data and then fine tune or train from scratch.

This is getting so real so fast.

Post reply on HN