Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

cerebras.net

Post reply on HN
Print view View original

51–60 of 231 posts

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#51

Post by ericd »

alchemist1e9 wrote: post #37
brucethemoose2 wrote: post #35

Earlier quoted context omitted.

Yeah, and that doesn't even do the nutty IO on these things justice. A 16x CS2 cluster like they describe is like a huge Nvidia cluster in terms of throughput, but more like a single Nvidia node structurally.

It’s unbelievable stuff. Does anyone know how much a single box costs? They are selling them it looks like.

On the order of several million USD for the second gen system. Last I heard, they’re still at lowish volumes, selling some to national labs and the like.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#52

Post by vintermann »

bogwog wrote: post #47
mometsi wrote: post #14

Summary: This is a company that makes AI accelerator ICs. They reimplemented Chinchilla and released the model weights under a permissive license.

In other words, they’re actually incentivized to help make LLMs as accessible as possible, rather than try to keep them locked up to hide them from competitors. Which makes me wonder if Nvidia is doing anything with LLMs too?

NVidia has certainly pushing the envelope on image generation. StyleGAN3 was really cool when it came.

But it is an issue that their chips are hardly optimized for LLMs.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#53

Post by [deleted] »

thewataccount wrote: post #43
johnchristopher wrote: post #8

OT: I don't know about their scaling strategy for LLM but their scaling strategy for displaying pictures is disappointing. (it's all blurry)

They're dynamically scaled and something must be broken. If you inspect source you can find the raw images, here's a few: https://www.cerebras.net/wp-content/uploads/2023/03/Downstre... https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-... https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-... EDIT: Looks like it scores better with less training - up until it matches GPT-J/Pythia/OPT and doesn't ap…

[deleted]

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#54

Post by sbierwagen »

alchemist1e9 wrote: post #37
brucethemoose2 wrote: post #35

Earlier quoted context omitted.

Yeah, and that doesn't even do the nutty IO on these things justice. A 16x CS2 cluster like they describe is like a huge Nvidia cluster in terms of throughput, but more like a single Nvidia node structurally.

It’s unbelievable stuff. Does anyone know how much a single box costs? They are selling them it looks like.

CS-1 costs "$2-3 million", CS-2 costs "several" million.

A single Nvidia H100 costs somewhere around $30,000 each, so a GPU server with every slot populated costs about $300,000.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#55

Post by gpm »

freeqaz wrote: post #39
gpm wrote: post #30

Earlier quoted context omitted.

Is there a benchmark comparing the two that I missed? Edit: The huggingface page has 0-shot benchmarks which you can compare against the llama paper https://huggingface.co/cerebras/Cerebras-GPT-13B https://arxiv.org/pdf/2302.13971.pdf

I'm on mobile and struggled to compare these two tables properly. Would you mind posting a summary of your findings? Here are some values but I don't know what they mean. LLama 60B on the left, Cerebras 13B on the right. PiQA: 82.8 / 76.6 WinoGrade: 77.0 / 64.6 ARC-e: 78.9 / 71.4

Really short summary: LLaMa is better, even smaller LLaMa models.

Table format: Benchmark, Cerebras 13B, LLama 7B, LLama 13B, LLama 60B

HellaSwag, 51.3, 76.1, 79.2, 84.2

Piqa, 76.6, 79.8, 80.1, 82.8

Wino-Grande, 64.6, 70.1, 73.0, 77.0

Arc-e, 71.4, 72.8, 74.8, 78.9

Arc-c, 36.7, 47.6, 52.7, 56.0

OpenBookQA, 28.6, 57.2, 56.4, 60.2

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#56

Post by wsgeorge »

simonw wrote: post #20

"Cerebras open sources seven GPT-3 models from 111 million to 13 billion parameters." I don't understand why they describe them as GPT-3 models here as opposed to calling them GPT models. Or even LLMs - but I guess that acronym isn't as widely recognized.

I think GPT-3 is used as a benchmark for performance, so saying a model is on par with GPT-3 should give you an idea of what you can get out of it.

IIRC most open source models to date - including the semi-open LLaMAs - have GPT-3-like performance. Nothing gets close to GPT-3.5 and beyond.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#57

Post by vintermann »

haldujai wrote: post #27
gamegoblin wrote: post #5

Earlier quoted context omitted.

True. There was a good blog post published about this a few weeks ago: https://finbarr.ca/llms-not-trained-enough/ Money quote for those who don't want to read the whole thing: ''' When people talk about training a Chinchilla-optimal model, this is what they mean: training a model that matches their estimates for optimality. They estimated the optimal model size for a given compute budget, and the optimal number of t…

While true I think this also misses that “for almost everyone else” you’re probably not (or at least should not) be trying to optimize zero-shot performance if you have an intended high inference use case so I don’t think Chinchilla would be all that relevant.

I have a suspicion that good zero-shot performance is a good starting point for fine-tuning. If you have more than one intended high inference use case, or can imagine a couple of new ones on the horizon, it might still be best to not target the first use case directly.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#58

Post by ftxbro »

eldenring wrote: post #3

> Trained using the Chinchilla formula, these models provide the highest accuracy for a given compute budget. I'm confused as to why 111 million parameter models are trained with the Chinchilla formula. Why not scale up the training data? If you're training smaller models, surely optimizing performance is better than optimizing total compute. Seems like a silly misunderstanding of the Chinchilla paper, but I'm sure I…

The point of those smaller models is for the "Cerebras Scaling Law for Compute-Optimal Training" which is the straight line plot in the image at the top of their webpage when you click the link.

They want you to think it's reasonable that because the line is so straight (on a flops log scale) for so long, it could be tempting to extrapolate the pile-loss consequences of continuing compute-optimal training for larger models beyond their largest 13B one, with the obvious caveat that the extrapolation can't continue linearly much further if for no other reason than the test loss isn't going to go below zero (it will flatten out sooner than that).

If you trained beyond compute-optimality on smaller models, it would mess up their straight line and make it look like we are sooner hitting diminishing returns on test loss.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#59

Post by johnchristopher »

thewataccount wrote: post #43
johnchristopher wrote: post #8

OT: I don't know about their scaling strategy for LLM but their scaling strategy for displaying pictures is disappointing. (it's all blurry)

They're dynamically scaled and something must be broken. If you inspect source you can find the raw images, here's a few: https://www.cerebras.net/wp-content/uploads/2023/03/Downstre... https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-... https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-... EDIT: Looks like it scores better with less training - up until it matches GPT-J/Pythia/OPT and doesn't ap…

Disabling ublock on Firefox fixes the issue for me.

edit: scratch that, it seems the AJAX endpoint returns 504 more often that not.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#60

Post by alchemist1e9 »

freeqaz wrote: post #48
alchemist1e9 wrote: post #37

Earlier quoted context omitted.

It’s unbelievable stuff. Does anyone know how much a single box costs? They are selling them it looks like.

If you have to ask, you can't afford it! Mostly teasing but my guess would be $500k+ since they'll likely price it so that it is the same $ as the equivalent NVIDIA cluster (or very close to it).

Actually if they are around $2M looks like my company can afford one. Given this is just getting started it looks promising as I’m sure future generations will be more affordable.
Post reply on HN