Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

cerebras.net

Post reply on HN
Print view View original

221–230 of 231 posts

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#221

Post by anon291 »

IshKebab wrote: post #200
anon291 wrote: post #180

Earlier quoted context omitted.

Compile times are not a whole lot different than any other large model build. It's a kernel based compilation pipeline and the kernels are simply tiled over a 'core' area in the weight streaming architecture.

I seriously doubt that. What's your source? I used to work for a competitor with a more flexible architecture and even our compile times were bad (significant fractions of a day in some cases). And we didn't have to do place and route! I just googled it and it's apparently bad enough that they had to implement incremental place and route.

I used to work at Cerebras. They don't do place and route anymore. That's the old pipeline mode. They've shifted to weight streaming

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#222

Post by Timon3 »

leereeves wrote: post #214
Timon3 wrote: post #203

Earlier quoted context omitted.

In a way, this comment perfectly encapsulates why the argument "machines will never replicate human behavior" is so ridiculous. Instead of engaging with the discussion and topic, you chose a position, and then tried to justify it without really thinking about why one example works and the other one doesn't. In doing so you're literally showing that for certain topics, machines are already more capable than some human…

I didn't say "machines will never replicate human behavior" so I don't think you're engaging with what I said.

Sure, you didn't say that. You merely said:

> And it's exactly the one thing LLMs are trained to do: reproduce patterns of words. They are (perhaps) already better than humans at that one specific skill (another win for AI research) but I don't think it's a sign of general intelligence.

while completely missing why the machine did a better job.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#223

Post by DoctorOetker »

pmoriarty wrote: post #205
roenxi wrote: post #198

Earlier quoted context omitted.

The issue I see here is you are doing a worse job at this than ChatGTP. Creating idioms is hard, that is why we left most of them to Shakespeare. - I regularly return cereal to its box. - "helium" and "balloon" have a more awkward rhythm than "confetti" and "cannon". It also loses the connotations of sudden, explosive and exciting change. - Snow & globe I'm not even sure what that means in practice. It has poor prosp…

> "helium" and "balloon" have a more awkward rhythm than "confetti" and "cannon". It also loses the connotations of sudden, explosive and exciting change. Not only that, but "the confetti has left the cannon" is an alliteration, which makes the phrase even more poetic.

it also attests to the ramifications, the magnificent difficulty of cleaning confetti, you will keep occasionally finding confetti around the house for months or years

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#224

Post by ioulaum »

I wonder if they've done some Alpaca style training on it... Granted, what made Alpaca useful was that it was finetuned with GPT-3's instruction following completions as examples.

And, at least officially, OpenAI's outputs can't be used to train other AI models.

Otherwise, if GPT-4 outputs were used to finetune these models, they may become much more interesting.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#225

Post by haldujai »

vintermann wrote: post #57
haldujai wrote: post #27

Earlier quoted context omitted.

While true I think this also misses that “for almost everyone else” you’re probably not (or at least should not) be trying to optimize zero-shot performance if you have an intended high inference use case so I don’t think Chinchilla would be all that relevant.

I have a suspicion that good zero-shot performance is a good starting point for fine-tuning. If you have more than one intended high inference use case, or can imagine a couple of new ones on the horizon, it might still be best to not target the first use case directly.

Well yeah that’s kind of intuitive, my point is that if you just optimize for zero-shot you end up with something like GPT4 when an enterprise could probably be using finetuned LLaMA-7B with similar performance.

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#226

Post by bjornsing »

ftxbro wrote: post #163
bjornsing wrote: post #152

Earlier quoted context omitted.

> the extrapolation can't continue linearly much further if for no other reason than the test loss isn't going to go below zero Isn’t the test loss logarithmic? If so it sure can go below zero.

According to https://pile.eleuther.ai/paper.pdf the test loss on the pile is the log of the perplexity, and the perplexity is 2^H where H is an entropy which is non-negative. So the perplexity is always at least one, so its log is always at least zero. So yes the test loss can be seen as a log, but no it's not allowed to go below zero. The intuition is that the test loss is the number of bits that the model would nee…

Good point!

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#227

Post by barking_biscuit »

pmoriarty wrote: post #207
barking_biscuit wrote: post #89

Earlier quoted context omitted.

Only a few years? Nobody can predict accurately in years anymore. Feels more like "only a few months" away.

I'm not so confident. The speed of recent advances may be due to picking low-hanging fruit. Soon there may not be much low-hanging fruit left.

True, though perhaps maybe "low hanging fruit is all you need?" :P

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#228

Post by Garcia98 »

capableweb wrote: post #219
Garcia98 wrote: post #136

Earlier quoted context omitted.

Google's Flan-T5, Flan-UL2 and derivatives, are so far the most promising open (including commercial use) models that I have tried, however they are very "general purpose" and don't perform well in specific tasks like code understanding or generation. You could fine-tune Flan-T5 with a dataset that suits your specific task and get much better results, as shown by Flan-Alpaca. Sadly, there's no open model yet that act…

Iterating on the question, what model/weights would be the most appropriate for the specific use case of code generation right now?

Sorry for the late reply, as I said Flan-UL2 (or Flan-T5 if you want lighter models) fine-tuned against a dataset like CodeAlpaca's[0] is probably the best solution if it's intended for commercial use (otherwise LLaMa should perform better).

[0]: https://github.com/sahil280114/codealpaca

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#229

Post by gverrilla »

Tommstein wrote: post #113
2bitencryption wrote: post #105

Earlier quoted context omitted.

For the curious, here was the conversation I had: ME: What's another saying similar to: "The cat is out of the bag" or "The genie is out of the bottle"? chatgpt: Another similar saying is "the toothpaste is out of the tube." Like the other two expressions, this phrase conveys the idea that something has been revealed or unleashed, and it cannot be undone or put back into its original state. ME: Can you invent a new p…

That is impressive, especially given that neither Google nor DuckDuckGo find any previous trace of that idiom.

did you search in different languages?

Re: Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models

#230

Post by cs-fan-101 »

ipsum2 wrote: post #133
brucethemoose2 wrote: post #35

Earlier quoted context omitted.

Yeah, and that doesn't even do the nutty IO on these things justice. A 16x CS2 cluster like they describe is like a huge Nvidia cluster in terms of throughput, but more like a single Nvidia node structurally.

Cerebras makes impressive hardware, but Nvidia still performs better in every regard. The most telling factor is that Cerebras claims they're too busy to run common benchmarking (e.g. MLPerf) to compare against Nvidia.

Simply focusing on the "better in every regard" part of the comment.

One example where Cerebras systems perform well is when a user is interested in training models that require long sequence lengths or high-resolution images.

One example is in this publication, https://www.biorxiv.org/content/10.1101/2022.10.10.511571v2, where researchers were able to build genome-scale language models that can learn the evolutionary landscape of SARS-CoV-2 genomes. In the paper mentions, researchers mention "We note that for the larger model sizes (2.5B and 25B), training on the 10,240 length SARS-CoV-2 data was infeasible on GPU clusters due to out-of-memory errors during attention computation."

Post reply on HN