Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YaLM-100B: Pretrained language model with 100B parameters

github.com

Post reply on HN
Print view View original

491–500 of 666 posts

Re: YaLM-100B: Pretrained language model with 100B parameters

#491

Post by BonoboIO »

MivLives wrote: post #482
lumost wrote: post #456

To add a voice of skepticism. The recent rush to open source these models may be indicative that the tens of millions that’s spent training these things has relatively poor roi. There may be a hope that someone else figures out how to make these commercially useful.

We're using these at where I work (large retail site) to help make filler text on generated articles. Think the summary blurb no one reads at the top. As for why we're writing these articles (we have a paid team that writes them too), the answer is SEO. This is probably the only thing I've seen done with a text model in production usage. I'm not 100% sure what model they're using.

Content made for machines. Probably a billion dollar industry.

Re: YaLM-100B: Pretrained language model with 100B parameters

#492

Post by HeavyStorm »

lumost wrote: post #456

To add a voice of skepticism. The recent rush to open source these models may be indicative that the tens of millions that’s spent training these things has relatively poor roi. There may be a hope that someone else figures out how to make these commercially useful.

Maybe training it is not that expensive?

I know from practice that it takes a really really long time to train even a small nn (thousands of params) , so you'll need a lot more hardware to train one with billions... But, it's expensive to buy the hardware, not necessarily to use it. If you, for some reason, have a few hundred GPU lying around, it might be "cheap" to do the necessary training.

Now, that's not your point - cost != price. But, still...

Re: YaLM-100B: Pretrained language model with 100B parameters

#493

Post by tobr »

MivLives wrote: post #482
lumost wrote: post #456

To add a voice of skepticism. The recent rush to open source these models may be indicative that the tens of millions that’s spent training these things has relatively poor roi. There may be a hope that someone else figures out how to make these commercially useful.

We're using these at where I work (large retail site) to help make filler text on generated articles. Think the summary blurb no one reads at the top. As for why we're writing these articles (we have a paid team that writes them too), the answer is SEO. This is probably the only thing I've seen done with a text model in production usage. I'm not 100% sure what model they're using.

Sorry but every part of that sounds so terrible.

Re: YaLM-100B: Pretrained language model with 100B parameters

#494

Post by dvirsky »

r3trohack3r wrote: post #334
dav_Oz wrote: post #150

Earlier quoted context omitted.

The bots/machine vs human reminds me of that famous experiment from the 30s in which Winthrop Kellogg[0], a comparative psychologist, and his wife decided to raise their human baby (Donald) simultaneously with a chimpanzee baby (Gua) in an effort to "humanize the ape". It was set out to last 5 years but was relatively quickly abrupted after only 9 months. The explicit reason wasn't stated only that it successfully pr…

A tangentially related thought: Actors attempt to imitate humans. “Good acting” is convincing; the audience believes the actor is giving a reasonable response to the portrayed situation. But the audience is also trying to imitate the actors to some degree. Like you point out, humans imitate. For some subset of the population, I’d imagine the majority of social situations they are exposed to, and the responses to situ…

Someone wrote once about how Wall Street people started behaving like the slick image projected of them in movies in the 80s, namely of Michael Douglas; before that they were more like the "boring accountant" type.

Re: YaLM-100B: Pretrained language model with 100B parameters

#495

Post by justinlloyd »

uniqueuid wrote: post #197
taink wrote: post #182

Earlier quoted context omitted.

I believe you're confusing the amount of A100 graphics cards used to train the model (the cluster was actually made up of 800 A100s), and the amount you need to run the model : > The model [...] is supposed to run on multiple GPUs with tensor parallelism. > It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, [but should work] with ≈200GB of GPU memory. I don't know what the price of a V100 is, but given $10k a piece…

The $10k price is for an A100 with 40GB ram, so you need 8 of those. If you can get your hands on the 80GB variant, 4 are enough. Also, if you want to have a machine with eight of these cards, it will need to be a pretty high-spec rack-mounted or large tower. To feed these GPUs, you will want to have a decent amount of PCIe-4 lanes, meaning EPYC are the logical choice. So that's $20k for an AMD EPYC server with at le…

You don't need a "decent amount" of PCIe-4 lanes. You just need 16 of them. And they can be PCIe 3.0 and will work just fine. Deep learning compute boxes predominantly use a PCIe switch. e.g. the ASUS 8000 box, which handles eight cards just fine. You only need a metric tonne of PCIe bandwidth if you are constantly shuttling data in and out of the GPU, e.g. in a game or exceedinyl large training sets of computer vision data. A little latency of a few hundred milliseconds moving data to your GPU in a training session that will take hours if not days to complete is neither here nor then. I suspect this model, with a little tweaking, will run just fine on an eight way RTX A5000 setup, or a five-way A6000 completely unhindered. That puts the price around $20,000 to $30,000. If I put two more A5000s in my machine, I suspect I could figure out how to get the model to load.

It also sounds like they haven't optimized their model, or done any split on it, but if they did, I suspect they could load it up and have it infer slower on fewer GPUs, by using main memory.

Re: YaLM-100B: Pretrained language model with 100B parameters

#496

Post by sailingparrot »

qayxc wrote: post #467
sailingparrot wrote: post #262

Earlier quoted context omitted.

If you don't care about inference speed being in the 1-5sec range, then that should be doable with CPU offloading, with e.g. DeepSpeed.

200+ GiB of RAM still sounds like a pretty steep hardware requirement.

If you have an nvme deepspeed can offload there as a second tier once the RAM is full.

175 GB aggregate on both RAM and nvme is in the realm of home deep learning workstation.

As long as you aren’t too fussy about inference speed of course.

Re: YaLM-100B: Pretrained language model with 100B parameters

#497

Post by justinlloyd »

riku_iki wrote: post #299
uniqueuid wrote: post #197

Earlier quoted context omitted.

The $10k price is for an A100 with 40GB ram, so you need 8 of those. If you can get your hands on the 80GB variant, 4 are enough. Also, if you want to have a machine with eight of these cards, it will need to be a pretty high-spec rack-mounted or large tower. To feed these GPUs, you will want to have a decent amount of PCIe-4 lanes, meaning EPYC are the logical choice. So that's $20k for an AMD EPYC server with at le…

There is also $5k A6000 with 48GB

Which will work just fine with NVIDIA SWITCH and a decent GPU compute case from ASUS or IBM or even building your own out of an off-the-shelf PCIe switch and consumer motherboard.

Re: YaLM-100B: Pretrained language model with 100B parameters

#498

Post by justinlloyd »

qayxc wrote: post #467
sailingparrot wrote: post #262

Earlier quoted context omitted.

If you don't care about inference speed being in the 1-5sec range, then that should be doable with CPU offloading, with e.g. DeepSpeed.

200+ GiB of RAM still sounds like a pretty steep hardware requirement.

Oh yeah, that $750 for 256GB of DDR-4 is going to totally break the bank.

Re: YaLM-100B: Pretrained language model with 100B parameters

#499

Post by agalunar »

melenaboija wrote: post #473
thejosh wrote: post #65

Earlier quoted context omitted.

It's also a power issue. The 4090 sounds like you're going to need a much, MUCH higher PSU than you currently use.. or it'll suddenly turn off as it uses 2-3x the power. You'll need your own wiring to run your PC soon :-)

I think it is a stupid question, but does the power consumption needed by processors to infer compared to human brains demonstrate that there is something fundamentally wrong for the AI approach or is it more physics related? I am not a physicist or biologist or anything like that so my intuition is probably completely wrong but it seems to me that for more basic inference operations (lets say add two numbers) power…

Around room temperature, an ideal silicon transistor has a 60 mV/decade subthreshold swing, which (roughly speaking) means that a 10-fold increase in current requires at least a 60 mV increase in gate potential. There are some techniques (e.g. tunneling) that can allow you to get a bit below this, but it's a fairly fundamental limitation of transistors' efficiency.

[It's been quite a while since I studied this stuff, so I can't recall whether 60 mV/decade is a constant for silicon specifically or all semiconductors.]

Re: YaLM-100B: Pretrained language model with 100B parameters

#500

Post by justinlloyd »

perryizgr8 wrote: post #62
ketzu wrote: post #21

Seeing those gigantic models it makes me sad that even the 4090 is supposed to stay at 24GB of RAM max. I really would like to be able to run/experiment on larger models at home.

Nvidia deliberately keeps their consumer/gamer cards limited in memory. If you have a use for more RAM, they want you to buy their workstation offerings like RTX A6000 which has 48G DDR6 RAM or A100 which has 80G.

What NVIDIA predominantly does on their consumer cards is limit the RAM sharing, not the RAM itself. The inability for each GPU to share RAM is the limiting factor. It is why I have RTX A5000 GPUs and not RTX 3090 GPUs.
Post reply on HN