No post body was provided.
YaLM-100B: Pretrained language model with 100B parameters
391–400 of 666 posts
Re: YaLM-100B: Pretrained language model with 100B parameters
#392Re: YaLM-100B: Pretrained language model with 100B parameters
#393What is the TLDR on this model? What exactly does it do? Its not clear from the source examples.
Re: YaLM-100B: Pretrained language model with 100B parameters
#394Side note: Yandex search is awesome, and I really hope they stay alive forever. It's the only functional image search nowadays, after our Google overlords neutered their own product out of fear over lawyers/regulation and a disdain for power users. You can't even search for images "before:date" in Google anymore.
Yandex Image Search is today is what Google Image Search should have been. End of the day I’ll use what actually gets the job done. Same goes for OpenAI and Google AI. If you don’t actually ever release and let others use your stuff and end paralyzed in fear at what your models may do then someone else is gonna release the same tech, and at this rate it seems like that’ll be Chinese or Russian companies who don’t sha…
This reminded me of a shitpost comparing Google and Yandex.
Re: YaLM-100B: Pretrained language model with 100B parameters
#395Re: YaLM-100B: Pretrained language model with 100B parameters
#396Earlier quoted context omitted.
no it's not. they straight up serve kremlin, promoting kremlin fake news and silencing russian opposition (not much to silence but still). they can have whatever functionality they like, I still won't use it in billion years.
can you name any Russian company that doesn't? obeying to Kremlin is just an aspect of running business in Russia the only option would be not to operate in Russia at all. Yandex can't do this, because their audience is primarily in Russia
Not really, it's very different for Yandex in particular. Along with several other companies like Vimpelcom, they started the "Safe Internet League", an organization which exploited the think of the children argument to build the censorship regime from scratch. They practically created the original censorship laws, or participated in the creation, when they were in the best position to resist the government (and had the incentive to do so). As an example, Telegram successfully resisted the censorship while having much less leverage, much later.
Of course Yandex likes to pose as the victim of censorship, but the truth is that they are the censors themselves. They've been steamrolled by a runaway process they helped to create.
Re: YaLM-100B: Pretrained language model with 100B parameters
#397Earlier quoted context omitted.
I assume if you force the file system to allocate inodes you are likely to have a less fragmented file than if you create a sparse file that gets inodes assigned over time when each part is used.
Which won't matter on SSDs
On all the benchmarks of SSDs I've seen they perform 1.5 to 4 times better on sequential reads than on random reads. That's a much better ratio than HDDs, but still enough to care about it.
You're also likely to get less write amplification if your swap file is continuous.
Of course with all the layers of indirection it's a numbers game, you don't know if your file system allocates adjacent inodes, and you don't know how your SSD will remap the blocks. But all else being equal, trying to make the file as sequential as possible seems preferable.
Re: YaLM-100B: Pretrained language model with 100B parameters
#398It's just crazy how much it costs to train such models. As I undestand 800 A100 cards would cost about 25.000.000 without considering the energy costs for 61 days of training.
https://coreweave.com/ offers some of the cheapest GPU compute out there
Re: YaLM-100B: Pretrained language model with 100B parameters
#399Earlier quoted context omitted.
Lambda labs will rent you an 8xA100 instance for 3 months for $21,900. That would put it at around $2m
Still a bit to expensive for my sideproject ; ) To be honest it seems only big corp can do that kind of stuff. By the way if try to do hyper parameter tuning or some exploration in the architecture it becomes guess 10x or 100x more expensive.
AWS has them in US-EAST1 for $9.83/hr spot with 96 CPU cores, 1152GB of ram, 8 A100s with 320 GB of RAM, 8TB of NVME, and 19 Gbps of EBS bandwidth to load your data quickly.
https://aws.amazon.com/ec2/instance-types/p4/
p4d.24xlarge
An alternative is the p3.16xlarge for 8 V100s with 256GB of GPU RAM but you might as well get the A100s since it's only $0.50/hr cheaper