Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YaLM-100B: Pretrained language model with 100B parameters

github.com

Post reply on HN
Print view View original

341–350 of 666 posts

Re: YaLM-100B: Pretrained language model with 100B parameters

#341

Post by waffleiron »

riku_iki wrote: post #335
waffleiron wrote: post #328

Earlier quoted context omitted.

You asked for proof of the following: > Google and Facebook feed their data to NSA. We know that at least some companies were ordered to handover all data, continuously [1]. edit: I think we have enough evidence that I would assume that it's valid for the other companies on the slides, and if it's not true you'll have to provide some proof of that. edit 2: [2] > It searches that database and lets them listen to the c…

You brought two links on: - phone calls surveillance in Venezuella: no Google no FB mentioned - plain words of some reporter without any evidence provided, no Google no FB mentioned

Weird how there is limited hard evidence of a secret, illegal government program... It's a lot more than I've seen than evidence for the claims of Yandex proactively sharing data with the Russian government.

Also where do you see Venezuela?

Re: YaLM-100B: Pretrained language model with 100B parameters

#342

Post by f311a »

londons_explore wrote: post #194
edf13 wrote: post #41

Wonder what the split is between Russian and English in the model?

Open the vocab file (from the script in the download directory) and you can get a pretty good idea. Looks to be approximately 50/50 from my random scrolling through the list.

That's because English and Russian have pretty similar vocabulary size. Vocabulary does not reflect the size of the data.

Re: YaLM-100B: Pretrained language model with 100B parameters

#343

Post by SXX »

tomp wrote: post #309
SXX wrote: post #167

First of all regardless for political situation this is great step in making ML research actually open. So huge thanks for those developers who pushed to make it public. Still... Yandex is in fact share responsibility for Russian government actions. While it impossible to fight censorship they could certainly shut down their News service completely. Yandex could also certainly move more of their company and staff out…

How much responsibility does Google share for US wrecking Afghanistan, Iraq, Libya and Syria?

Google is US company which pays taxes in US. The company just like everyone in US obviously do share responsibility for what US government does. Fortunately Google and it's leadership actually does have political positions even if you dont like it.

In any case as unfortunate owner of Russian passport with friends and collegues in Ukraine I am more affected by Putins war than by anything US does.

So I want Yandex to be seen as part of Kremlin propoganda machine and threated accordingly. This company grew monopoly in many markets in Russia and they directly benefited from Putin regime. Since Ilya Segalovich died company started to be "out of politics" and this complete lack of any political activity lead country to these terrible events.

Re: YaLM-100B: Pretrained language model with 100B parameters

#344

Post by sereja »

whywhywhywhy wrote: post #161
pembrook wrote: post #138

Side note: Yandex search is awesome, and I really hope they stay alive forever. It's the only functional image search nowadays, after our Google overlords neutered their own product out of fear over lawyers/regulation and a disdain for power users. You can't even search for images "before:date" in Google anymore.

Yandex Image Search is today is what Google Image Search should have been. End of the day I’ll use what actually gets the job done. Same goes for OpenAI and Google AI. If you don’t actually ever release and let others use your stuff and end paralyzed in fear at what your models may do then someone else is gonna release the same tech, and at this rate it seems like that’ll be Chinese or Russian companies who don’t sha…

IMO the main reason these companies don't release their models is not ethical concerns but money:

- NVIDIA sells GPUs and interconnect needed for training large models. Releasing a pretrained LM would hurt sales, while only publishing a teaser paper boosts them.

- Google, Microsoft, and Amazon offer ML-as-a-service and TPU/GPU hardware as a part of their cloud computing platforms. Russian and Chinese companies also have their clouds, but they have low global market share and aren't cost-efficient, so nobody would use them to train large LMs anyway.

- OpenAI are selling their models as an API with a huge markup over inference costs; they are also largely sponsored by the aforementioned companies, further aligning their interests with them.

Companies that release large models are simply those who have nothing to lose by doing so. Unfortunately, you need a lot of idle hardware to train them, and companies that have it tend to also launch a public cloud with it, so there is a perpetual conflict of interests here.

Re: YaLM-100B: Pretrained language model with 100B parameters

#345

Post by SXX »

vbezhenar wrote: post #339
SXX wrote: post #167

First of all regardless for political situation this is great step in making ML research actually open. So huge thanks for those developers who pushed to make it public. Still... Yandex is in fact share responsibility for Russian government actions. While it impossible to fight censorship they could certainly shut down their News service completely. Yandex could also certainly move more of their company and staff out…

If that would make you happier, Yandex is selling its News service to Mail.ru.

This only happen now after the war began.

Re: YaLM-100B: Pretrained language model with 100B parameters

#346

Post by uniqueuid »

taink wrote: post #215
uniqueuid wrote: post #197

Earlier quoted context omitted.

The $10k price is for an A100 with 40GB ram, so you need 8 of those. If you can get your hands on the 80GB variant, 4 are enough. Also, if you want to have a machine with eight of these cards, it will need to be a pretty high-spec rack-mounted or large tower. To feed these GPUs, you will want to have a decent amount of PCIe-4 lanes, meaning EPYC are the logical choice. So that's $20k for an AMD EPYC server with at le…

Do you happen to know the cost of the 80GB variant?

The PNY variant is pretty much the only one you can try to buy as an individual part, and those go for ~$15k. If you can get them.

Note that A100 like other datacenter GPUs are passively cooled. You need a strong airflow and duct in any case that would house them.

Re: YaLM-100B: Pretrained language model with 100B parameters

#347

Post by [deleted] »

lotusmars wrote: post #208
ushakov wrote: post #198

Earlier quoted context omitted.

can you name any Russian company that doesn't? obeying to Kremlin is just an aspect of running business in Russia the only option would be not to operate in Russia at all. Yandex can't do this, because their audience is primarily in Russia

Well they've made their choice and silenced our protest and opposition, and later spewed pro-war anti-Ukrainian propaganda using country's largest media (Yandex News). If you're profiteering from our suffering and choose Kremlin's needs over ours, don't be suprised then when we tell you to shove your AI models and your search.

[deleted]

Re: YaLM-100B: Pretrained language model with 100B parameters

#348

Post by waffleiron »

dekhn wrote: post #340
waffleiron wrote: post #306

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/PRISM#The_slides

You misunderstand. The NSA went out of their way to tap Google's lines outside of the US, which made the leadership at Google furious . It accelerated the work to encrypt international fiber (I think many people were really bothered by the tcpdump of a bigtable RPC containing a user ID). I was at a conference shortly after an saw a SVP rip an NSA rep to pieces. If Google is doing anything that is required of them leg…

That's what Google claims, however the leaked slides claimed "direct access".

edit: Does it really matter if they setup an FTP server instead of direct access, when we know a request can literally ask for "all" data (see Verizon).

> When required to comply with these requests, we deliver that information to the US government — generally through secure FTP transfers and in person," Google spokesman Chris Gaither told Wired, among other news outlets. [1]

[1] https://www.theatlantic.com/technology/archive/2013/06/googl...

Re: YaLM-100B: Pretrained language model with 100B parameters

#349

Post by MrBuddyCasino »

Havoc wrote: post #264
MrBuddyCasino wrote: post #53

Earlier quoted context omitted.

Wondering if Apple Silicon will bring arge amounts of unified main memory with high bandwidth to the masses? The Mac Studio maxes out at 128GB currently for around $5K, so 256GB isn't that far out and might work with the ~200GB Yandex says is required.

Perhaps on quantity. Substantially slower though around ~3x from what I can tell…substantial roadblock if you’re training models that take weeks.

I meant for inference, not training. People just want to run the magic genies locally and post funny AI content.
Post reply on HN