Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YaLM-100B: Pretrained language model with 100B parameters

github.com

Post reply on HN
Print view View original

101–110 of 666 posts

Re: YaLM-100B: Pretrained language model with 100B parameters

#101

Post by uniqueuid »

tehsauce wrote: post #97
ggktk wrote: post #91

Earlier quoted context omitted.

I'm predicting that the upcoming Mac Pro will be very popular among ML developers, thanks to unified memory. It should be able to fit the entire model in memory. Combine that with the fact that PyTorch recently added support for Apple silicon GPUs.

upcoming mac pro will have pretty poor ML performance when compared to even an old nvidia gpu sadly.

Although memory capacity may matter more than speed for inference. As long as you're not training or fine tuning, the mac pro / studio may be just fine.

apart from the fact that you can't use any of the many nvidia-specific things; if you're dependent on cuda, nvcuvid, AMP or other things that's a hard no.

Re: YaLM-100B: Pretrained language model with 100B parameters

#103

Post by egorfine »

f6v wrote: post #8
egorfine wrote: post #6

I have huge respect for developers at Yandex. It's kind of sad that achievements like these are tainted by the fact that they come from Russia (and I speak as a Ukrainian). I wonder if the permissive license is able to mitigate that.

The achievements aren’t in any way tainted by their nationality, citizenship, sex, sexual orientation, age, etc.

Some people see it differently.

Re: YaLM-100B: Pretrained language model with 100B parameters

#104

Post by uniqueuid »

espadrine wrote: post #50
alexb_ wrote: post #13

I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Have to wonder what the knock-on effects of that will be, especially if the models improve drastically. With so much of our social lives being moved online, if we have the easy ability to create fake lives of fake people one has to wonder what's real and what isn't. Maybe the dead internet theory will rea…

> I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Isn’t that already the case? Sure, it costs $60K, but that is accessible to a surprisingly large minority, considering the potency of this software.

Nitpick: This uses 8x A100 which are at least $10k a piece to my knowledge. Add in the computer and you're closer to $100k.

Re: YaLM-100B: Pretrained language model with 100B parameters

#106

Post by kamray23 »

niek_pas wrote: post #36
egorfine wrote: post #6

I have huge respect for developers at Yandex. It's kind of sad that achievements like these are tainted by the fact that they come from Russia (and I speak as a Ukrainian). I wonder if the permissive license is able to mitigate that.

Are American developers’ achievement tainted by the fact they come from the United States?

Often, yes. In many places we're even wary of using US-based services at all. The EU has been having a bit of a back and forth with the US and many companies because EU law prohibits foreign states gaining access to personal information whereas US law requires foreign personal information transfers to CC the NSA on request. There's some big legislative deadlocks where American companies simply cannot operate fully in countries other than the US because the US laws require wild violations of basic rights to privacy of anyone who isn't a US citizen.

Lots of things have to be cleared for backdoors, Intel and AMD are scary with their built-in ME and whatever AMD had, I can't exactly remember, proprietary hardware in general is very scary outside the US due to surveillance and possible backdoors, it's kind of weird. Same goes for China, though they don't surveil foreigners exactly as hard, at least here they don't. It's not exactly an ideal situation and I think there should be agreements done internationally on stuff like this to keep the US and China out of our devices or allowing them to kindly fuck off entirely.

Not exactly the same kind of taint though, your products aren't as much morally tainted as they are simply dangerous to use, like little telescreens you have to carry around.

Re: YaLM-100B: Pretrained language model with 100B parameters

#107

Post by uniqueuid »

londons_explore wrote: post #90

The download fails because the vocab file link returns HTTP 403... :-( https://yalm-100b.s3.yandex.net/vocab/voc_100b.sp EDIT: It seems fine if you download with a browser useragent not CURL... I guess I just got hit by some anti-bot thing they have accidentally have turned on.

Try opening the inspector in firefox, selecting the download request and using "copy as CURL". That gives you a working curl command.

Re: YaLM-100B: Pretrained language model with 100B parameters

#108

Post by thelamest »

natly wrote: post #68
alexb_ wrote: post #13

I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Have to wonder what the knock-on effects of that will be, especially if the models improve drastically. With so much of our social lives being moved online, if we have the easy ability to create fake lives of fake people one has to wonder what's real and what isn't. Maybe the dead internet theory will rea…

I know it's a sort of exaggerated paranoid thought. But like these things do all come down to scale and some areas of the world definitely could have the amount of compute available to make dall-e level quality full scale videos which we might be consuming right now. It really does make you start to wonder at what point we will rationally be able to have zero trust that not everything we watch online is fabricated.

Historically, hard-to-falsify documents are an anomaly, the norm was mostly socially conditional and enforced trust. Civilizations leaned and still lean on limited-trust technologies like personal connections, word of mouth, word on paper, signatures, seals, careful custody etc. I agree losing cheap trust can be a setback, just want to point out we’re adaptable.

Re: YaLM-100B: Pretrained language model with 100B parameters

#109

Post by lukestateson »

denysvitali wrote: post #102
lukestateson wrote: post #99

[flagged]

I don't want to defend them, but I'm genuinely curious: aren't they maybe doing it because the opposite will cause them huge legal issues?

They can protest, they can boycott, they can disagree, they can tell the truth.

But... They chose to obey.

It's the choice that matters.

Re: YaLM-100B: Pretrained language model with 100B parameters

#110

Post by vegai_ »

danuker wrote: post #9
egorfine wrote: post #6

I have huge respect for developers at Yandex. It's kind of sad that achievements like these are tainted by the fact that they come from Russia (and I speak as a Ukrainian). I wonder if the permissive license is able to mitigate that.

Coming from Russia doesn't mean you agree with government policy. If you saw people get arrested as soon as they start protesting, what would you do?

> If you saw people get arrested as soon as they start protesting, what would you do?

Probably move if that's anywhere near a possibility, and if not, cowardly stay as unnoticeable as possible. I know that at least in Finland Russian refugees are mostly welcome (although the border might be closed right now), even if they probably will face a lot of scrutiny from various authorities for obvious reasons. Most certainly it's nothing like the attention such people would face in Russia.

We fondly remember even the smallest acts of defiance that ordinary Germans acted out against their regime during 1933-1945. We all would like to be those people in times of crisis, but obviously most of us are not. They were probably ultimately pretty futile acts during that time, though, but put together with all the other actions that happened against the Nazis played a significant grander role. And we know that more than a few significant Jewish scientists and engineers fled from Nazi Germany and made significant contributions to the war effort. For instance, one guy called Einstein.

Post reply on HN