Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YaLM-100B: Pretrained language model with 100B parameters

github.com

Post reply on HN
Print view View original

311–320 of 666 posts

Re: YaLM-100B: Pretrained language model with 100B parameters

#311

Post by sanxiyn »

SXX wrote: post #273
sanxiyn wrote: post #261

Earlier quoted context omitted.

They can (and do) revoke API access from bad guys. They can't do that to downloaded models. Look, I don't like what OpenAI does, but "API access, but no model download" makes sense if you are worried about misuses.

Every company out there says it will "revoke API access for misuse", but do they have transparency reports? Who do they even consider bad guys and what do they consider as misuse? I would be totally on their side if their reasoning was that they dont publish models to compete with FAANG more efficiently and get more income for their research, but this moral reasoning just sounds completely fake because bad actors do…

OpenAI published "Lessons Learned on Language Model Safety and Misuse" in March. https://openai.com/blog/language-model-safety-and-misuse/ It also promised "forthcoming publication".

Examples of "real cases of misuse encountered in the wild" include "spam promotions for dubious medical products and roleplaying of racist fantasies".

Yes, some bad actors can train their own models, but OpenAI can't do much about that either way. It is doubtful whether spam promoters of dubious medical products can, at least for a while.

Re: YaLM-100B: Pretrained language model with 100B parameters

#312

Post by josu »

ketzu wrote: post #21

Seeing those gigantic models it makes me sad that even the 4090 is supposed to stay at 24GB of RAM max. I really would like to be able to run/experiment on larger models at home.

For the people that didn't click on the link:

>but is able to work with different configurations with ≈200GB of GPU memory in total which divide weight dimensions correctly (e.g. 16, 64, 128).

Re: YaLM-100B: Pretrained language model with 100B parameters

#313

Post by riku_iki »

waffleiron wrote: post #306
riku_iki wrote: post #293

Earlier quoted context omitted.

What proof Snowden provided about Google and FB feeding data to NSA exactly?

https://en.wikipedia.org/wiki/PRISM#The_slides

There is no clarity on these slides if collection happened proactively or it was a way to transfer information for FISA warrants.

Re: YaLM-100B: Pretrained language model with 100B parameters

#314

Post by boplicity »

dav_Oz wrote: post #150
alexb_ wrote: post #13

I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Have to wonder what the knock-on effects of that will be, especially if the models improve drastically. With so much of our social lives being moved online, if we have the easy ability to create fake lives of fake people one has to wonder what's real and what isn't. Maybe the dead internet theory will rea…

The bots/machine vs human reminds me of that famous experiment from the 30s in which Winthrop Kellogg[0], a comparative psychologist, and his wife decided to raise their human baby (Donald) simultaneously with a chimpanzee baby (Gua) in an effort to "humanize the ape". It was set out to last 5 years but was relatively quickly abrupted after only 9 months. The explicit reason wasn't stated only that it successfully pr…

Case in point: recently, I've noticed that I'm getting more and more emails with the sign off "Warm regards." This is not a coincidence. It is an autosuggestion from Google. If you start signing off an email, it will automatically suggest "Warm regards." It just appears there -- probably an idea generated from an AI network. There are more and more of these algorithmic "suggestions" appearing every day, in more and more contexts. This is true for many text messaging programs: There are "common" replies suggested. How often do people just click on one of the suggested replies, as opposed to writing their own? These suggestions push us into conforming to the expectations of the algorithm, which then reinforces those expectations, creating a cycle of further pushing us into the language use patterns generated by software -- as opposed to idiosyncratic language created by a human mind.

In other words, people are already behaving like bots; and we're building more and more software to encourage such behavior.

Re: YaLM-100B: Pretrained language model with 100B parameters

#315

Post by sanxiyn »

ggktk wrote: post #272
sanxiyn wrote: post #261

Earlier quoted context omitted.

They can (and do) revoke API access from bad guys. They can't do that to downloaded models. Look, I don't like what OpenAI does, but "API access, but no model download" makes sense if you are worried about misuses.

Bad actors still can get access to such models. It even makes them more dangerous than it would if everyone had access to them. Here's an alternative: progressively release better and better models (like 3B params, 10B, 50B, 100B) and let people figure out the best way to fight against bad actors using them.

> It even makes them more dangerous than it would if everyone had access to them.

This is the sort of argument that proves guns would be less dangerous if everyone had access to them.

Re: YaLM-100B: Pretrained language model with 100B parameters

#316

Post by remram »

daniel-cussen wrote: post #243
SXX wrote: post #202

Earlier quoted context omitted.

OpenAI should just rebrand since nothing they do is actually open.

You know 100 years ago you could just buy uranium openly? Leo Szílard hustled up 200 kilograms, pleted, in the 30's.

Maybe they should rename to SafeAI, if their concern is controlling access.

Re: YaLM-100B: Pretrained language model with 100B parameters

#317

Post by lostmsu »

redox99 wrote: post #73
albertzeyer wrote: post #32

Earlier quoted context omitted.

If your disk has enough space to store the model, I think in theory you could run them, using the disk to store states. But it will be slow. I'm not sure how slow though, and also if anyone has implemented this. It actually should not be too difficult.

Disk makes no sense considering RAM is pretty cheap. But even then RAM is way too slow (and the communication overhead way too high). You probably get like a 100x slowdown or more.

I think you are overestimating compute and I/O for this model. If you assume it is RAM bandwidth bound, with a single channel top DDR4 you will get inference time as a low multiple of 7 seconds (200GB/25GBs). In a workstation you can have 8 channels.

Re: YaLM-100B: Pretrained language model with 100B parameters

#318

Post by jeanlucas »

whoami_nr wrote: post #239
pembrook wrote: post #138

Side note: Yandex search is awesome, and I really hope they stay alive forever. It's the only functional image search nowadays, after our Google overlords neutered their own product out of fear over lawyers/regulation and a disdain for power users. You can't even search for images "before:date" in Google anymore.

FWIW, https://same.energy/ seems to work fine for me

A 500 days old product in beta? I hope they do well.

Re: YaLM-100B: Pretrained language model with 100B parameters

#319

Post by lostmsu »

tomp wrote: post #309
SXX wrote: post #167

First of all regardless for political situation this is great step in making ML research actually open. So huge thanks for those developers who pushed to make it public. Still... Yandex is in fact share responsibility for Russian government actions. While it impossible to fight censorship they could certainly shut down their News service completely. Yandex could also certainly move more of their company and staff out…

How much responsibility does Google share for US wrecking Afghanistan, Iraq, Libya and Syria?

Google doesn't censor antiwar propaganda.

Re: YaLM-100B: Pretrained language model with 100B parameters

#320

Post by lostmsu »

manishsharan wrote: post #234

Is there a way for developers, who do not have AI/ML background, to get started using this ? I have been curious about GPT-3 but I do not have any AI/ML experience or knowledge. Is there a "approachable" course on Coursera or Udemy that could help me get started with technologies like GPT ?

https://www.deepspeed.ai/
Post reply on HN