Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind’s new AI with a memory outperforms algorithms 25 times its size

singularityhub.com

Post reply on HN
Print view View original

71–80 of 141 posts

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#71

Post by regularfry »

vkazanov wrote: post #60
LudwigNagasena wrote: post #55

Earlier quoted context omitted.

Well, it is pretty hard to make something in a language when it is a dialect continuum and not a standardized variety that is forced onto the whole population through the education system and media.

Not really relevant to the topic in question but... Isn't this how most languages begin? First you have a continuum of language dialects, then one of them dominates for political reasons, then it gets codified, then enforced onto everybody through centralised education. Dialects not under direct unified political control become related but separate languages... And so on.

Yes, but I think the point is that trying to record a standardised version of something that has not yet been codified through that process is going to inevitably be somewhat approximate at best.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#72

Post by regularfry »

muzani wrote: post #70
AJRF wrote: post #51

I've not kept track of where large transformers like this have gotten to, GPT3 and the like - has GPT3 made any real difference to the world? Are people using it? Has it vastly improved any software?

I don't know about world changing but it's saved me hundreds of hours. I use it to help read academic papers, put formatting on things like markdown and subtitles, and creative writing. A lot of the things that take it 15 seconds to do take me 2 minute and drain me mentally for about 15 mins. If anything, it's being used in force for social media marketing, where you're trying to say "buy this thing" in different way…

Forgive the ignorance, but how? What tools are you using on top of GPT3 to do those things?

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#75

Post by schleck8 »

sbierwagen wrote: post #21
whazor wrote: post #15

Very interesting. GPT-J is an opensource free alternative to GPT-3 and requires at least 12.1GB memory to run the model (which is reduced from original 48GB ram). But if the model stores some kind of index and does internet searches (or hard drive) instead, then it could scale much further as there is a limit on how much memory you can use in production.

>48GB ram 48GB VRAM? 48+ gigabytes of system ram is cheap, 48 gigabytes of ram on a GPU is still painfully expensive.

GPUs are the slowing factor in general if I'm not mistaken when it comes to Deep Learning progress.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#77

Post by GistNoesis »

endymi0n wrote: post #58
whazor wrote: post #15

Very interesting. GPT-J is an opensource free alternative to GPT-3 and requires at least 12.1GB memory to run the model (which is reduced from original 48GB ram). But if the model stores some kind of index and does internet searches (or hard drive) instead, then it could scale much further as there is a limit on how much memory you can use in production.

Just doing some napkin math, the whole GPT-J corpus was around 500 billion tokens, which at 4 tokens per byte would be roundabout 2 Terabyte. That, parked on a fast NVMe SSD will give you roundabout 1MM random lookups per second. Even with some transfers inbetween, this should be more than enough to not just perform in equal time, but probably less — as well as cost you less than the GPU you need for the reduced size…

The real problem with (NVMe) SSD is that they have a limited number of write cycles (a max TB written).

If you don't update your database and indices they are great. But that's something really tempting to do when you do some machine learning, (specially if you know that people with deeper pockets will do so).

Typically you will have a neural network, you run it on your dataset, it produces a new dataset of embeddings, you index them, and you use this index to train a new neural network, and you repeat the loop, hopefully improving results along the way.

NVMe SSD can write at 6GB/s but can only write ~800TB that's about 37 hours of lifetime at max speed.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#78

Post by tveita »

AJRF wrote: post #51

I've not kept track of where large transformers like this have gotten to, GPT3 and the like - has GPT3 made any real difference to the world? Are people using it? Has it vastly improved any software?

It's a safe bet that Google is using transformers at scale for search and translations - the full extent isn't public but they release a fair amount of research papers, e.g. the current article, or https://ai.googleblog.com/2020/06/recent-advances-in-google-...

Github Copilot is definitely GPT-3-based and is seeing real-world use https://copilot.github.com

Transformers are state of the art for many tasks so they are likely to be used for "intelligent" processing of text or speech data, but due to practical limitations you are probably interacting with them mostly through web services.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#79

Post by motoboi »

regularfry wrote: post #72
muzani wrote: post #70

Earlier quoted context omitted.

I don't know about world changing but it's saved me hundreds of hours. I use it to help read academic papers, put formatting on things like markdown and subtitles, and creative writing. A lot of the things that take it 15 seconds to do take me 2 minute and drain me mentally for about 15 mins. If anything, it's being used in force for social media marketing, where you're trying to say "buy this thing" in different way…

Forgive the ignorance, but how ? What tools are you using on top of GPT3 to do those things?

There is an API for GPT-3.

GitHub copilot uses Codex, a descendent of GPT-3.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#80

Post by wiz21c »

muzani wrote: post #70
AJRF wrote: post #51

I've not kept track of where large transformers like this have gotten to, GPT3 and the like - has GPT3 made any real difference to the world? Are people using it? Has it vastly improved any software?

I don't know about world changing but it's saved me hundreds of hours. I use it to help read academic papers, put formatting on things like markdown and subtitles, and creative writing. A lot of the things that take it 15 seconds to do take me 2 minute and drain me mentally for about 15 mins. If anything, it's being used in force for social media marketing, where you're trying to say "buy this thing" in different way…

Seconding someone else's comment : what is your workflow for those tasks ? How does it help you to read academic papers ? Or to put formatting on markdown ?
Post reply on HN