DeepMind’s new AI with a memory outperforms algorithms 25 times its size
singularityhub.com
DeepMind’s new AI with a memory outperforms algorithms 25 times its size
1–10 of 141 posts
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#2Dup https://news.ycombinator.com/item?id=29486607
(This is a different blogpost, but does not seem to add over the original)
Edit: following derac's comment see https://news.ycombinator.com/item?id=29646112 for RETRO
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#3Dup https://news.ycombinator.com/item?id=29486607 (This is a different blogpost, but does not seem to add over the original) Edit : following derac's comment see https://news.ycombinator.com/item?id=29646112 for RETRO
This article is about RETRO [0], not Gopher.
[0] https://deepmind.com/research/publications/2021/improving-la...
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#4Could someone explain the article to layman engineer?
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#5Seems odd to claim 25x reduction in size when the algo involves looking into a database of a trillion chunks of text.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#6Seems odd to claim 25x reduction in size when the algo involves looking into a database of a trillion chunks of text.
The "algo" here refers to the neural net itself. The text index is considered an easy problem as you can do lookups in logarithmic time.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#7A neural net with access to wikipedia is faster than than a neural net that contains Wikipedia? Seems odd to call it AI with a memory though... unless I'm misunderstanding. It's more like AI with a decent memory and an understanding of how to use an encyclopedia.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#8Could someone explain the article to layman engineer?
It's language modelling with search engine in-the-loop.
Instead of training GPT-3 with 178B weights, you train a 25x smaller model and allow it to retrieve useful snippets from a large text index as additional information.
This solves the problem of very large models and the problem of updating an already trained model, as you can swap the text corpus with a newer one. The model learns mostly syntax, burning less trivia in its weights than a regular LM as it can simply copy the relevant information from the index.
This development was bound to happen as large LMs are expensive to use and it was an obvious idea. We've had these semantic search text indices for a few years already[1], they just weren't combined with text generation.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#9A neural net with access to wikipedia is faster than than a neural net that contains Wikipedia? Seems odd to call it AI with a memory though... unless I'm misunderstanding. It's more like AI with a decent memory and an understanding of how to use an encyclopedia.
Yeah, memory implies persisted state in the model, this is static lookups separate from the transformer.
Still superb, though, there's no reason you can't use other gofai tools vs a static database, to trigger expert systems or formalized reasoning.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#10If we point it at the horrendously bad scots wiki (some kid in the US decided he'd translate Wikipedia into what he thought was lowland scots/Doric.. it's a disaster) we might get entertainingly bad outcomes.