Live data from Hacker News

DeepMind’s new AI with a memory outperforms algorithms 25 times its size

singularityhub.com

81–90 of 141 posts

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#81
post #80
post #70

Earlier quoted context omitted.

I don't know about world changing but it's saved me hundreds of hours. I use it to help read academic papers, put formatting on things like markdown and subtitles, and creative writing. A lot of the things that take it 15 seconds to do take me 2 minute and drain me mentally for about 15 mins. If anything, it's being used in force for social media marketing, where you're trying to say "buy this thing" in different way…

Seconding someone else's comment : what is your workflow for those tasks ? How does it help you to read academic papers ? Or to put formatting on markdown ?

For academic papers, I basically just copy the abstract or particularly hard parts of a paper into this prompt: https://beta.openai.com/examples/default-tldr-summary

For formatting, I copy from our spreadsheet instruction, and copy what the "markdown" format looks like.

  Convert the following text to markdown format.

  Text:
  1. Open the app

  2. Enter your USERNAME and PASSWORD

  3. Go to the menu and click "Foobar"

  4. Enter the following into the field [CORRECT INPUT HERE]

  ```
  Markdown:
  # How to Use Foobar
  ## Subtitle
  - Open the app
  - Enter your USERNAME and PASSWORD
  - Select the menu and click Foobar
  - Enter the following into the field {correctInput}

  ```
  Text:
  (your input here)
Yes, I know it's not really markdown. But you give it an example of input and output and it will figure it out as easily as a human can.

The problem we faced was that it was meant to be markdown but it's not easy to write the parser, so we had a different format of "markdown" on different front ends. You'd have something a little different on Android, iOS, HTML.

But the beautiful thing is we can keep the same input and change the output to whatever the parser needs. And instead of having to write up regex that detects [CAPITAL LETTER INPUT] and converts that, the AI can just recognize it.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#82
post #70

Earlier quoted context omitted.

I don't know about world changing but it's saved me hundreds of hours. I use it to help read academic papers, put formatting on things like markdown and subtitles, and creative writing. A lot of the things that take it 15 seconds to do take me 2 minute and drain me mentally for about 15 mins. If anything, it's being used in force for social media marketing, where you're trying to say "buy this thing" in different way…

Forgive the ignorance, but how ? What tools are you using on top of GPT3 to do those things?

It's a funny question, like asking how to write a book with a computer, but perfectly valid.

You can access GPT-3 directly now. There's no waitlist, but there still are restrictions. There's some examples here: https://beta.openai.com/examples

You don't even need the API. Once you get access, it comes with access to the playground, which is enough to do anything you like.

If you look at the examples, it's very "no code". You literally tell the AI what you're trying to do and it tries its best. Most of the work in prompt engineering is writing something that can't be misunderstood. But you just have to explain to it what you want like you would to a child.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#83
post #81
post #80

Earlier quoted context omitted.

Seconding someone else's comment : what is your workflow for those tasks ? How does it help you to read academic papers ? Or to put formatting on markdown ?

For academic papers, I basically just copy the abstract or particularly hard parts of a paper into this prompt: https://beta.openai.com/examples/default-tldr-summary For formatting, I copy from our spreadsheet instruction, and copy what the "markdown" format looks like. Convert the following text to markdown format. Text: 1. Open the app 2. Enter your USERNAME and PASSWORD 3. Go to the menu and click "Foobar" 4. Ente…

Thanks ! Pretty smart !

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#84
post #8

Earlier quoted context omitted.

It's language modelling with search engine in-the-loop. Instead of training GPT-3 with 178B weights, you train a 25x smaller model and allow it to retrieve useful snippets from a large text index as additional information. This solves the problem of very large models and the problem of updating an already trained model, as you can swap the text corpus with a newer one. The model learns mostly syntax, burning less tri…

So the memory doesn't solve the context problem of e.g. "conversation context"? I.e. the storage isn't modified while the model is used? If I make an app that makes conversation using such a model model, then the storage isn't modified to insert knowledge about what the early parts of the conversation was about, and it's only bringing a database of fixed information into the conversation? (I have a friend who is just…

You could update your storage as you go, the indexing doesn't appear to be that expensive.

For many tasks it wouldn't be helpful because the input is small enough to be covered by the context already, and for summarizing and question answering tasks, you want it to repeat information from other documents, but not from earlier in its own output.

It might be interesting for a long-context task like "given the first parts of this book, complete the next chapter".

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#85
post #32

Earlier quoted context omitted.

What’s your argument?

I hope I didn't come off like I think know anything about this field, because I don't. A friend who use to work for openai (Jack Clark) and I spend some time once discussing over beers some stuff around general purpose AI, and I proposed that I believe quantum is a dark horse on the road to general purpose AI, and he disagreed.

I think the person you're replying to is asking for your specific argument. As in what exactly do you think is the advantage gained by quantum computing and why is it important.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#86

Earlier quoted context omitted.

That proportions in that analogy seem way off. Every single neuron of the human brain could be utilizing quantum mechanical effects at the same time. Not very much at all like a kid's sand castle against the world's tides. I don't think anyone really understand "thinking" and quantum mechanics well enough for the mechanisms to be apparent. I mean, squirrels convert forest detritus into general intelligence. I don't u…

I believe we've been able to model the processes behind neural firings for quite some time now, the issue is more mapping between that micro-scale understanding of how each individual piece in a 10^10 piece puzzle with 10^15 edges affects the macro-scale emergent properties of said puzzle. It's less a lack of understanding of the processes and more a complexity so great that it exceeds the grasp of our best modeling…

Can we barely model it yet? I thought they weren’t at the point of getting able to simulate any actual work behaviours?

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#87

Earlier quoted context omitted.

Yes. It's unavoidably "quantum" in the sense that, as a physical machine in our universe it's subject to the rules, including quantum physics. However there is no apparent mechanism by which "thinking" could harness any interesting properties of quantum physics, it's just not happening at the right scale, like the way kids sand walls on the beach don't alter the world's tides.

That proportions in that analogy seem way off. Every single neuron of the human brain could be utilizing quantum mechanical effects at the same time. Not very much at all like a kid's sand castle against the world's tides. I don't think anyone really understand "thinking" and quantum mechanics well enough for the mechanisms to be apparent. I mean, squirrels convert forest detritus into general intelligence. I don't u…

> Every single neuron of the human brain could be utilizing quantum mechanical effects

Yes, they are, in that neurons are made up of molecules which are held together and bind due to quantum effects. But that's the limit of it.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#88
post #15

Very interesting. GPT-J is an opensource free alternative to GPT-3 and requires at least 12.1GB memory to run the model (which is reduced from original 48GB ram). But if the model stores some kind of index and does internet searches (or hard drive) instead, then it could scale much further as there is a limit on how much memory you can use in production.

Isn't this what watson used to do?

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#90
post #7

A neural net with access to wikipedia is faster than than a neural net that contains Wikipedia? Seems odd to call it AI with a memory though... unless I'm misunderstanding. It's more like AI with a decent memory and an understanding of how to use an encyclopedia.

Yeah, memory implies persisted state in the model, this is static lookups separate from the transformer. Still superb, though, there's no reason you can't use other gofai tools vs a static database, to trigger expert systems or formalized reasoning.

It's not gofai. It's locality sensitive hashing published in 2008.
Post reply on HN