Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Infinite Context LLMs: Going Beyond RAG with Extended Minds

blog.normalcomputing.ai

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds

#31

Post by ilaksh »

mike_hearn wrote: post #29
ilaksh wrote: post #23

Their experiment seems to handle less than 100,000 tokens of text. I wonder if this method could scale to more than 1,000,000 or 10,000,000 tokens. It seems like it's limited to context length? So with 512 token chunks at 4096 context length that is 2 million tokens. Which could be good for some things but is not going to help for a really large knowledgebase. Edit: I see the source code is in the Files section on th…

It's open source. The PyTorch sources are linked from the blog post.

Thanks. I just assumed that it would be on their Github. But it's in the Files section on HuggingFace.

Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds

#32

Post by naasking »

mike_hearn wrote: post #26
naasking wrote: post #25

Earlier quoted context omitted.

> Instead we have short term memory [...], a sort of medium-term memory that can absorb lots of information [...], and a general long term memory that is rarely forgotten This is basically folk psychology though, not necessarily representative of the actual mechanisms that are being used. This is a common mistake I see in comparisons of ML and neuroscience or psychology. ML is all about mechanistic models for things,…

Sure it's just an analogy. We clearly don't have 3 distinct separate levels, it's more like one memory where recalling memories strengthens them (but can also change them). But whatever we have, it's clearly very different to an LLM. We don't even want to replicate the human mind exactly. If we did that what'd be the point? AI is useful exactly because it's not the same as how people work. I don't think RAG is a good…

> But whatever we have, it's clearly very different to an LLM.

My only real quibble with your post is, again, that it seems different to an LLM. Current ML models are still multiple orders of magnitude away from being on the same order of complexity as the human brain, and scaling doesn't yet seem to have hit a limit (more a data limit).

I just don't think we can draw such conclusions at this time.

Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds

#33

Post by mike_hearn »

zby wrote: post #30
mike_hearn wrote: post #28

Earlier quoted context omitted.

I think it will have been fine-tuned to use functions if they exist, as for most use cases if you specify a function then it means the other end of the connection is an automated program and not a human, so returning pure text without a function call would be a failure.

Maybe - but why they forget all reasoning and try to jump directly to the answer?

If there's a search function the implication is that you want the LLM to search rather than guess? I mean, if I were an LLM, that's what I'd infer ...

Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds

#34

Post by mike_hearn »

naasking wrote: post #32
mike_hearn wrote: post #26

Earlier quoted context omitted.

Sure it's just an analogy. We clearly don't have 3 distinct separate levels, it's more like one memory where recalling memories strengthens them (but can also change them). But whatever we have, it's clearly very different to an LLM. We don't even want to replicate the human mind exactly. If we did that what'd be the point? AI is useful exactly because it's not the same as how people work. I don't think RAG is a good…

> But whatever we have, it's clearly very different to an LLM. My only real quibble with your post is, again, that it seems different to an LLM. Current ML models are still multiple orders of magnitude away from being on the same order of complexity as the human brain, and scaling doesn't yet seem to have hit a limit (more a data limit). I just don't think we can draw such conclusions at this time.

Sure, but what I'm getting at is that even if you scale a GPT-4 style transformer up 1000x, it won't suddenly develop a human-like memory. It will still be read only, whereas there's no such thing as a read only human unless they are unfortunate enough to have severe brain damage.

Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds

#35

Post by bsenftner »

mike_hearn wrote: post #29
ilaksh wrote: post #23

Their experiment seems to handle less than 100,000 tokens of text. I wonder if this method could scale to more than 1,000,000 or 10,000,000 tokens. It seems like it's limited to context length? So with 512 token chunks at 4096 context length that is 2 million tokens. Which could be good for some things but is not going to help for a really large knowledgebase. Edit: I see the source code is in the Files section on th…

It's open source. The PyTorch sources are linked from the blog post.

It's open source with a noncommercial use license.

Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds

#37

Post by vorticalbox »

dragonwriter wrote: post #20
mike_hearn wrote: post #8

The blog post format is so much nicer than a PDF paper! More seriously, this does feel like a real advance. Vector search+context stuffing (RAG) is clearly a hack that doesn't resemble how we actually think or do things in reality. I've been wondering for the last year whether it's possible to extend the attention mechanism to more naturally connect to a bigger set of per-session weights or activations. The moment yo…

> Vector search+context stuffing (RAG) is clearly a hack that doesn't resemble how we actually think or do things in reality. Its actually a very close model of how people use external data sources to supplement thought in reality. Its a hack, but it is very much the same hack (indexing external data, and looking up and stuffing some in our brain temporarily at need) that humans have used forever (with a novel indexi…

I've noticed this as I've gotten older, I have started to remember "where" to find information rather than remembering it.

I end up with a lot of book marks as a result.

Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds

#39

Post by _boffin_ »

vorticalbox wrote: post #37
dragonwriter wrote: post #20

Earlier quoted context omitted.

> Vector search+context stuffing (RAG) is clearly a hack that doesn't resemble how we actually think or do things in reality. Its actually a very close model of how people use external data sources to supplement thought in reality. Its a hack, but it is very much the same hack (indexing external data, and looking up and stuffing some in our brain temporarily at need) that humans have used forever (with a novel indexi…

I've noticed this as I've gotten older, I have started to remember "where" to find information rather than remembering it. I end up with a lot of book marks as a result.

I have severe ADHD and have never been good at tests due to this issue right here.

I’m now a professional and I carry my books, notes, articles, papers, etc… with me everywhere via my iPad and margin notes.

I’ve found, since a young age that knowing where to find the information I need vs memorization is how my brain works best.

Wish it was the opposite though.

Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds

#40

Post by mike_hearn »

bsenftner wrote: post #35
mike_hearn wrote: post #29

Earlier quoted context omitted.

It's open source. The PyTorch sources are linked from the blog post.

It's open source with a noncommercial use license.

Huh. I wonder if that meets the definition of open source. I didn't notice the special license.
Post reply on HN