Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

δ-mem: Efficient Online Memory for Large Language Models

arxiv.org

Post reply on HN
Print view View original

61–69 of 69 posts

Re: δ-mem: Efficient Online Memory for Large Language Models

#61

Post by xcvbnu »

usernametaken29 wrote: post #54
ltbarcly3 wrote: post #52

Earlier quoted context omitted.

You don't remember a lifetime of smells. You don't have any memories from huge swaths of time. There are entire years of your life compressed down to vibes and a handful of events you largely misremember.

That’s a very weak argument. Memories are not exact replica of experiences. We know that many memories are retained through a lifetime, particularly the ones from early childhood. Unlike computers we always reconstruct memories from several modalities. Even if we remember largely on vibes as you say (which is not true when you look into neuroscience), the sheer amount of information is overwhelming. Again, try to run…

[flagged]

Re: δ-mem: Efficient Online Memory for Large Language Models

#62

Post by RandomBK »

in-silico wrote: post #46
usernametaken29 wrote: post #11

> δ-mem compresses past information into a fixed-size state matrix updated by delta-rule learning This doesn’t solve the capacity problem of memory. You can cram more into one context window, but then again you need to associate them with input queries. That’s very hard because slight variations in input create hugely different activations. So really, it doesn’t improve caching. This paper might do a thing or two app…

While there is a limit to the amount of information you can fit in a fixed-size state, the theoretical ceiling is pretty high. A Hebbian associative matrix (one of the simplest and weakest memory constructions) can store about 0.7 bits of information per parameter. If you have a state with 300M parameters (the size of a Llama 3 8B KV cache at 10K context length), and a context with 2.1 bits of entropy per token (a re…

> context with 2.1 bits of entropy per token

Can you elaborate on this? I'm seen estimates of ~1.5bit per English letter, and tokens encode a lot more than that - sometimes full words, with multimodal even more. If KV cache embedding are storing more than just simple tokens but entire concepts with context and nuance, that'll bump the entropy up quite quickly.

Re: δ-mem: Efficient Online Memory for Large Language Models

#63

Post by in-silico »

RandomBK wrote: post #62
in-silico wrote: post #46

Earlier quoted context omitted.

While there is a limit to the amount of information you can fit in a fixed-size state, the theoretical ceiling is pretty high. A Hebbian associative matrix (one of the simplest and weakest memory constructions) can store about 0.7 bits of information per parameter. If you have a state with 300M parameters (the size of a Llama 3 8B KV cache at 10K context length), and a context with 2.1 bits of entropy per token (a re…

> context with 2.1 bits of entropy per token Can you elaborate on this? I'm seen estimates of ~1.5bit per English letter , and tokens encode a lot more than that - sometimes full words, with multimodal even more. If KV cache embedding are storing more than just simple tokens but entire concepts with context and nuance, that'll bump the entropy up quite quickly.

> Can you elaborate on this? I'm seen estimates of ~1.5bit per English letter

The reference I always go back to is the GPT-3 paper. The cross-entropy loss (an upper bound for entropy) got down to 1.75 nats (2.5 bits). I took 2.1 because 2.5 is an upper bound and I wanted the estimate to end up as a round number.

> If KV cache embedding are storing more than just simple tokens but entire concepts with context and nuance, that'll bump the entropy up quite quickly.

Here's the thing: the concepts that the model stores in the KV cache are a deterministic function of the input tokens. Similar to the data processing inequality, this implies that no entropy is actually added.

Looking at it mechanically, a sufficiently powerful model only needs to encode the tokens and can recompute concepts later as needed.

Re: δ-mem: Efficient Online Memory for Large Language Models

#64

Post by jsemrau »

ogogmad wrote: post #59
jsemrau wrote: post #22

Earlier quoted context omitted.

I am currently working on deep context query which uses dynamically generated regex to pull only the relevant context blocks. By using lightweight RegEx pattern matching to detect semantic intent and filter structured context sections accordingly, you avoid the attention degradation that comes from stuffing semantically redundant information into the window https://jdsemrau.substack.com/p/tokenmaxxing-and-optimizing-…

This is one of the most interesting comments I've read on this website.

Thank you.

Re: δ-mem: Efficient Online Memory for Large Language Models

#65

Post by adrian_b »

djoldman wrote: post #38

I would love for the standard to be to ALWAYS report the required amount of memory to load and run a model in bytes of RAM alongside any other metrics. I'd love to see time to first token, token throughput, token latency as well but I'd settle for memory size as described above. Essentially, many people want to know what the minimum amount of memory is to run a particular model. Parameter count obscures important det…

As a proxy for the total size of the parameters, you can just look at the download size of a model on Huggingface.co.

Because for most models the weights are provided in many *.safetensors files of approximately the same size, you can estimate the total size without adding all file sizes by multiplying the number of *.safetensors files with the approximate size of one file.

For quantized models, estimating the size is simpler, because there is just one GGUF file, which also includes metadata, but most of the file is occupied by the parameters.

While there are models where the native size of all parameters is BF16, there are also models that use multiple parameter sizes, e.g. a large number of parameters with a small size, even down to 4 bits, together with a small number of parameters with a bigger size, up to FP32. Therefore, as you say, the number of parameters is much less informative about memory requirements than the file sizes.

While the download size of the *.safetensors files or GGUF files is not the same as the total memory requirement, it can give an approximate estimate and it can be used to assess which of 2 models will need more memory. It becomes more complicated when you must use multiple kinds of memory, e.g. GPU memory and CPU memory, or even SSDs, when you must know more about the structure of the model to determine how much of each kind of memory is needed.

Re: δ-mem: Efficient Online Memory for Large Language Models

#67

Post by magicalhippo »

adrian_b wrote: post #65
djoldman wrote: post #38

I would love for the standard to be to ALWAYS report the required amount of memory to load and run a model in bytes of RAM alongside any other metrics. I'd love to see time to first token, token throughput, token latency as well but I'd settle for memory size as described above. Essentially, many people want to know what the minimum amount of memory is to run a particular model. Parameter count obscures important det…

As a proxy for the total size of the parameters, you can just look at the download size of a model on Huggingface.co. Because for most models the weights are provided in many *.safetensors files of approximately the same size, you can estimate the total size without adding all file sizes by multiplying the number of *.safetensors files with the approximate size of one file. For quantized models, estimating the size i…

The KV cache size is a joker though. Different models use very different amounts of memory per token in the KV cache. The VRAM requirements for say 64k context can vary almost by an order of magnitude. So the download size might indicate you should have room for the model, how much context you can fit in the leftover VRAM budget is harder to predict at a glance.

That some models like Qwen3.6 27B seems to not be very affected by Q8 quantized KV cache while others degrade heavily doesn't make it easier.

Re: δ-mem: Efficient Online Memory for Large Language Models

#68

Post by pohl »

DeathArrow wrote: post #2

I see lots of techniques proposed to give LLM the capacity to recall things, I even saw a lot of memory plugins for AI coding agents, I tried some myself. What I want to see is something that was tested and proved in practice to be genuinely useful, especially for coding agents.

There’s probably never going to be one answer. The most fascinating thing about this quest for memory is that it’s a Rorschach test. Exploring the myriad attempts to implement memory shows that everyone has a slightly different itch they’re trying to scratch, but we talk about it like we all want the same thing.

Re: δ-mem: Efficient Online Memory for Large Language Models

#69

Post by ltbarcly3 »

usernametaken29 wrote: post #54
ltbarcly3 wrote: post #52

Earlier quoted context omitted.

You don't remember a lifetime of smells. You don't have any memories from huge swaths of time. There are entire years of your life compressed down to vibes and a handful of events you largely misremember.

That’s a very weak argument. Memories are not exact replica of experiences. We know that many memories are retained through a lifetime, particularly the ones from early childhood. Unlike computers we always reconstruct memories from several modalities. Even if we remember largely on vibes as you say (which is not true when you look into neuroscience), the sheer amount of information is overwhelming. Again, try to run…

blah blah we know that blah neuroscience blah blah blah.

This isn't an argument you are making, it's just an assertion that you could make an argument if you are so inclined, but you won't be doing so at this time, but "science" is obviously on your side, but you can't be bothered to say how or even enough detail for someone to check what you are referring to. I can do that to, see my first sentence in this reply.

I don't know how LLM memory systems work. I do know that you don't have a lifetime of remembering everything with high precision. Not only do most people not remember the plot of most of the movies they have seen, they can't reliably list most of the movies they have seen. Not everyone has a good memory. My point is that it's not valid to reference a false model of how human memory works as a reason some specific LLM memory implementation isn't useful for solving some problems.

Post reply on HN