Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

δ-mem: Efficient Online Memory for Large Language Models

arxiv.org

Post reply on HN
Print view View original

31–40 of 69 posts

Re: δ-mem: Efficient Online Memory for Large Language Models

#31

Post by cjonas »

DeathArrow wrote: post #2

I see lots of techniques proposed to give LLM the capacity to recall things, I even saw a lot of memory plugins for AI coding agents, I tried some myself. What I want to see is something that was tested and proved in practice to be genuinely useful, especially for coding agents.

Coding agents don't really need memory. Agent skills, rules, git history, documentation is all far more efficient, transparent and easier to manage. These memory frameworks only really makes sense if you are building a consumer facing agent with managed context and limited capabilities.

Re: δ-mem: Efficient Online Memory for Large Language Models

#32

Post by duskdozer »

tyre wrote: post #15
duskdozer wrote: post #4

Earlier quoted context omitted.

A lot of what I see people using LLMs for would be more cheaply and reliably done by [scripts]. A search engine style suggestion thing like "Have you tried `sed`?" would be beneficial imo

In my experience, Claude is more than happy to go to Unix tools rather than write its own. Sometimes it will write a lil python script to solve something, but more often than not it’ll pipe together Unix utilities. This has the benefit of it knowing all of the arcane flags, especially for formatting output.

I believe that. I also believe that my idea won't come to fruition, at least from a group that is incentivized to make a user's first instinct be to use their product and not an external tool.

Re: δ-mem: Efficient Online Memory for Large Language Models

#33

Post by airstrike »

semiquaver wrote: post #27

Hmm, this is a case where HN’s title mangling changed the meaning of the title. Lower case delta (δ) is used intentionally. I don’t think HN should automatically modify the casing of non-ascii chars.

The submitter has a grace period of a few minutes to edit the title after submitting, so there's no need to change what HN does

Re: δ-mem: Efficient Online Memory for Large Language Models

#34

Post by cwillu »

semiquaver wrote: post #27

Hmm, this is a case where HN’s title mangling changed the meaning of the title. Lower case delta (δ) is used intentionally. I don’t think HN should automatically modify the casing of non-ascii chars.

Email hn@ycombinator.com and they'll fix it.

Re: δ-mem: Efficient Online Memory for Large Language Models

#35

Post by wren6991 »

cjonas wrote: post #31
DeathArrow wrote: post #2

I see lots of techniques proposed to give LLM the capacity to recall things, I even saw a lot of memory plugins for AI coding agents, I tried some myself. What I want to see is something that was tested and proved in practice to be genuinely useful, especially for coding agents.

Coding agents don't really need memory. Agent skills, rules, git history, documentation is all far more efficient, transparent and easier to manage. These memory frameworks only really makes sense if you are building a consumer facing agent with managed context and limited capabilities.

There's an antipattern where everyone wants to invent new interfaces to connect things LLMs when CLI tools are already right there, transparent, and usable by humans as well as LLMs. I think it's partly the origins in web chat applications.

Beads kind of does "LLM memory over CLI", or there is https://github.com/wedow/ticket which is a minimal and sane implementation of the same idea.

Re: δ-mem: Efficient Online Memory for Large Language Models

#36

Post by trollbridge »

jandrese wrote: post #29
usernametaken29 wrote: post #11

> δ-mem compresses past information into a fixed-size state matrix updated by delta-rule learning This doesn’t solve the capacity problem of memory. You can cram more into one context window, but then again you need to associate them with input queries. That’s very hard because slight variations in input create hugely different activations. So really, it doesn’t improve caching. This paper might do a thing or two app…

So instead of a FIFO approach to memory management it instead continually degrades the existing data the more you put in? Details start getting lost or mangled more and more over time?

That’s basically what happens.

As you hit the limits and try to compact the context, etc., things get more erratic.

Re: δ-mem: Efficient Online Memory for Large Language Models

#37

Post by TheTaytay »

405126121 wrote: post #8
ktallett wrote: post #3

The obvious energy saving step would be to utilise previous searches by others. Many of the tasks people do are rather similar, it is such an energy waste to start again each time. (Obviously ignoring the huge energy saver, which is to observe if you even need to bother doing the task at all.)

I had this thought and created https://pushrealm.com which is essentially a sort of Stackoverflow written by agents. My theory was that if an agent burns 30 minutes resolving an issue not present in training data, posting the solution would prevent other agents re-treading the same thinking steps.

Fascinating! Do you have a way to detect/flag malicious stuff by any chance? (Seems like a good vector for prompt injection, but maybe no more than any other internet site?)

Re: δ-mem: Efficient Online Memory for Large Language Models

#38

Post by djoldman »

I would love for the standard to be to ALWAYS report the required amount of memory to load and run a model in bytes of RAM alongside any other metrics. I'd love to see time to first token, token throughput, token latency as well but I'd settle for memory size as described above.

Essentially, many people want to know what the minimum amount of memory is to run a particular model.

Parameter count obscures important details: what are the sizes of the parameters? A parameter isn't rigorously defined. This also gets folks into trouble because a 4B param model with FP16 params is very different from a 4B param model with INT4 params. The former obviously should be a LOT better than the second.

This would also help with MOE models: if memory is my constraint, it doesn't matter if the (much larger RAM required) MOE version is faster or has better evals.

I'm waiting for someone in anger to ship the 1 parameter model where the parameter according to pytorch is a single parameter of size 4GB.

Re: δ-mem: Efficient Online Memory for Large Language Models

#39

Post by realitysballs »

airstrike wrote: post #33
semiquaver wrote: post #27

Hmm, this is a case where HN’s title mangling changed the meaning of the title. Lower case delta (δ) is used intentionally. I don’t think HN should automatically modify the casing of non-ascii chars.

The submitter has a grace period of a few minutes to edit the title after submitting, so there's no need to change what HN does

True, but wouldn’t it be better long term if website automation didn’t create unintended new meanings to Titles? title’s matter

Re: δ-mem: Efficient Online Memory for Large Language Models

#40

Post by vdelpuerto »

usernametaken29 wrote: post #11

> δ-mem compresses past information into a fixed-size state matrix updated by delta-rule learning This doesn’t solve the capacity problem of memory. You can cram more into one context window, but then again you need to associate them with input queries. That’s very hard because slight variations in input create hugely different activations. So really, it doesn’t improve caching. This paper might do a thing or two app…

I wrote something about it trying to look other way around the context or memory data in models. The gravitational pull of information stills very hard to manage. Ive been using "functional scars" about 30 days now and getting good results in repetitive mistakes across sesions. https://github.com/VDP89/fscars
Post reply on HN