Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenClaw’s memory is unreliable, and you don’t know when it will break

blog.nishantsoni.com

Post reply on HN
Print view View original

161–170 of 196 posts

Re: OpenClaw’s memory is unreliable, and you don’t know when it will break

#161

Post by mike_hearn »

gbro3n wrote: post #149
mike_hearn wrote: post #145

Earlier quoted context omitted.

This is interesting. I haven't used OpenClaw but I set up my own autonomous agent using Codex + ChatGPT Plus + systemd + normal UNIX email and user account infrastructure. And it's been working great! I'm very happy with it. It's been doing all kinds of tasks for me, effectively as an employee of my company. I haven't seen any issues with memory so far. Using one long rolling context window, a diary and a markdown wi…

What sort of tasks do you have it do for you?

Two categories: actual useful work for the company, and improving the bot's own infrastructure.

Useful work includes: bug triage, matching up external user bug reports on GitHub to the internal YouTrack, fixing easy looking bugs, working on a redesign of the website. I also want to extend it to handling the quarterly accounting, which is already largely automated with AI but I still need to run the scripts myself, preparing answers to support queries, and more work on bug fixing+features. It has access to the bug tracker, internal git and CI system as if it were an employee and uses all of those quite successfully.

Meta-work has so far included: making a console so I can watch what it's doing when it wakes up, regularly organizing its own notes and home directory, improving the wakeup rhythm, and packaging up its infrastructure to a repeatable install script so I can create more of them. I work with a charity in the UK whose owner has expressed interest in an OpenClaw but I warned him off because of all the horror stories. If this experiment continues to work out I might create some more agents for people like him.

I'm not sure it's super useful for individuals. I haven't felt any great need to treat it as a personal assistant yet. ChatGPT web UI works fine for most day to day stuff in my personal life. It's very much acting like an extra employee would at a software company, not a personal secretary or anything like that.

It sounds like our experience differs because you wanted something more controlled with access to your own personal information like email, etc, whereas I gave "Axiom" (it chose its own name) its own accounts and keep it strictly separated from mine. Also, so far I haven't given it many regular repeating tasks beyond a nightly wakeup to maintain its own home directory. I can imagine that for e.g. the accounting work we'd need to do some meta-work first on a calendar integration so it doesn't forget.

Re: OpenClaw’s memory is unreliable, and you don’t know when it will break

#162

Post by osigurdson »

_pdp_ wrote: post #48

IMHO, the biggest problem with OpenClaw and other AI agents is that the use-cases are still being discovered. We have deployed several hundred of these to customers and I think this challenge comes from the fact that AI agents are largely perceived as workflow automation tools so when it comes to business process they are seen as a replacement for more established frameworks. They can automate but they are not reliab…

Can you be more specific about these? For instance can you map "analyzes our current marketing efforts" to precisely what is being fed into the context window. What prompt is used to "find something useful"?

Like many here, I am struggling to see a meaningful delta between OC and CC but fully willing to accept that my skepticism is misplaced. Basically, I am in "trying to care about OC" mode right now.

Re: OpenClaw’s memory is unreliable, and you don’t know when it will break

#164

Post by BeetleB »

fg137 wrote: post #92
BeetleB wrote: post #44

If you look at my comment history, you'll see what seems to be someone defending OpenClaw (even though I stopped using it). I have some issues with the article, but I agree with some of the conclusions: It's great tinkering with it if you have time to spare, but not worth using weeks of your time trying to get a perfect setup. It's just not that reliable to use up so much of your time. I will say, it's still amongst…

> I could tell it (via Telegram) to add something to my TODO list at home while I'm in the office. It would call a custom API I had set up that adds items to my TODO list. All the existing, commodity todo list apps on the market can't address your use cases? At least I can't tell there is anything you can't do on your personal phone.

> All the existing, commodity todo list apps on the market can't address your use cases?

Nope. I've custom honed my TODO system since 2009. I'm not switching for some one else's app.

And I don't use phones.

Re: OpenClaw’s memory is unreliable, and you don’t know when it will break

#165

Post by wg0 »

ejeje wrote: post #144
wg0 wrote: post #126

Earlier quoted context omitted.

I am sorry. I understand that many of us are heavily invested financially or emotionally into AI but the key insight about OpenClaw remains the same that it really isn't much useful beyond maybe a daily news summary. Nothing that cannot be done otherwise.

I think it broadly speaks to the issue of llm’s in general. People have to actively interact with these things to get outputs of them. Why can’t they do it passively? Because on the surface LLM’s seem intelligent. But they’re not. They need constant attention to ensure they are doing what one wants and not going off track. I experienced this myself many times - the variance in the output can be jarring at times. From…

Then intellectual honesty demands that we present these limitations as such but capitalist grifters have their own principles it seems.

Re: OpenClaw’s memory is unreliable, and you don’t know when it will break

#166

Post by catlifeonmars »

ambewas wrote: post #124
loehnsberg wrote: post #122

As long as there's no solution to the long-term memory problem, we will have a "country of geniuses in a data center" that are all suffering from anterograde amnesia (movie: Memento), which requires human hand-holding. I have experimented with a lot of hacks, like hierarchies of indexed md files, semantic DBs, embeddings, dynamic context retrieval, but none of this is really a comprehensive solution to get something…

You're right to be skeptical. Without a way to actually implement how the human brain processes experiences into a consolidated memory, we won't be able to solve the long term memory problem at all. Not with the current technology. An LLM context is a pretty well extended short term memory, and the trained network is a very nice comprehensive long term memory, but due to the way we currently train these networks, an…

Why does a language model have to be monolithic? I think retraining a model is expensive (relatively speaking). Is there some way to bolt on specialization?

Re: OpenClaw’s memory is unreliable, and you don’t know when it will break

#167

Post by indigodaddy »

mike_hearn wrote: post #145
gbro3n wrote: post #130

Earlier quoted context omitted.

I've used open claw (just for learning, I agree with the author it's not reliable enough to do anything useful) but also have a similar daily summary routine which is a basic gemini api call to a personal mcp server that has access to my email, calendar etc. The latter is so much more reliable. Open claw flows sometimes nail it, and then the next day fails miserably. It seems like we need a way to 'bank' the correct…

This is interesting. I haven't used OpenClaw but I set up my own autonomous agent using Codex + ChatGPT Plus + systemd + normal UNIX email and user account infrastructure. And it's been working great! I'm very happy with it. It's been doing all kinds of tasks for me, effectively as an employee of my company. I haven't seen any issues with memory so far. Using one long rolling context window, a diary and a markdown wi…

Any chance you might put this on GH? Sounds really interesting.

Re: OpenClaw’s memory is unreliable, and you don’t know when it will break

#168

Post by ambewas »

loehnsberg wrote: post #131
ambewas wrote: post #124

Earlier quoted context omitted.

You're right to be skeptical. Without a way to actually implement how the human brain processes experiences into a consolidated memory, we won't be able to solve the long term memory problem at all. Not with the current technology. An LLM context is a pretty well extended short term memory, and the trained network is a very nice comprehensive long term memory, but due to the way we currently train these networks, an…

I think if we want to build on what we have, instead of compaction at the end of the context window, the LLM would have to 'sleep', i.e. adjust its weights, then wake up with the last bits of the old context window in the new one, and have a 'feel' for what it did before through the change in weights. I just sense it's not that simple to get there, because simply updating the weights based on a single context sample…

Sleep would probably be a part of the equation for consolidating , but there's still the question of how exactly does the brain process the information during sleep in a way that it permanently consolidates the information.

It's not how an llm can work right now, it needs too much iterations & a much bigger dataset than what we can work with. A single time experiencing something and we can remember it. That's orders of magnitude more efficient than an LLM right now can achieve.

Re: OpenClaw’s memory is unreliable, and you don’t know when it will break

#169

Post by ambewas »

catlifeonmars wrote: post #166
ambewas wrote: post #124

Earlier quoted context omitted.

You're right to be skeptical. Without a way to actually implement how the human brain processes experiences into a consolidated memory, we won't be able to solve the long term memory problem at all. Not with the current technology. An LLM context is a pretty well extended short term memory, and the trained network is a very nice comprehensive long term memory, but due to the way we currently train these networks, an…

Why does a language model have to be monolithic? I think retraining a model is expensive (relatively speaking). Is there some way to bolt on specialization?

That's exactly the issue. Retraining is too expensive & needs too much iteration to work efficiently I think.

Re: OpenClaw’s memory is unreliable, and you don’t know when it will break

#170

Post by ambewas »

airstrike wrote: post #151
ambewas wrote: post #124

Earlier quoted context omitted.

You're right to be skeptical. Without a way to actually implement how the human brain processes experiences into a consolidated memory, we won't be able to solve the long term memory problem at all. Not with the current technology. An LLM context is a pretty well extended short term memory, and the trained network is a very nice comprehensive long term memory, but due to the way we currently train these networks, an…

> I think this is probably the most interesting field of research right now. Actually understanding in depth how the brain learns, and figuring out a way to build a model that implements this. This field of research has been around for decades, so who's to say when there'll be a breakthrough. In fact, LLMs are great despite our very limited understanding, and not because we had some breakthrough about the human brain…

Exactly. It's been around so long and we still don't know how to mimic it.

The way an llm learns is a very interesting way of doing it, but it sure isn't what the brain is doing.

But it's indisputable.. We can get enormous results with this technique. It's just probably not the way forward for faster learning to remediate the issue of context loss.

Post reply on HN