Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude’s memory architecture is the opposite of ChatGPT’s

shloked.com

Post reply on HN
Print view View original

41–50 of 240 posts

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#41

Post by glial »

ornornor wrote: post #14
modeless wrote: post #9

The link to the breakdown of ChatGPT's memory implementation is broken, the correct link is: https://www.shloked.com/writing/chatgpt-memory-bitter-lesson This is really cool, I was wondering how memory had been implemented in ChatGPT. Very interesting to see the completely different approaches. It seems to me like Claude's is better suited for solving technical tasks while ChatGPT's is more suited to improving casual…

> It may actually be the final breakthrough we need for AGI. I disagree. As I understand them, LLMs right now don’t understand concepts. They actually don’t understand, period. They’re basically Markov chains on steroids. There is no intelligence in this, and in my opinion actual intelligence is a prerequisite for AGI.

Just leaving this here:

https://ai.meta.com/research/publications/large-concept-mode...

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#42

Post by perching_aix »

ornornor wrote: post #38
perching_aix wrote: post #34

Earlier quoted context omitted.

They are capable of extracting arbitrary semantic information and generalize across it. If this is not an understanding, I don't know what is.

To me, understanding the world requires experiencing reality. LLMs dont experience anything. They’re just a program. You can argue that living things are also just following a program but the difference is that they (and I include humans in this) experience reality.

But they're experiencing their training data, their pseudo-randomness source, and your prompts?

Like, to put it in perspective. Suppose you're training a multimodal model. Training data on the terabyte scale. Training time on the weeks scale. Let's be optimistic and assume 10 TB in just a week: that is 16.5 MB/s of avg throughput.

Compare this to the human experience. VR headsets are aiming for what these days, 4K@120 per eye? 12 GB/s at SDR, and that's just vision.

We're so far from "realtime" with that optimistic 16.5 MB/s, it's not even funny. Of course the experiencing and understanding that results from this will be vastly different. It's a borderline miracle it's any human-aligned. Well, if we ignore lossy compression and aggressive image and video resizing, that is.

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#43

Post by coldtea »

lyime wrote: post #30
ornornor wrote: post #14

Earlier quoted context omitted.

> It may actually be the final breakthrough we need for AGI. I disagree. As I understand them, LLMs right now don’t understand concepts. They actually don’t understand, period. They’re basically Markov chains on steroids. There is no intelligence in this, and in my opinion actual intelligence is a prerequisite for AGI.

How do you define "LLMs don't understand concepts"? How do you define "understanding a concept" - what do you get if a system can "understand" concept vs not "understanding" a concept?

Didn't Apple had a paper proving this very thing, or at least addressing it?

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#44

Post by coldtea »

sixo wrote: post #16
SweetSoftPillow wrote: post #15

Earlier quoted context omitted.

What is "actual intelligence" and how are you different from a Markov chain?

Roughly, actual intelligence needs to maintain a world model in its internal representation, not merely an embedding of language, which is a very different data structure and probably will be learned in a very different way. This includes things like: - a map of the world, or concept space, or a codebase, etc - causality - "factoring" which breaks down systems or interactions into predictable parts Language alone is…

>Roughly, actual intelligence needs to maintain a world model in its internal representation

And how's that not like stored information (memories) and weighted links between each and/or between groups of them?

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#45

Post by coldtea »

sindercal wrote: post #24
ornornor wrote: post #18

Earlier quoted context omitted.

What I mean is that the current generation of LLMs don’t understand how concepts relate to one another. Which is why they’re so bad at maths for instance. Markov chains can’t deduce anything logically. I can.

You and Chomsky are probably the last 2 persons on earth to believe that.

It wouldn't matter if they are both right. Social truth is not reality, and scientific consensus is not reality either (just a good proxy of "is this true", but its been shown to be wrong many times - at least based on a later consensus, if not objective experiments).

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#46

Post by coldtea »

ForHackernews wrote: post #21
SweetSoftPillow wrote: post #15

Earlier quoted context omitted.

What is "actual intelligence" and how are you different from a Markov chain?

For one thing, I have internal state that continues to exist when I'm not responding to text input; I have some (limited) access to my own internal state and can reason about it (metacognition). So far, LLMs do not, and even when they claim they are, they are hallucinating https://transformer-circuits.pub/2025/attribution-graphs/bio...

>For one thing, I have internal state that continues to exist when I'm not responding to text input

Do you? Or do you just have memory and are run on a short loop?

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#47

Post by LeicaLatte »

Curious about the interaction between this memory behavior and fine-tuning. If the base model has these emergent memory patterns, how do they transfer or adapt when we fine-tune for specific domains?

Has anyone experimented with deliberately structuring prompts to take advantage of these memory patterns?

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#48

Post by Workaccount2 »

ankit219 wrote: post #37

The difference is implementation comes down to business goals more than anything. There is a clear directionality for ChatGPT. At some point they will monetize by ads and affiliate links. Their memory implementation is aimed at creating a user profile. Claude's memory implementation feels more oriented towards the long term goal of accessing abstractions and past interactions. It's very close to how humans access mem…

Don't fool yourself into thinking Anthropic won't be serving up personalized ads too.

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#49

Post by jmcgough »

creata wrote: post #19
ornornor wrote: post #14

Earlier quoted context omitted.

> It may actually be the final breakthrough we need for AGI. I disagree. As I understand them, LLMs right now don’t understand concepts. They actually don’t understand, period. They’re basically Markov chains on steroids. There is no intelligence in this, and in my opinion actual intelligence is a prerequisite for AGI.

> As I understand them, LLMs right now don’t understand concepts. In my uninformed opinion it feels like there's probably some meaningful learned representation of at least common or basic concepts. It just seems like the easiest way for LLMs to perform as well as they do.

Humans assume that being able to produce meaningful language is indicative of intelligence, because the only way to do this until LLMs was through human intelligence.

Re: Claude’s memory architecture is the opposite of ChatGPT’s

#50

Post by lupusreal »

techbruv wrote: post #22
ornornor wrote: post #14

Earlier quoted context omitted.

> It may actually be the final breakthrough we need for AGI. I disagree. As I understand them, LLMs right now don’t understand concepts. They actually don’t understand, period. They’re basically Markov chains on steroids. There is no intelligence in this, and in my opinion actual intelligence is a prerequisite for AGI.

I don’t understand the argument “AI is just XYZ mechanism, therefore it cannot be intelligent”. Does the mechanism really disqualify it from intelligence if behaviorally, you cannot distinguish it from “real” intelligence? I’m not saying that LLMs have certainly surpassed the “cannot distinguish from real intelligence” threshold, but saying there’s not even a little bit of intelligence in a system that can solve more…

What it really boils down to is "the machine doesn't have a soul". Just an unfalsifiable and ultimately meaningless objection.
Post reply on HN