Earlier quoted context omitted.
Clearly there's a difference, because the architectures we have don't know how to persist information or further train. Without persistence outside of the context window, they can't even maintain a dynamic, stable higher level goal. Whether you can bolt something small to these architectures for persistence and do some small things and get AGI is an open question, but what we have is clearly insufficient by design. I…
From my experience there's definitely context beyond the current set of LLM state. It's how they're able to regurgitate facts or speak at all.
> regurgitate facts or speak at all.
Most of that is encoded into weights during training, though external function call interfaces and RAG are broadening this.