Earlier quoted context omitted.
Its inevitable. You can argue about what's possible right now, but I'm not looking at it from that angle. I think these issues will be solved with time
That belief is at odds with the mechanics of how LLMs work. It's not a question of more effort/investment/compute/whatever, it's just a reality of how the underlying systems work (non-deterministic). If you can find a way to make the context window on the scale of the human brain, you may be able to mostly mitigate this. People want us to be at "Her" levels of AI, but we're at a far earlier stage. We can fake certain…
> If you can find a way to make the context window on the scale of the human brain, you may be able to mostly mitigate this.
Human brains have a much smaller context window than AI do. We can't pay attention to the last 128,000 concepts that filtered past our sensory systems — our conscious considerations are for about seven things.
There's a lot of stuff that we don't yet understand well enough to reproduce with AI, but context length is the wrong criticism for these models.