Earlier quoted context omitted.
this works fine for like 2-3 small instruction sets. once you start getting to scale of a real enterprise system, the AI falls down and can't handle that amount of context. It will start ignoring critical pieces or not remember them. And without constant review AI will start priotizing things that are not your business priority. I don't agree they have solved this problem, at all, or really in any way that's actually…
What I'm saying is, don't get to thinking that the memory problem is some kind of insurmountable, permanent barrier that's going to keep us safe. It's already being addressed, maybe crudely at first, but the situation is already much better than it was - I no longer have to bring the model up to speed completely every time I start a new session. Part of this is much larger context windows (1M tokens now). New archite…
Re: I'm not worried about AI job loss
#491Context windows are a natural improvement, but new architectures are completely speculative and it’s unclear we can make any sort of predictable progress with new, better architectures. Most progress has been made on essentially the same architecture paradigms, although we did move from dense models to MoE at some point.