Earlier quoted context omitted.
It blows my mind that I can ask for 50 synonyms, instantly get a great list with great meaning summaries. Then ask for the same list sorted and get that nearly instantly, These models have a short time context for now, but they already have a huge “working memory” relative to us . It is very cool. And indicative that vastly smarter models are going to be achieved fairly easily, with new insight. Our biology has had t…
> These models [...] have a huge “working memory” relative to us. [This is] indicative that vastly smarter models are going to be achieved fairly easily, with new insight. I don't think your second sentence logically follows from the first. Relative to us, these models: - Have a much larger working memory. - Have much more limited logical reasoning skills. To some extent, these models are able to use their superior w…
Only easily accessible text data. We haven't really started using video at scale yet for example. It looks like data for specific tasks goes really far too ... for example agentic coding interactions aren't something that has generally been captured on the internet. But capturing interactions with coding agents, in combination with the base-training of existing programming knowledge already captured is resulting in significant performance increases. The amount of specicialed data we might need to gather or synthetically generate is perhaps orders of magnitude less that presumed with pure supervised learning systems. And for other applications like industrial automation or robotics we've barely started capturing all the sensor data that lives in those systems.