Earlier quoted context omitted.
We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?
No. But do you need to run every small problem through a 10B-30B model? We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code. Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for th…
Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin.
I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.