If this is true, the hyperscalers are toast

This means that the Frontier labs are under immense pressure to maintain that lead, and could end up in serious trouble if they stumble at all.

The other thing id point out is that a lot of us who are token-sensitive do things like build plans using expensive, smart models, and then execute those plans using cheaper dumber models.

Then there's the fact that we are still in the age of heavily subsidized Frontier subscriptions + tokenmaxxing initiatives from megacorps. Neither of which are sustainable, and will drive more usage to smaller open models once they end.

1. Training a large model with lots of information, then stripping the "useless" information from that model to obtain a small model => nobody has shown this.

2. Training a small model, letting it use a database tool so it scores the same as a large model without database => nobody has shown this.

You can imagine a reasoning model as a huge set of rules that generate the next statement from previous statements (written in context). In that sense, a reasoning model can be compared to a logical theory - you have certain deduction rules which can generate new judgments.

Often, logical theories are structured that the rules are remade into axioms, and the deduction rule is only modus ponens (which corresponds to function application and is a building block of program execution).

In the case of an LLM, the set of rules (or axioms) they have in the theory is quite large, but most likely semantically unsound (with respect to their their own representation of truth) - that's why LLM's make mistakes.

It would be desirable to break the logical theory represented by LLM into a smaller set of axioms, which would:

a) remove rules easily deductible from the smaller core of axioms (for example, LLM doesn't need to remember "Socrates is mortal", as it can derive it from "Socrates is a man" and "all men are mortal")

b) remove rules that have low value (facts that aren't used often or have weak validity) which cause ruleset to become unsound

I suspect that's what SLM distillation is doing, to some extent.

The question is, how far this process can go? I personally believe there is a useful logic for commonsense reasoning that has less than thousand rules (still several orders more than your typical mathematical logic, but orders less than SLMs). These axioms do not contain much facts about the world, but that could be added.

So I believe there is a sweet spot (deductive core, encyclopedic shell) which we have not yet found (it's a little bit more formal language than natural language) but is very efficient for general reasoning.