21–30 of 132 posts
missed opportunity to explain how compression and prediction are related, and that the better you can predict the next token the better your compression gets, then your article gets to mention GPT hey
Apparently compression and intelligence are synonymous: https://mattmahoney.net/dc/rationale.html
Caches drive the internet. They are orders of magnitude more important than compression. 99.99% of requests hit a local cache. (1) Compression is important too. (1) I worked in a telco. Check it out for yourself!
The internet is "driven" by web/mobile which is driven by multimedia, which lives and dies on compression / codecs.
Quanta magazine and all these pop science websites need to be stopped. (Not because popular science is bad, but because they do it badly, and the clickbait is insufferable)
By the way, there was an old Soviet magazine called "Kvant" (Russian for Quantum, I think). I do not know Russian, but I have 2 collected volumes of selected articles from them. [1] [2] Their quality is astonishingly good, and high-level. The difference is this:
The Kvant articles were written by professional research mathematicians, trying to present their ideas to an audience that were willing to follow them with pencil and paper in hand.
Quanta magazine articles are written by journalists trying to present advanced science to a lay audience - the articles are very stilted and present the articles in a tone that oversimplifies the problem and gives no idea about the actual solution, and uses hackneyed tropes like : oh look the solvers were just some random unknown guys (in a recent case, the random unknown guy is a tenured faculty at UCLA in theoretical computer science, apparently "a world away from mathematics" [3])
[1] http://www.iri.upc.edu/people/thomas/Collection/details/5660...
[2] http://www.personal.psu.edu/sot2/kvant_preface1.pdf
[3] https://www.quantamagazine.org/surprise-computer-science-pro...
missed opportunity to explain how compression and prediction are related, and that the better you can predict the next token the better your compression gets, then your article gets to mention GPT hey
Apparently compression and intelligence are synonymous: https://mattmahoney.net/dc/rationale.html
It's basically the curve-fitting problem: You don't want the simplest curve that fits all the available data points perfectly, you want an even simpler curve that still fits the evidence reasonably well. If you hit the right balance between simplicity and fit, you can expect your model to generalize to unseen data, to make successful predictions. That would be intelligence, or some major part of it.
Earlier quoted context omitted.
I don't like that show because whenever I watch it I start imagining a world with inside-out compression and I get sad we'll never have it.
What is inside-out compression?
Earlier quoted context omitted.
> selects the optimal algorithm Here's the catch: how does the "system" know which algorithm would be the best? It could try encoding it with multiple algorithms and see which one is shorter, but that's extra CPU. And the "system" can be called acompression algorithm itself.
I mean yeah that's basically what high compression solutions like paq have done, depending on the compression level desired apply increasingly speculative and computationally intensive models to the block and pick whichever one worked the best.