Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

241–250 of 324 posts

Re: Compression is prediction

#241

Post by detourdog »

pishpash wrote: post #190
Teever wrote: post #172

Earlier quoted context omitted.

What specific historical citations would you like to see in this post? Perhaps you could contact the author with your suggestions?

They did not come up with the ideas themselves, so they got them somewhere. Follow the source and all the citations show up. It must be a modern thing where online blogging randos pretend they are all geniuses.

Better too assume they are just not aware. Technology is multi-layered cake of development. I have no doubt the only reason I know a lot of details is that I lived their development.

When standing on the shoulders of giants it's hard to tell what is below them.

Re: Compression is prediction

#242

Post by 27183 »

jgraham wrote: post #228
farfatched wrote: post #97

Earlier quoted context omitted.

He really was fantastic, and prolific in multiple fields. He wrote https://www.withouthotair.org/ (creative commons) and was the Chief Scientific Advisor to the UK Department of Energy and Climate Change. Dedicated to "to those who will not have the benefit of two billion years' accumulated energy reserves".

I also went to a couple of his (fantastic) undergraduate courses, and have a huge amount of respect for him. That said, I think it's worth mentioning that Climate Change Without the Hot Air has aged pretty badly, and I'd be reluctant to recommend it to people who don't already have the background to understand what's aged well and what hasn't. The high level approach of making high level numerical estimates makes sen…

> biofuels are even more inefficient, and therefore a dead end

Only if energy density doesn't matter. But it really does, though. Battery powered electric trucking? Dead end. Battery powered aviation? Dead end. Battery powered shipping? Dead end.

[edit] Maybe there's some sustainable way to convert solar energy into sufficiently energy dense fuels that isn't biological, but so far it seems like seed oils or algae are probably the least bad?

Re: Compression is prediction

#243

Post by detourdog »

hellohello2 wrote: post #115
chermi wrote: post #48

Earlier quoted context omitted.

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

You're reading this the wrong way I think, citations aren't given because its obviously a pedagogical article about well established stuff. Much like you wouldn't give citations in a blog post explaining calculus.

One could give citations regarding calculus it's pretty interesting. Since it was done twice by both Newton and Liebniz. There must have been cultural developments in the 1660's that demanded calculus be invented.

Re: Compression is prediction

#244

Post by bpicolo »

ziofill wrote: post #240

Compression and error correction also go hand in hand: in compressed data every bit carries more information and therefore errors are more detrimental. This is one of the results that Shannon phrased exactly in terms of entropy. My PhD supervisor had a beautiful example. Take the English message "errors can make messages unreadable" and ‘compress it’ by removing vowels. It’s still intelligible because English has red…

I don't think any of those are readable without the context of your message.

Re: Compression is prediction

#245

Post by znnajdla »

Intuitively, the idea makes sense to me. You can only compress something when you reduce the content to “what matters” in it. And understanding “what matters” is to understand the patterns in the data. Understanding the patterns in the data IS intelligence.

There's an important consequence here which I take as a lesson in life and business: it is worth optimizing a process or a workflow in your life or business even when there’s no obvious economic benefit. Because to optimize it is the only way to truly understand it. I am very wary of businesses and software that don’t optimize for performance (not just for profit) because it signals they don’t understand what they are doing. Slow software is poorly understood software. Fast software is also likely to be bug-free and secure because someone understands it.

Re: Compression is prediction

#246

Post by speedgoose »

adamgordonbell wrote: post #29
andai wrote: post #9

See also: Bellard's Lossless Data Compression With Neural Networks https://news.ycombinator.com/item?id=19589848 https://news.ycombinator.com/item?id=27244004

And also the LLM version, and LLMZip https://bellard.org/ts_zip/ https://arxiv.org/abs/2306.04050

I tried to reproduce those results, at least in terms of compression ratios, not speed.

However I would say that testing on alice29, enwiki8, text8 data is kinda cheating. Alice in Wonderland and Wikipedia are very likely part of the training data of the LLM models used there.

So I tried on HN comments from a few days ago, extracted from the text column of the public HN bigquery dataset.

Using RWKV v7 0.1B instead of RWKV v4, I get 0.962 bits per byte on alice29, and 1.156 bits per bytes on the HN comments. Still a lot better than 2.826 bits per bytes of xz level 9.

Re: Compression is prediction

#247

Post by wpietri »

hnfong wrote: post #177
beagle3 wrote: post #159

Earlier quoted context omitted.

You expect every blog post to find the earliest relevant paper to cite, just so one could look at the year (without reading said paper - which would have made clear that the connection isn’t recent) to assess novelty? I don’t think that’s reasonable. It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: t…

Why would blog posts not be subject to such criticism? Either the author knew of prior work that argues the same thing and they ignored it, or they didn't know. And if one writes a 1000+ word article premised on this idea, wouldn't one be presumed to know at least in which century the idea originated from? Arguably these kind of blog posts should be more subject to such criticisms, because the blog posts purport to "…

> Why would blog posts not be subject to such criticism?

You're asking why casual comments from amateurs made for fun on the internet shouldn't be held to the same standard as those made by funded career academic experts writing for other experts over months and meant as part of the permanent record of the field?

Personally, I think that's a bit like asking why a friend having you over for dinner isn't always an elegant 7-course meal with wine pairings. I guess you can expect that if you want, but to me it sounds like a child expecting to go to Disney every day: ignoring the economic realities of the situation is a recipe for eternal disappointment.

Re: Compression is prediction

#248

Post by Kinrany »

RandomLensman wrote: post #223

If a string produced from random noise gets compressed (because it has invariably some repetitions in it if long enough), is there any prediction? Even getting the probability distributions right doesn't get to any way to reliably to predict the next symbol out of the sample string. Any functions fitted etc. will be incorrect, too.

I believe compressed random noise cannot be shorter in expectation than the original

Re: Compression is prediction

#249

Post by wpietri »

detourdog wrote: post #137
bonoboTP wrote: post #107

Earlier quoted context omitted.

I'm of two minds here. The pro is that the "you could have invented this" walkthrough from first principles is more engaging than "and then so and so introduced this term in 1972 and the definition is such and such". This style is a reaction to that boring and dry teaching style and tries to push towards what eg Feynman pointed at in the Brazil critique. The con is that you don't get to understand and see any of the…

Disconnecting idea development from it's historic development is a disservice to the audience that may want to dig deeper.

If only there were some sort of way for a reader to dig deeper on a topic without a writer having to spoon-feed them the entire history of everything!

It's wild to me what people here expect out of something they got for free and that was offered as a gift.

Re: Compression is prediction

#250

Post by 27183 »

qsera wrote: post #195
sdenton4 wrote: post #163

Earlier quoted context omitted.

Let alone a sack of wet, self replicating protein! Just endless copying... How could it ever do anything more?

This self replicating proteins can do a lot of stuff that a program running inside a computer cannot do. Like producing secretions, generate electricity, generate light, generate force etc etc....

They're also way more robust distributed systems. Imagine a compute substrate that was continuously mutating, dying, regenerating, cancering, being polluted by various environmental and self-generated contaminants (e.g. alcohol, adrenaline) yet all the while still able to function as a coherent whole. I think this is sort of the pipe dream of the Internet as a system that is robust under strategic nuclear attack. But us meatbags actually do it.
Post reply on HN