Live data from Hacker News

The $5000 Compression Challenge (2001)

patrickcraig.co.uk

11–20 of 192 posts

Re: The $5000 Compression Challenge (2001)

#11

This guy clearly failed because he didn't actually do any compression, he just ab-used the filesystem to store parts of the data and then tried to argue that metadata was not data... But FYI someone else actually managed to compress that exact same data: https://jasp.net/tjnn/2018/1.xhtml

That argument was made, and addressed in the emails. How are you defining compression? Actual compassion doesn't seem to have been a requirement in the challenge.

Re: The $5000 Compression Challenge (2001)

#12

> It's not my fault that a file system uses up more space storing the same amount of data in two files rather than a single file. It's not interesting to see people play word games with each other in the HN comments, so why is it interesting to see this kind of disingenuous behavior? Trolls have always existed. I don't get what's interesting about it.

The act of offering a cash prize to prove you wrong kind of makes you look like a jerk. It's assumed that you will use every loop hole at your disposal. Seeing the situation reversed so elegantly makes me root for the challenger.

Re: The $5000 Compression Challenge (2001)

#13

> It's not my fault that a file system uses up more space storing the same amount of data in two files rather than a single file. It's not interesting to see people play word games with each other in the HN comments, so why is it interesting to see this kind of disingenuous behavior? Trolls have always existed. I don't get what's interesting about it.

Who's the bigger troll?

Putting up the initial challenge, especially with the accompanying commentary, is just as much trolling as that.

Re: The $5000 Compression Challenge (2001)

#14
post #9

I wonder if it would have been possible to win the challenge legitimately? If a randomly-generated file happened to contain some redundancy through sheer chance, you could hand-craft a compressor to take advantage of it. This compressor would not work in general for random data, but it could work for this one particular case. It's a bet worth taking, because the payoff, 50:1 ($5,000 to $100), is pretty good. Play the…

I think you can make some argument about why this isn't possible at 50:1 odds.

A plausible "decompressor" is at least, say, 30 or 100 bytes, so the random file needs to have 30 bytes less entropy than you expected, which happens with probability X where X This argument could do with more rigor, but I think it's correct. Give me 100 million to 1 odds, though, and I'll take my chances trying to brute force a compressor.

Re: The $5000 Compression Challenge (2001)

#15
Take random bytes from /dev/urandom, encrypt it. There is no way any compressor can reduce the size of that file. I'm pretty sure there is no way the second player can win this game. Actually, the encryption part isn't even necessary. There is no compressor that can reduce the size of a random stream of bytes.

Re: The $5000 Compression Challenge (2001)

#16

This guy clearly failed because he didn't actually do any compression, he just ab-used the filesystem to store parts of the data and then tried to argue that metadata was not data... But FYI someone else actually managed to compress that exact same data: https://jasp.net/tjnn/2018/1.xhtml

Is there any more information about this solution?

Re: The $5000 Compression Challenge (2001)

#17
The original email thread was from 2001, and it gets posted to HN periodically:

https://news.ycombinator.com/from?site=patrickcraig.co.uk

For another compression challenge that is still ongoing, try "500000€ Prize for Compressing Human Knowledge" (also known as "Hutter Prize"):

http://prize.hutter1.net/

https://news.ycombinator.com/item?id=37502329 - Hutter Prize for compressing human knowledge (2023-09-13, 215 comments)

Re: The $5000 Compression Challenge (2001)

#19
suppose the data were generated by a csprng with 256 bits of state. then in fact there are just slightly more than 256 bits of entropy there (prng state plus the length of the generated file). only, you'd have a snail's chance in hell of actually finding that seed

Re: The $5000 Compression Challenge (2001)

#20
post #9

I wonder if it would have been possible to win the challenge legitimately? If a randomly-generated file happened to contain some redundancy through sheer chance, you could hand-craft a compressor to take advantage of it. This compressor would not work in general for random data, but it could work for this one particular case. It's a bet worth taking, because the payoff, 50:1 ($5,000 to $100), is pretty good. Play the…

What if we didn't even attempt to compress the file? Theoretically, there is a random number generator + seed combination that would output the same bytes as the source file.
Post reply on HN