Text Compression as a Test for Artificial Intelligence (1999) [pdf]
1–10 of 17 posts
Re: Text Compression as a Test for Artificial Intelligence (1999) [pdf]
#2This article doesn't mention the Hutter Prize of course, since it was started a year later, but it looks like Matt has written about it since then: http://mattmahoney.net/dc/rationale.html
Right now the biggest thing holding Hutter Prize back is the shamefully low compute and memory limits: "Restrictions: Must run in ≲50 hours using a single CPU core and http://prize.hutter1.net/
To even begin approaching intelligence, the compute and memory limits probably need to be 100x or 1000x larger.
Re: Text Compression as a Test for Artificial Intelligence (1999) [pdf]
#3This article doesn't mention the Hutter Prize of course, since it was started a year later, but it looks like Matt has written about it since then: http://mattmahoney.net/dc/rationale.html Right now the biggest thing holding Hutter Prize back is the shamefully low compute and memory limits: "Restrictions: Must run in ≲50 hours using a single CPU core and http://prize.hutter1.net/ To even begin approaching intelligenc…
Strong disagree.
In a completely unrelated field, I'm seeing pretty great results with knowledge distillation to generate AI language models that fit into 20 mio float16 parameters (40MB) and can process 1GB of raw text in roughly 12 hours.
Re: Text Compression as a Test for Artificial Intelligence (1999) [pdf]
#4I wonder how the use of static world-prior information is considered here.
Or said in a different way: What stops me from supplying a hard-coded pre-compressed file to circumvent their RAM and HDD limits?
Re: Text Compression as a Test for Artificial Intelligence (1999) [pdf]
#5I'd be interested to see lossy text compression. That would also give much more scope for AI methods. It turned out that recently Stable Diffusion gives excellent compression
https://pub.towardsai.net/stable-diffusion-based-image-compr...
Funny thing is, on a per pixel basis, Stable Diffusion and jpeg are just as good. Stable diffusion looks much better, though. The reason is that when Stable diffusion lacks the information it just invents what it expects to see there. If you zoom in on the background entire appartement buildings are invented, moved or disappeared.
So, one wonders what GPT-3 could do for text compression. On a per-character basis, I would not expect any miracles. On a generally, the same, kind-of, basis, I'd would expect something special.
Re: Text Compression as a Test for Artificial Intelligence (1999) [pdf]
#6This article doesn't mention the Hutter Prize of course, since it was started a year later, but it looks like Matt has written about it since then: http://mattmahoney.net/dc/rationale.html Right now the biggest thing holding Hutter Prize back is the shamefully low compute and memory limits: "Restrictions: Must run in ≲50 hours using a single CPU core and http://prize.hutter1.net/ To even begin approaching intelligenc…
Strong disagree. In a completely unrelated field, I'm seeing pretty great results with knowledge distillation to generate AI language models that fit into 20 mio float16 parameters (40MB) and can process 1GB of raw text in roughly 12 hours.
12 hours on a single CPU core, or on a GPU?
Re: Text Compression as a Test for Artificial Intelligence (1999) [pdf]
#7I'd be interested to see lossy text compression. That would also give much more scope for AI methods. It turned out that recently Stable Diffusion gives excellent compression https://pub.towardsai.net/stable-diffusion-based-image-compr... Funny thing is, on a per pixel basis, Stable Diffusion and jpeg are just as good. Stable diffusion looks much better, though. The reason is that when Stable diffusion lacks the info…
A lossy text compressor can be converted into a lossless compressor quite easily by encoding the remaining differences between the lossy reproduction and the original text. The more accurate your lossy compressor is, the less additional information you need to encode the differences. You can get even better results if your lossy compressor is probabilistic, and can compute approximate probabilities for different text continuations. GPT-3 is AFAIK probabilistic and should be applicable... maybe someone has already tried?
Re: Text Compression as a Test for Artificial Intelligence (1999) [pdf]
#8I wonder how the use of static world-prior information is considered here. Or said in a different way: What stops me from supplying a hard-coded pre-compressed file to circumvent their RAM and HDD limits?
No post body was provided.
Re: Text Compression as a Test for Artificial Intelligence (1999) [pdf]
#9Earlier quoted context omitted.
Strong disagree. In a completely unrelated field, I'm seeing pretty great results with knowledge distillation to generate AI language models that fit into 20 mio float16 parameters (40MB) and can process 1GB of raw text in roughly 12 hours.
12 hours on a single CPU core, or on a GPU?
12 hours on a Ryzen 5900X with 12 cores. So not quite fast enough yet, but only 3x too slow.
Re: Text Compression as a Test for Artificial Intelligence (1999) [pdf]
#10I'd be interested to see lossy text compression. That would also give much more scope for AI methods. It turned out that recently Stable Diffusion gives excellent compression https://pub.towardsai.net/stable-diffusion-based-image-compr... Funny thing is, on a per pixel basis, Stable Diffusion and jpeg are just as good. Stable diffusion looks much better, though. The reason is that when Stable diffusion lacks the info…
I hope that lossy text compression never becomes popular, it should ideally remain a technological curiosity. Think of all the damage that can be caused by incompetence combined with usage of lossy text compression in the wrong places.