Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

78% MNIST accuracy using GZIP in under 10 lines of code

jakobs.dev

Post reply on HN
Print view View original

71–80 of 141 posts

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#71

Post by TimPC »

GaggiX wrote: post #17

For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic regression (prev assignment): 89 % Naive Bayes (prev assignment): 81 % From this blog page: https://dmkothari.github.io/Machine-Learning-Projects/SVM_wi... Also it seems from reading online articles that people are able to obtain much better results just by using…

The whole point isn't to do better. It's to show that enough information survives compression that you can still get very large signal. Compression is intended to make things harder and still does.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#72

Post by nextos »

IKantRead wrote: post #51
GaggiX wrote: post #17

For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic regression (prev assignment): 89 % Naive Bayes (prev assignment): 81 % From this blog page: https://dmkothari.github.io/Machine-Learning-Projects/SVM_wi... Also it seems from reading online articles that people are able to obtain much better results just by using…

Thanks for posting this. Most people don't realize that Logistic regression can get ~90% accuracy on MNIST. As a big fan of starting with simple models first and adding complexity later, I've frequently been told that "logistic regression won't work!" for problems where it can in fact perform excellent. When faced with this resistance to logistic regression I'll often ask what they think the baseline performance of i…

Came here to say the same thing, actually NCD can probably do much better than 78%. Li & Vitanyi's book about Kolmogorov complexity has some interesting unsupervised examples.

A simple CNN as implemented in Keras tutorial can easily exceed 98%. 78% is very poor performance for MNIST even if model complexity is penalized.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#73

Post by a_wild_dandan »

esafak wrote: post #44
tdr2d wrote: post #11

Obviously, the code may be elegant and compact, 78% accuracy is considered very very bad for MNIST. A dummy model written with Tensorflow easilly reaches 90% accuracy. The best models ranked at 99,87%, see the benchmark : https://paperswithcode.com/sota/image-classification-on-mnis...

The article emphasizes the wrong thing, in my view. The interesting part is that compression -- without learning a model -- can be used for classification. This raises the question of what other information-theoretic measures can be used; cheaper, lossy ones. To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review https://arxiv.org/abs/2304.09355\ *

An increasingly common refrain in machine learning is “intelligence is compression.” Folks who believe that might bristle at the distinction between learning and compression.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#74

Post by Detrytus »

simonw wrote: post #65
_a_a_a_ wrote: post #60

"MNIST"? accuracy - but of what? what's this about?

MNIST is a classic image classification exercise - a dataset of 60,000 training images and 10,000 testing images where each image is a handwritten numeral as a 28x28 pixel grayscale image. The challenge is to build a computer vision model that can tell which numeral each handwritten digit represents. https://en.wikipedia.org/wiki/MNIST_database 78% accuracy on a solution is pretty bad, but achieving it just using GZI…

What's the average human performance for this task?

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#75

Post by jdthedisciple »

tysam_and wrote: post #61

Additionally, try flipping your images and averaging the size before comparing the distance between them. I'd expect a boost of about 78% -> 84% or so, based on how this typically works as TTA.

Can you elaborate more on that technique?

Why would it improve the result so much? Sounds very interesting so I'm curious.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#76

Post by jp57 »

Leaving aside whether this problem is a good application of this compression trick, I want to say that everyone experimenting with this should stop using `gzip` and start using `zlib`.

If you change the first line from `gzip.compress` to `zlib.compress` you should get the same classifier performance with a 3x speedup.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#77

Post by krick »

jszymborski wrote: post #25

In fairness you can run MNIST through UMAP and get near perfect seperation. I'm of the belief that you have to try pretty hard not to do well on MNIST these days. https://github.com/lmcinnes/umap_paper_notebooks/blob/master... EDIT: I should add, unless it isn't clear, that we really should retire the dataset. Something like the QuickDraw dataset makes a lot more sense to me.

Yeah, but the thing is, gzip isn't "these days". It was a thing long before UMAP and even MNIST itself. And the approach isn't perticulary novel too, this is a very simple idea, if you do understand compression. It could've been written the first day MNIST was published, and it would be still 78% accuracy. This is kinda amazing to me.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#78

Post by tysam_and »

jdthedisciple wrote: post #75
tysam_and wrote: post #61

Additionally, try flipping your images and averaging the size before comparing the distance between them. I'd expect a boost of about 78% -> 84% or so, based on how this typically works as TTA.

Can you elaborate more on that technique? Why would it improve the result so much? Sounds very interesting so I'm curious.

facepalm I just realized I was talking about this in the context of MNIST -- my apologies.

It would be better in a problem with horizontal symmetry like CIFAR, esp if the zipping is serialized by unwinding the 2d pixel array into 1d.

One trick that _should_ work is by comparing the distances of starting the compression at the 4 different corners of the image, in the 2 separate directions for each corner. That should provide way more than enough information for k-means clustering.

My apologies again for my mistake, thank you for asking, I wouldn't have really seen that otherwise :')))).

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#79

Post by GaggiX »

TimPC wrote: post #71
GaggiX wrote: post #17

For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic regression (prev assignment): 89 % Naive Bayes (prev assignment): 81 % From this blog page: https://dmkothari.github.io/Machine-Learning-Projects/SVM_wi... Also it seems from reading online articles that people are able to obtain much better results just by using…

The whole point isn't to do better. It's to show that enough information survives compression that you can still get very large signal. Compression is intended to make things harder and still does.

The article OP linked is inspired by Gzip+Knn paper that was released a few months ago, compression is not intended to make things harder but to extract useful features that can be used by simple algorithms like Knn or SVG etc...

For example it would be almost impossible to distinguish cat and dog images by using SVG or Knn directly on the data but it would be much easier if the images were first passed into an image encoder and a small embeddings vector would be used for each one instead. In the article OP linked it doesn't seem that Gzip is very good at extracting useful patterns for the MNIST dataset.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#80

Post by amelius »

TimPC wrote: post #71
GaggiX wrote: post #17

For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic regression (prev assignment): 89 % Naive Bayes (prev assignment): 81 % From this blog page: https://dmkothari.github.io/Machine-Learning-Projects/SVM_wi... Also it seems from reading online articles that people are able to obtain much better results just by using…

The whole point isn't to do better. It's to show that enough information survives compression that you can still get very large signal. Compression is intended to make things harder and still does.

Why? I mean it's trivial to add a layer of encryption ...

(Also the terminology seems off here, as 100% of information survives gzip.)

Post reply on HN