I'm merely a hobbyist in this domain but isn't highly compressed data (like encrypted data) also high entropy? If this is finding patterns in the compressed data to figure out which digit the uncompressed data represents, shouldn't those patterns be exploitable for better compression?
78% MNIST accuracy using GZIP in under 10 lines of code
91–100 of 141 posts
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#92For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic regression (prev assignment): 89 % Naive Bayes (prev assignment): 81 % From this blog page: https://dmkothari.github.io/Machine-Learning-Projects/SVM_wi... Also it seems from reading online articles that people are able to obtain much better results just by using…
Thanks for posting this. Most people don't realize that Logistic regression can get ~90% accuracy on MNIST. As a big fan of starting with simple models first and adding complexity later, I've frequently been told that "logistic regression won't work!" for problems where it can in fact perform excellent. When faced with this resistance to logistic regression I'll often ask what they think the baseline performance of i…
Extending existing features with their non-linear mappings would improve logistic regression too, probably to the svc level (rbf or poly kernel is that but implicit).
Linear models are really well researched, and today with the compute and with proper training and data preparation they can easily get to satisfying levels of performance for a variety of tasks.
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#93Obviously, the code may be elegant and compact, 78% accuracy is considered very very bad for MNIST. A dummy model written with Tensorflow easilly reaches 90% accuracy. The best models ranked at 99,87%, see the benchmark : https://paperswithcode.com/sota/image-classification-on-mnis...
The article emphasizes the wrong thing, in my view. The interesting part is that compression -- without learning a model -- can be used for classification. This raises the question of what other information-theoretic measures can be used; cheaper, lossy ones. To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review https://arxiv.org/abs/2304.09355\ *
I mean, they’re using knn underneath. You can apparently get 97% accuracy with normal knn, at n=4 if you compare pixel distance.
So another way to frame this might be that gzip costs a lot of accuracy but may lead to better performance.
https://newpblog.netlify.app/2018-01-24-knn-analysis-on-mnis...
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#94For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic regression (prev assignment): 89 % Naive Bayes (prev assignment): 81 % From this blog page: https://dmkothari.github.io/Machine-Learning-Projects/SVM_wi... Also it seems from reading online articles that people are able to obtain much better results just by using…
Thanks for posting this. Most people don't realize that Logistic regression can get ~90% accuracy on MNIST. As a big fan of starting with simple models first and adding complexity later, I've frequently been told that "logistic regression won't work!" for problems where it can in fact perform excellent. When faced with this resistance to logistic regression I'll often ask what they think the baseline performance of i…
To a large degree this happens because logistic regression is not a sexy approach that one can add to their CV. Everyone wants to solve problems with big, complicated and buzzwordy models, because that sells (and is perhaps more interesting as well).
It's really a tragedy, because so many classic models would work fine for real world applications.
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#95For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic regression (prev assignment): 89 % Naive Bayes (prev assignment): 81 % From this blog page: https://dmkothari.github.io/Machine-Learning-Projects/SVM_wi... Also it seems from reading online articles that people are able to obtain much better results just by using…
Thanks for posting this. Most people don't realize that Logistic regression can get ~90% accuracy on MNIST. As a big fan of starting with simple models first and adding complexity later, I've frequently been told that "logistic regression won't work!" for problems where it can in fact perform excellent. When faced with this resistance to logistic regression I'll often ask what they think the baseline performance of i…
If you stack logistic regression in layers, you get a neural network.
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#96For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic regression (prev assignment): 89 % Naive Bayes (prev assignment): 81 % From this blog page: https://dmkothari.github.io/Machine-Learning-Projects/SVM_wi... Also it seems from reading online articles that people are able to obtain much better results just by using…
Thanks for posting this. Most people don't realize that Logistic regression can get ~90% accuracy on MNIST. As a big fan of starting with simple models first and adding complexity later, I've frequently been told that "logistic regression won't work!" for problems where it can in fact perform excellent. When faced with this resistance to logistic regression I'll often ask what they think the baseline performance of i…
When would a 90% accuracy on a dataset like mnist ever be useful? And I mean useful as in usable for actual products or software. especially considering mnist is more of a toy dataset.
I think that's why machine learning is the way to go for this type of detection, why go with anything else than a CNN (in this case) when it is now trivial to set up and train? Again, unless it's just to mess around with, 90% mnist accuracy is not useful in the real world
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#97I'm merely a hobbyist in this domain but isn't highly compressed data (like encrypted data) also high entropy? If this is finding patterns in the compressed data to figure out which digit the uncompressed data represents, shouldn't those patterns be exploitable for better compression?
The demonstration isn't classifying based on compressed data, rather it's classifying on the compressibility of data. The idea is that "7 7" should be more compressible than "7 3", and similarly raster images of "7 7" should be more compressible than raster images of "7 3".
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#98I'm merely a hobbyist in this domain but isn't highly compressed data (like encrypted data) also high entropy? If this is finding patterns in the compressed data to figure out which digit the uncompressed data represents, shouldn't those patterns be exploitable for better compression?
Encrypted data ideally is incompressible. Incompressibility is a hallmark of efficient cryptographic operations.
See the Wikipedia article on Kolmogorov complexity, which has a short section about compression. https://en.wikipedia.org/wiki/Kolmogorov_complexity#Compress...
Edit: One of my favorite concepts in the domain of compression is the pigeonhole principle, that states that for all compression algorithms, some outputs will be larger than the inputs.
Well designed encrypted payloads may be compressed, but the outputs should on average be larger than the inputs, rendering compression useless thus it is said to be "incompressible".
https://en.wikipedia.org/wiki/Pigeonhole_principle#Uses_and_...
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#99I tried replacing the distance function in the code with some simpler distance measures:
Gzip distance: ~3 minutes, 78% accuracy
Euclidean distance: ~0.5 seconds, 93% accuracy
Jaccard distance * : ~0.7 seconds, 94% accuracy
Dice dissimilarity * : ~0.8 seconds, 94% accuracy
* after binarising the images
I'm not that familiar with how the GZIP algorithm works, it's kind of interesting that it's so much lower. I wonder whether image focused compression algorithms might do better?
(Edit: Btw, I enjoyed the post; it's a creative idea, the writing and code was great, and it's sparked some good discussion. But after having a closer look, I think the baselines above provide some context to the gzip scores.)
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#100Earlier quoted context omitted.
MNIST (and OCR) are acronyms that are so well-known to anyone who has taken any kind of intro to ML class that there's no more need to define them in a short blog post than there would be for us to define HTML. I learned about MNIST in 2008, and I was just taking a class on numerical methods as a physics major in Matlab where MNIST was just one project.
This isn’t a forum about machine learning, though. It’s a general forum of geek news. I could talk to you all day about compression. Gzip plus MNIST rings no bells.
i think the idea is that compression exploits recognized similarities, so that begs the question of if a compressor is actually recognizing things usefully.