Having some artificial intelligence algorithm that is completely understood and tu able like gzip is would be great for many uses.
I think it’s pretty hard to just improve a NN without more data or some non trivial amount of effort.
78% MNIST accuracy using GZIP in under 10 lines of code
41–50 of 141 posts
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#42I don't immediately find it, but couple of years back there was a "meta-feature" which was the size of the MNIST image. I think that scored about 90'ish % accurate results on its own - without even looking at the image.
What do you mean by “size”? Gzipped size? If you simply look at how dark a Mnist image is (count the percentage of dark pixels) you’ll get about 20% accuracy, which is twice better than random guess but a long way from 90’ish %.
What do you mean with accuracy here?
Usually 50% accuracy means cointoss, meaning 20% accuracy is equal to 80% accuracy, which is better than the article's 78% and not that far from 90%.
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#43In fairness you can run MNIST through UMAP and get near perfect seperation. I'm of the belief that you have to try pretty hard not to do well on MNIST these days. https://github.com/lmcinnes/umap_paper_notebooks/blob/master... EDIT: I should add, unless it isn't clear, that we really should retire the dataset. Something like the QuickDraw dataset makes a lot more sense to me.
> I should add, unless it isn't clear, that we really should retire the dataset.
From a research perspective, MNIST is basically solved. I think current performance on MNIST is better than humans, correct?
But it's still valuable as a teaching tool or a "Hello world" data set, because it's so simple and because most reasonable algorithms will reach 97% accuracy. Even if you build your tools from scratch, it makes a nice homework-sized problem. And it's an obviously useful task that anyone can understand. "Oh, digit recognition for mail!"
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#44Obviously, the code may be elegant and compact, 78% accuracy is considered very very bad for MNIST. A dummy model written with Tensorflow easilly reaches 90% accuracy. The best models ranked at 99,87%, see the benchmark : https://paperswithcode.com/sota/image-classification-on-mnis...
The article emphasizes the wrong thing, in my view. The interesting part is that compression -- without learning a model -- can be used for classification. This raises the question of what other information-theoretic measures can be used; cheaper, lossy ones.
To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review https://arxiv.org/abs/2304.09355\*
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#45Earlier quoted context omitted.
What do you mean by “size”? Gzipped size? If you simply look at how dark a Mnist image is (count the percentage of dark pixels) you’ll get about 20% accuracy, which is twice better than random guess but a long way from 90’ish %.
What do you mean with accuracy here? Usually 50% accuracy means cointoss, meaning 20% accuracy is equal to 80% accuracy, which is better than the article's 78% and not that far from 90%.
There are ten choices, so getting the answer right 20% of the time is very plausible.
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#46Earlier quoted context omitted.
What do you mean by “size”? Gzipped size? If you simply look at how dark a Mnist image is (count the percentage of dark pixels) you’ll get about 20% accuracy, which is twice better than random guess but a long way from 90’ish %.
What do you mean with accuracy here? Usually 50% accuracy means cointoss, meaning 20% accuracy is equal to 80% accuracy, which is better than the article's 78% and not that far from 90%.
There are 10 classes in Mnist. Random guess would is 10%.
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#47In fairness you can run MNIST through UMAP and get near perfect seperation. I'm of the belief that you have to try pretty hard not to do well on MNIST these days. https://github.com/lmcinnes/umap_paper_notebooks/blob/master... EDIT: I should add, unless it isn't clear, that we really should retire the dataset. Something like the QuickDraw dataset makes a lot more sense to me.
> I should add, unless it isn't clear, that we really should retire the dataset. From a research perspective, MNIST is basically solved. I think current performance on MNIST is better than humans, correct? But it's still valuable as a teaching tool or a "Hello world" data set, because it's so simple and because most reasonable algorithms will reach 97% accuracy. Even if you build your tools from scratch, it makes a n…
I was thinking about this just now... the real utility of MNIST is that, since it's basically impossible to get a bad result, it's a good sanity check.
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#48It goes along the same lines as the recent paper from Deepmind stating that DL (language modeling in their case) is compression.
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#49Obviously, the code may be elegant and compact, 78% accuracy is considered very very bad for MNIST. A dummy model written with Tensorflow easilly reaches 90% accuracy. The best models ranked at 99,87%, see the benchmark : https://paperswithcode.com/sota/image-classification-on-mnis...
It's not trying to break records, it just shows a neat aspect of compression. It's still 8 times better than baseline, which showcases that compression can learn representation.
Re: 78% MNIST accuracy using GZIP in under 10 lines of code
#50I don't immediately find it, but couple of years back there was a "meta-feature" which was the size of the MNIST image. I think that scored about 90'ish % accurate results on its own - without even looking at the image.
A few years back I worked on a project that involved fingerprinting screenshots of web pages, and compressed image size was pretty much as good as any fingerprinting method we could come up with for comparing the similarity between them.