Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

78% MNIST accuracy using GZIP in under 10 lines of code

jakobs.dev

Post reply on HN
Print view View original

121–130 of 141 posts

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#121

Post by minitech »

TimPC wrote: post #71
GaggiX wrote: post #17

For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic regression (prev assignment): 89 % Naive Bayes (prev assignment): 81 % From this blog page: https://dmkothari.github.io/Machine-Learning-Projects/SVM_wi... Also it seems from reading online articles that people are able to obtain much better results just by using…

The whole point isn't to do better. It's to show that enough information survives compression that you can still get very large signal. Compression is intended to make things harder and still does.

No, that wasn’t the point at all. Compressibility was used to determine similarity.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#122

Post by selimthegrim »

jszymborski wrote: post #37
sundarurfriend wrote: post #32

Earlier quoted context omitted.

Y'all are making the (rude) person below complaining about acronyms look reasonable. The repository doesn't define UMAP either, but if you believe ChatGPT it is: > UMAP, which stands for Uniform Manifold Approximation and Projection, is a dimensionality reduction technique and data visualization method commonly used in machine learning and data analysis.

It's[0] a non-linear dimensionality reduction technique in the vein of t-SNE[1] that is very well known in the field. My two cents is that if your problem can be solved with something like UMAP + kNN[2], then you really shouldn't be using Deep Learning to solve it. [0] https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduc... [1] https://en.wikipedia.org/wiki/T-distributed_stochastic_neigh... [2] https://en.wik…

Doesn’t your choice of initialization really affect the results? Lior Pachter really seems to keep beating that drum

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#123

Post by tysam_and »

sr192 wrote: post #118
montebicyclelo wrote: post #99

I tried replacing the distance function in the code with some simpler distance measures: Gzip distance: ~3 minutes, 78% accuracy Euclidean distance: ~0.5 seconds, 93% accuracy Jaccard distance * : ~0.7 seconds, 94% accuracy Dice dissimilarity * : ~0.8 seconds, 94% accuracy * after binarising the images So, as a distance measure for classifying MNIST digits, GZIP has lower accuracy, and is much more computationally de…

ben recht's kernel method implementation in 10 lines hits 98% https://github.com/benjamin-recht/mnist_1_pt_2/tree/main

Great link, thank you so much, lol.

This line freaking killed me:

https://github.com/benjamin-recht/mnist_1_pt_2/blob/c0fe96bc...

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#124

Post by tysam_and »

TimPC wrote: post #69
IKantRead wrote: post #66

Earlier quoted context omitted.

It's a good benchmark because it's so trivial. Sure it's not great at differentiating between SotA techniques, but it's very useful for sanity checks like this one. Even for SotA models, it's still useful to verify that you can get greater than 98% accuracy on MNIST, before exploring larger, more complex bench marks. It certainly shouldn't be the only benchmark but it's a great place to start iterating on ideas.

It's a bad benchmark because it's artificially clean. It's effectively a 2d dataset with no occlusions. So nearly everything you try on it will work, and many things you try on it won't scale to typical image problems. There are good 3d datasets with more realistic examples that are still fairly simplistic compared to the state of the art large datasets, but at least give you signal that your technique is robust to c…

That's what makes it a good benchmark.

It's a benchmark.

Not a real world problem.

That's why the traditional path has been MNIST -> CIFAR10 (optionally -> CIFAR100) -> ImageNet -> ????!?.

Because it gets gradually more complicated.

Your iteration time is the constraint to development progress.

Keep that down, and the bugs from your initial implementation will be significantly less impactful.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#125

Post by IcyWindows »

burnished wrote: post #82
Detrytus wrote: post #74

Earlier quoted context omitted.

What's the average human performance for this task?

I don't know off hand, but go take a look at the images - I would expect near 100%.

Sadly,there are several errors in the labeled data, so no one should get 100%.

See https://labelerrors.com/

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#126

Post by yieldcrv »

I want someone to show me how to use compression to compile exploits on emulated logic gates, like NSOGroup keeps doing

Seems like a sufficiently novel and important advancement that should be taught in universities at this point, since we need to harden software around this kind of possibility.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#127

Post by SomewhatLikely »

Wouldn't it make more sense to create ten streams of images of the same class and then see which one results in the smallest compressed size for a test image? That is, if I gzip a hundred '6's plus my test image and get a compressed size for my test image of 10 bytes, but doing the same for other digits gives me say 15 bytes then I conclude the test image is a '6'.

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#128

Post by jszymborski »

selimthegrim wrote: post #122
jszymborski wrote: post #37

Earlier quoted context omitted.

It's[0] a non-linear dimensionality reduction technique in the vein of t-SNE[1] that is very well known in the field. My two cents is that if your problem can be solved with something like UMAP + kNN[2], then you really shouldn't be using Deep Learning to solve it. [0] https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduc... [1] https://en.wikipedia.org/wiki/T-distributed_stochastic_neigh... [2] https://en.wik…

Doesn’t your choice of initialization really affect the results? Lior Pachter really seems to keep beating that drum

There's some work[0] that argues that UMAP's performance relative to t-SNE is not that great when you control for the initialization.

I'm not sure where I sit on that, but regardless, my understanding is that getting UMAP to separate MNIST doesnt really require you to turn knobs.

[0] https://www.biorxiv.org/content/10.1101/2019.12.19.877522v1

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#129

Post by 6510 »

anitil wrote: post #113
esafak wrote: post #44

Earlier quoted context omitted.

The article emphasizes the wrong thing, in my view. The interesting part is that compression -- without learning a model -- can be used for classification. This raises the question of what other information-theoretic measures can be used; cheaper, lossy ones. To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review https://arxiv.org/abs/2304.09355\ *

I remember seeing an example of using zip to classify languages. You take a set of documents of equal size where you know the languages, then individually concatenate and zip them with the unknown text. The smallest compressed output is likely to be the target language. I can't find the original blog, but there's a note about it here - https://stackoverflow.com/questions/39142778/how-to-determin...

How much better would that get if you append all but one of the equal size documents? (or other combinations like 2 of the top results after using a single one)

Re: 78% MNIST accuracy using GZIP in under 10 lines of code

#130

Post by vintermann »

minitech wrote: post #121
TimPC wrote: post #71

Earlier quoted context omitted.

The whole point isn't to do better. It's to show that enough information survives compression that you can still get very large signal. Compression is intended to make things harder and still does.

No, that wasn’t the point at all. Compressibility was used to determine similarity.

And what it accidentally showed, was that NCD between individual digits in the training set is a really terrible distance metric for classification.

You can do classification with KNN, which is obvious. You can also do classification with compression, which is less obvious, and neat. This approach tries to combine them in a way which doesn't work.

Post reply on HN