Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Text classification with Python 3.14's ZSTD module

maxhalford.github.io

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Text classification with Python 3.14's ZSTD module

#31

Post by stephantul »

The speed comparison is weird.

The author sets the solver to saga, doesn’t standardize the features, and uses a very high max_iter.

Logistic Regression takes longer to converge when features are not standardized.

Also, the zstd classifier time complexity scales linearly with the number of classes, logistic regression doesn’t. You have 20 (it’s in the name of the dataset), so why only use 4.

It’s a cool exploration of zstd. But please give the baseline some love. Not everything has to be better than something to be interesting.

Re: Text classification with Python 3.14's ZSTD module

#32

Post by physicsguy »

In my PhD more than a decade ago, I ended up using png image file sizes to classify different output states from simulations of a system under different conditions. Because of the compressions, homogenous states led to much smaller file size than the heterogenous states. It was super super reliable.

Re: Text classification with Python 3.14's ZSTD module

#34

Post by Pedro_Ribeiro »

matheus-rr wrote: post #26

[flagged]

Is this an AI response? This account was created 4 days ago and all its comments follow the exact same structure. The comments are surprisingly not easy to tell it's AI but it always makes sure to include a "it's X, not Y" conclusion.

Re: Text classification with Python 3.14's ZSTD module

#36

Post by srean »

duskwuff wrote: post #13
ks2048 wrote: post #8

This looks like a nice rundown of how to do this with Python's zstd module. But, I'm skeptical of using compressors directly for ML/AI/etc. (yes, compression and intelligence are very closely related, but practical compressors and practical classifiers have different goals and different practical constraints). Back in 2023, I wrote two blog-posts [0,1] that refused the results in the 2023 paper referenced here (bad i…

Concur. Zstandard is a good compressor, but it's not magical; comparing the compressed size of Zstd(A+B) to the common size of Zstd(A) + Zstd(B) is effectively just a complicated way of measuring how many words and phrases the two documents have in common. Which isn't entirely ineffective at judging whether they're about the same topic, but it's an unnecessarily complex and easily confused way of doing so.

I do not know inner details of Zstandard, but I would expect that it to least do suffix/prefix stats or word fragment stats, not just words and phrases.

Re: Text classification with Python 3.14's ZSTD module

#39

Post by duskwuff »

srean wrote: post #36
duskwuff wrote: post #13

Earlier quoted context omitted.

Concur. Zstandard is a good compressor, but it's not magical; comparing the compressed size of Zstd(A+B) to the common size of Zstd(A) + Zstd(B) is effectively just a complicated way of measuring how many words and phrases the two documents have in common. Which isn't entirely ineffective at judging whether they're about the same topic, but it's an unnecessarily complex and easily confused way of doing so.

I do not know inner details of Zstandard, but I would expect that it to least do suffix/prefix stats or word fragment stats, not just words and phrases.

It's not specifically aware of the syntax - it'll match any repeated substrings. That just happens to usually end up meaning words and phrases in English text.
Post reply on HN