Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Text classification with Python 3.14's ZSTD module

maxhalford.github.io

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Text classification with Python 3.14's ZSTD module

#21

Post by hxtk »

Scene_Cast2 wrote: post #2

Sweet! I love clever information theory things like that. It goes the other way too. Given that LLMs are just lossless compression machines, I do sometimes wonder how much better they are at compressing plain text compared to zstd or similar. Should be easy to calculate... EDIT: lossless when they're used as the probability estimator and paired with something like an arithmetic coder.

I've actually been experimenting with that lately. I did a really naive version that tokenizes the input, feeds the max context window up to the token being encoded into an LLM, and uses that to produce a distribution of likely next tokens, then encodes the actual token with Huffman Coding with the LLM's estimated distribution. I could get better results with arithmetic encoding almost certainly.

It outperforms zstd by a long shot (I haven't dedicated the compute horsepower to figuring out what "a long shot" means quantitatively with reasonably small confidence intervals) on natural language, like wikipedia articles or markdown documents, but (using GPT-2) it's about as good as zstd or worse than zstd on things like files in the Kubernetes source repository.

You already get a significant amount of compression just out of the tokenization in some cases ("The quick red fox jumps over the lazy brown dog." encodes to one token per word plus one token for the '.' for the GPT-2 tokenizer), where as with code a lot of your tokens will just represent a single character so the entropy coding is doing all the work, which means your compression is only as good as the accuracy of your LLM, plus the efficiency of your entropy coding.

I would need to be encoding multiple tokens per "word" with Huffman Coding to hit the entropy bounds, since it has a minimum of one bit per character, so if tokens are mostly just one byte then I can't do better than a 12.5% compression ratio with one token per word. And doing otherwise gets computationally infeasible very fast. Arithmetic coding would do much better especially on code because it can encode a word with fractional bits.

I used Huffman coding for my first attempt because it's easier to implement and most libraries don't support dynamically updating the distribution throughout the process.

Re: Text classification with Python 3.14's ZSTD module

#23

Post by staplung »

This has been possible with the zlib module since 1997 [EDIT: zlib is from '97. The zdict param wasn't added until 2012]. You even get similar byte count outputs to the example and on my machine, it's about 10x faster to use zlib.

  import zlib

  input_text = b"I ordered three tacos with extra guacamole"

  tacos = b"taco burrito tortilla salsa guacamole cilantro lime " * 50
  taco_comp = zlib.compressobj(zdict=tacos)
  print(len(taco_comp.compress(input_text) + taco_comp.flush()))
  # prints 41

  padel = b"racket court serve volley smash lob match game set " * 50
  padel_comp = zlib.compressobj(zdict=padel)
  print(len(padel_comp.compress(input_text) +  padel_comp.flush()))
  # prints 54

Re: Text classification with Python 3.14's ZSTD module

#24

Post by notpushkin »

notpushkin wrote: post #20
pornel wrote: post #16

The application of compressors for text statistics is fun, but it's a software equivalent of discovering that speakers and microphones are in principle the same device. (KL divergence of letter frequencies is the same thing as ratio of lengths of their Huffman-compressed bitstreams, but you don't need to do all this bit-twiddling for real just to count the letters) The article views compression entirely through Pytho…

> The application of compressors for text statistics is fun, but it's a software equivalent of discovering that speakers and microphones are in principle the same device. I think it makes sense to explore it from practical standpoint, too. It’s in Python stdlib, and works reasonably well, so for some applications it might be good enough. It’s also fairly easy to implement in other languages with zstd bindings, or eve…

Or with the newsgroup20 dataset:

  curl http://qwone.com/~jason/20Newsgroups/20news-19997.tar.gz | tar -xzf -
  cd 20_newsgroups
  for f in *; do zstd --train "$f/*" -o "../$f.dict"; done
  cd ..
  for d in *.dict; do
    cat 20_newsgroups/misc.forsale/74150 | zstd -D "$d" | wc -c | tr -d '\n'; echo " $d";
  done | sort | head -n 3
Output:

     422 misc.forsale.dict
     462 rec.autos.dict
     463 comp.sys.mac.hardware.dict
Pretty neat IMO.

Re: Text classification with Python 3.14's ZSTD module

#25

Post by notpushkin »

staplung wrote: post #23

This has been possible with the zlib module since 1997 [EDIT: zlib is from '97. The zdict param wasn't added until 2012]. You even get similar byte count outputs to the example and on my machine, it's about 10x faster to use zlib. import zlib input_text = b"I ordered three tacos with extra guacamole" tacos = b"taco burrito tortilla salsa guacamole cilantro lime " * 50 taco_comp = zlib.compressobj(zdict=tacos) print(l…

True. The post calls out that “you have to recompress the training data for each test document” with zlib (otherwise input_text would taint it), but you can actually call Compress.copy().

zdict was added in Python 3.3, though, so it’s 2012, not 1997. (It might have worked before, just not a part of the official API :-)

Re: Text classification with Python 3.14's ZSTD module

#27

Post by D-Machine »

duskwuff wrote: post #13
ks2048 wrote: post #8

This looks like a nice rundown of how to do this with Python's zstd module. But, I'm skeptical of using compressors directly for ML/AI/etc. (yes, compression and intelligence are very closely related, but practical compressors and practical classifiers have different goals and different practical constraints). Back in 2023, I wrote two blog-posts [0,1] that refused the results in the 2023 paper referenced here (bad i…

Concur. Zstandard is a good compressor, but it's not magical; comparing the compressed size of Zstd(A+B) to the common size of Zstd(A) + Zstd(B) is effectively just a complicated way of measuring how many words and phrases the two documents have in common. Which isn't entirely ineffective at judging whether they're about the same topic, but it's an unnecessarily complex and easily confused way of doing so.

Yup. Data compression ≠ semantic compression.

Re: Text classification with Python 3.14's ZSTD module

#28

Post by D-Machine »

notpushkin wrote: post #15
az09mugen wrote: post #6

Earlier quoted context omitted.

I do not agree on the "lossless" adjective. And even if it is lossless, for sure it is not deterministic. For example I would not want a zip of an encyclopedia that uncompresses to unverified, approximate and sometimes even wrong text. According to this site : https://www.wikiwand.com/en/articles/Size%20of%20Wikipedia a compressed Wikipedia without medias, just text is ~24GB. What's the medium size of an LLM, 10 GB ?…

With a temperature of zero, LLM output will always be the same. Then it becomes a matter of getting it to output the exact replica of the input: if we can do that, it will always produce it, and the fact it can also be used as a bullshit machine becomes irrelevant. With the usual interface it’s probably inefficient: giving just a prompt alone might not produce the output we need, or it might be larger than the thing…

> With a temperature of zero, LLM output will always be the same

Ignoring GPU indeterminism, if you are running a local LLM and control batching, yes.

If you are computing via API / on the cloud, and so being batched with other computations, then no (https://thinkingmachines.ai/blog/defeating-nondeterminism-in...).

But, yes, there is a lot of potential from semantic compression via AI models here, if we just make the efforts.

Re: Text classification with Python 3.14's ZSTD module

#29

Post by staplung »

notpushkin wrote: post #25
staplung wrote: post #23

This has been possible with the zlib module since 1997 [EDIT: zlib is from '97. The zdict param wasn't added until 2012]. You even get similar byte count outputs to the example and on my machine, it's about 10x faster to use zlib. import zlib input_text = b"I ordered three tacos with extra guacamole" tacos = b"taco burrito tortilla salsa guacamole cilantro lime " * 50 taco_comp = zlib.compressobj(zdict=tacos) print(l…

True. The post calls out that “you have to recompress the training data for each test document” with zlib (otherwise input_text would taint it), but you can actually call Compress.copy(). zdict was added in Python 3.3, though, so it’s 2012, not 1997. (It might have worked before, just not a part of the official API :-)

Ah, okay. Didn't realize that. I used either zlib or gzip long, long ago but never messed with the `zdict` param. Thanks for pointing that out.

Re: Text classification with Python 3.14's ZSTD module

#30

Post by D-Machine »

fwip wrote: post #4
Scene_Cast2 wrote: post #2

Sweet! I love clever information theory things like that. It goes the other way too. Given that LLMs are just lossless compression machines, I do sometimes wonder how much better they are at compressing plain text compared to zstd or similar. Should be easy to calculate... EDIT: lossless when they're used as the probability estimator and paired with something like an arithmetic coder.

Aren't LLMs lossy? You could make them lossless by also encoding a diff of the predicted output vs the actual text. Edit to soften a claim I didn't mean to make.

Yes LLMs are always lossy, unless their size / capacity is so huge they can memorize all their inputs. Even if LLMs were not resource-constrained, one would expect lossy compression due to batching and the math of the loss function. Training is such that it is always better for the model to accurately approximate the majority of texts than to approximate any single text with maximum accuracy.
Post reply on HN