Viewing profile — klauspost

Available via cli and Go API

https://github.com/klauspost/stdgozstd

About the versioning, glad you spotted it anyway. There isn't as much use of the gzhttp package compared to the other ones, so the bar is a bit higher for that one. Also making goo…

The main reason I took the effort to write the spec was to encourage ports - and commit to the format. I've converted a basic block encoder/decoder to C (see end of README for link…

Writeup on design and changes: https://gist.github.com/klauspost/a25b66198cdbdf7b5b224f670c... Block and Stream Format: https://raw.githubusercontent.com/minio/minlz/refs/heads/mai…

I just released about 2 years of work on improving compression with a fixed encoding LZ77 style compressor. Our goal was to improve compression by combining and tweaking the best a…

In my experience, the hybrid approach is practically better due to cache locality anyway, so I don't think even native GF16 or GF32 would be of much help. FWIW, I've ported rs-leop…

Almost all (x86) CPUs sold have GFNI. That can pretty much saturate memory bandwidth on a single core or two. You can use SSSE3 pshufb for the rest which is about half the speed. A…

Looking at it quickly, it seems like you just moving entropy to the frequency table. If the frequency table isn't included in the count, I could just make an "infinite" compressor …

> If I could change one thing about S3's API I would like an option to read the metadata with the listings. Agree. In MinIO (disclaimer: I work there) we added a "secret" parameter…

Repo: https://github.com/mxmlnkn/rapidgzip As someone who has dabbled quite a lot with deflate this was a very interesting find. It seems like the format that never wants to die. F…

Last I checked QAT was limited to 64KB backreferences and independent 128KB blocks. Now you know why they are comparing 16KB payloads only.

.. for small payloads. QAT cannot compress across blocks. So as soon as your input is more than 128KB the compression ratio tanks. Usually well below what even level 1 of software …

This could be very useful for a project I am just starting on. No documentation, and example has no content makes the learning curve a bit steep. Does anyone have any pointers on h…

Reading it, it seems very similar to the Playstation 3 Cell, with its fate mirroring it very much. A highly specialized processor that has very high computational throughput for sp…

You can use presigned urls for that.

Leopard-RS GF16 support added to reedsolomon package, with SSE3, AVX2 and AVX512 assembly. This allows to encode up to 65536 data+parity shards at O(K Log M), where K is the origin…

> If you see a match that would not be visible from given compression level[...] Yes, but you'd need to reconstruct the hash table and the link table. For that you'd need to decomp…

There is no way to "rule some levels out". The levels only differ in the quality of the lz matches. Slower levels search further potential matches - you cannot see that from a comp…

A block can end at any time and another huffman block with another table can start. There is no way to know that without decoding all symbols. Decoding the huffman codes is by far …