141–150 of 166 posts
Earlier quoted context omitted.
Presumably the ownership of the repo was transfered from Cyan4973, the main contributor, to facebook, and github automatically redirects.
yep, looks like it. Maybe just forgot to update the gh-pages branch.
Earlier quoted context omitted.
I don't see the problem: only 3 members of that namespace are currently claimed (4 out of the 36-member namespace: 7z), so we have room for 23 (or 32) more compression standards before running out. We've been using gzip for, what, 20 years now? Only recently have we gotten xz. At this rate, we won't run out of compression standards using this scheme for roughly 153 years. And after that, we could always start using c…
Perhaps we should require the less common algorithms to have a longer prefix.
Earlier quoted context omitted.
I thought that brotli was tuned for typical web workloads, that it contained a dictionary tuned for web workloads. Our internal testing shows that it performs very poorly for binary 3D vector data. So a test between zstd and brotli would show brotli in a poor light if it used a mixed corpus, but a test between zstd and brotli on a web corpus would give an advantage to brotli...
Not just tuned for web workloads in general, but for specific web workloads. The Brotli dictionary is mostly composed of English words and phrases, and fragments of HTML, CSS, and Javascript. It would perform poorly on non-English text. I have a feeling that the dictionary was designed with the specific goal of performing well on a specific corpus similar to the Large Text Compression Benchmark[1]. It has quite a few…
The plot of compression ratio against speed for the various compression levels is pretty helpful for understanding its performance: https://scontent.fsnc1-3.fna.fbcdn.net/t39.2365-6/14146892_9... "The x-axis is a decreasing logarithmic scale in megabytes per second; the y-axis is the compression ratio achieved." I'd love to see a version of this chart that also included Brotli. (And I'm somewhat surprised Brotli isn'…
Earlier quoted context omitted.
I thought that brotli was tuned for typical web workloads, that it contained a dictionary tuned for web workloads. Our internal testing shows that it performs very poorly for binary 3D vector data. So a test between zstd and brotli would show brotli in a poor light if it used a mixed corpus, but a test between zstd and brotli on a web corpus would give an advantage to brotli...
>Our internal testing shows that it performs very poorly for binary 3D vector data. Uh ? What format of data was this ? I did a pretty large test (2gb+) on OBJ/STL 3d data and brotli compressed within ~5% margin of lzma, and this holds true on other binary data I've compared. It also compressed better than zstd (as in compression ratio) on the same data at their highest respective compression settings: bro -quality 1…
Earlier quoted context omitted.
That was my first thought, too. I installed and ran both against a tar'd set of PDF files totaling 435MB in size. My timings: lzfse 45 MB/s encode, 229 MB/s decode, 1.12 comp ratio zstd 181 MB/s encode, 713 MB/s decode, 1.13 comp ratio The numbers are so dramatically different that I ran several different tests, but those results showed the same rough results. I used default command-line options for both tools, and b…
I use "fastest to complete == least power usage" as a rule of thumb because of "race to sleep". I suppose that might be thrown off by power usage characteristics varying based on number of cores working? How does one even begin to write code that prioritizes power-efficiency over performance?
As for how to do it, I'm not sure... but if I were a valued customer of various CPU suppliers, and were famous for the depth of my pockets, I'm sure I'd be able to find somebody to explain it to me ;)
From the bits of testing I've done today, it's phenomenally fast on x86. Much better than gzip (and pigz for that matter) in every metric I think I generally care about: CPU Usage, Compression Speed, Decompression Speed, Compression Ratio. On other architecture the picture gets a bit murky, it seems to get handily beaten by pigz through what at first blush I'd guess is just sheer parallelism. It's got solid performan…
In particular note the huge difference in branches between gzip and zstd on decompress:
8959780663 branches # 143.024 M/sec
2969481781 branches # 64.454 M/sec
and on misses:
542158823 branch-misses # 6.05% of all branches
89060880 branch-misses # 3.00% of all branches
Earlier quoted context omitted.
Curious what your issue is with it -- it basically says "if you dont sue us, we wont sue you". Thats about as good as I can expect from a large tech company these days with regards to patents.
Compare it to the Opus patent license (also including a retaliation clause), which includes grants from Broadcom, Mozilla and Microsoft. Using zstd gives Facebook a free license on ALL your patents. Using Opus gives Facebook only a license on patents that apply to Opus. So no, large tech companies can and have given MUCH better grants for compression tech, than Facebook is doing.
Earlier quoted context omitted.
Curious what your issue is with it -- it basically says "if you dont sue us, we wont sue you". Thats about as good as I can expect from a large tech company these days with regards to patents.
Compare it to the Opus patent license (also including a retaliation clause), which includes grants from Broadcom, Mozilla and Microsoft. Using zstd gives Facebook a free license on ALL your patents. Using Opus gives Facebook only a license on patents that apply to Opus. So no, large tech companies can and have given MUCH better grants for compression tech, than Facebook is doing.
I just read the Opus patent summary. It seems like if zstd followed the same license using it wouldn't give facebook any license but if I sue facebook I loose the licese to use zstd. Am I correct in that.
Earlier quoted context omitted.
Perhaps we should require the less common algorithms to have a longer prefix.
There's no way to know what's going to be a common algorithm in the future unless you have a time machine. When DEFLATE was first invented, it wasn't common either, it was brand-new. Now it's everywhere. This new algorithm might become just as ubiquitous in 10 years, or it might turn into the next bzip2, or worse, the next ZOO.
(I am not sure if i need to clarify that my original comment was meant to be a joke about Huffman coding)