Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Open-Sources its Compression Algorithm LZFSE

infoq.com

Post reply on HN
Print view View original

131–140 of 219 posts

Re: Apple Open-Sources its Compression Algorithm LZFSE

#131

Post by svckr »

vog wrote: post #111
svckr wrote: post #17

With energy efficiency as a primary goal I was expecting way more use of explicit SIMD instructions. The InfoQ post mentions xcodebuild, but there is also a Makefile. I really appreciate the presence of a no-nonsense Makefile. No autoconf, no pkgconfig, just plain and simple make. Also, because nobody mentioned it: yes, it compiles on Linux out of the box.

> I really appreciate the presence of a no-nonsense Makefile. Indeed, the current version of their Makefile is a great example of how to write a simple yet portable Makefile: https://github.com/lzfse/lzfse/blob/33629bc65f4b356072c9a7d5... > No autoconf, no pkgconfig, just plain and simple make. While I agree with your sentiment, I believe that your statement about pkg-config goes a bit over the top. Yes, the LZFSE pr…

> While I agree with your sentiment, I believe that your statement about pkg-config goes a bit over the top.

Granted, pkgconfig is often a good solution for a hairy problem. I was just so very delighted to see such a simple Makefile :)

Re: Apple Open-Sources its Compression Algorithm LZFSE

#132

Post by jmaygarden »

cmiller1 wrote: post #125
drinchev wrote: post #15

Earlier quoted context omitted.

Excerpt from the link : if (D == D_prev) { if (L == 0) { *q++ = 0xF0 + (x + 3); // XM! } else { *q++ = (L >8 in 0..5 *q++ = (D >> 8) + (L = (1 34) { // Long dist *q++ = (L > 2) + (L

In some of these lines I feel like I'd prefer to see more parenthesis. Sure, I can figure out what is happening here: } else if (D but if it were grouped with a few more parenthesis it would take a little less time for me to grok it.

There is a lot to be said for really knowing the order of operations. I write a lot of C code and this looks very clear to me. More parenthesis would just add noise from my perspective.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#133

Post by dozzie »

mschuster91 wrote: post #121
vog wrote: post #111

Earlier quoted context omitted.

> I really appreciate the presence of a no-nonsense Makefile. Indeed, the current version of their Makefile is a great example of how to write a simple yet portable Makefile: https://github.com/lzfse/lzfse/blob/33629bc65f4b356072c9a7d5... > No autoconf, no pkgconfig, just plain and simple make. While I agree with your sentiment, I believe that your statement about pkg-config goes a bit over the top. Yes, the LZFSE pr…

> Indeed, the current version of their Makefile is a great example of how to write a simple yet portable Makefile: Yet it forgets the MOST important thing: make uninstall. Nothing worse than software where I have to reverse engineer a makefile in order to uninstall!

> Yet it forgets the MOST important thing: make uninstall.

Why it forgets that? It's not the job of a library build system to install or uninstall things in your system. It's a job for your system's package manager.

Makefile that has a target (historically called "install") that puts things in appropriate places in a chroot-like manner is just good enough.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#134

Post by josephg »

randlet wrote: post #97
nicky0 wrote: post #59

Earlier quoted context omitted.

OK but which is more readable ... (I know it's a bit silly, I juts made up names) } else if (D >= (1 34) { } else if (DirectWeightingFactor >= HUYGENS_LIMIT || MariachiBand == 0 || (xylemNonce + STANDARD_PZSH_INCREMENT) + MariachiBand > SWIM_RATE_B) { I guess your opinion differs to mine. I like the one that looks like math.

The more verbose name is infinitely more preferable to me. If I am tasked one day with doing some maintenance on this code and have never encountered it then at least I have a hint that I should be researching the aquatic properties of mariachi bands.

As someone who's written and maintained code like this, in my experience the heavy algorithmic parts themselves usually requires almost no maintenance. Its the surrounding code - the API endpoints, makefiles and test suites that need to be modernized every few years. Algorithms like this are written, then used, then better algorithms are written from scratch.

Code like this also usually can't be edited piecemeal. Any change requires reloading the entire algorithm into your head before a single line is changed. And as others have mentioned, the normal way to do that is to read the paper. .. And if you do that, the paper's naming convention becomes the natural way to express the algorithm.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#135

Post by [deleted] »

dietrichepp wrote: post #130
tlrobinson wrote: post #60

Well, what's the Weissman score?

The Weissman score is the most moronic compression "metric" ever devised. I put "metric" in quotes because from a mathematical perspective, it is practically gibberish. I'm tired of seeing it mentioned in every HN post on compression.

[deleted]

Re: Apple Open-Sources its Compression Algorithm LZFSE

#136

Post by espadrine »

I feel like LZFSE is too little, too late. It would be great to have a proper comparison, but Zstd is stable, and offers a superior compression ratio with compression and decompression speeds that seem on par with LZFSE.

And Zstd is not proprietary. (This issue is relevant in this regard: https://github.com/lzfse/lzfse/issues/21)

https://github.com/Cyan4973/zstd

Edit: here is a quick comparison I did on Linux with Project Gutemberg's webster (http://sun.aei.polsl.pl/~sdeor/corpus/webster.bz2).

  $ time ./lzfse-master/build/bin/lzfse -encode -i webster -o webster.lzfse
  real    0m1.885s
  user    0m1.860s
  sys     0m0.024s

  $ time ./zstd-master/programs/zstd webster -8 -f -o webster.zstd
  webster              : 25.98%   (41458703 =>10772836 bytes, webster.zstd)      
  real    0m1.700s
  user    0m1.660s
  sys     0m0.036s

  $ ls -l
  -rw-r--r-- 1 tyl tyl 12209496 Jul  7 16:26 webster.lzfse
  -rw-rw-r-- 1 tyl tyl 10772836 Jul  7 16:31 webster.zstd

  $ time ./lzfse-master/build/bin/lzfse -decode -i webster.lzfse -o /dev/null
  real    0m0.127s
  user    0m0.112s
  sys     0m0.012s

  $ time ./zstd-master/programs/zstd -d webster.zstd -o /dev/null
  webster.zstd        : 41458703 bytes                                           
  real    0m0.116s
  user    0m0.112s
  sys     0m0.000s
LZFSE's -h option doesn't show a flag to tweak compression. Zstd's default -1 compression is super-fast, but obviously not optimal. Its -8 is the closest I got to LZFSE's compression speed; its -4 was the closest to LZFSE's compression ratio, with a speed of 0m0.527s real compression, 0m0.101s real decompression.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#137

Post by vog »

dozzie wrote: post #133
mschuster91 wrote: post #121

Earlier quoted context omitted.

> Indeed, the current version of their Makefile is a great example of how to write a simple yet portable Makefile: Yet it forgets the MOST important thing: make uninstall. Nothing worse than software where I have to reverse engineer a makefile in order to uninstall!

> Yet it forgets the MOST important thing: make uninstall. Why it forgets that? It's not the job of a library build system to install or uninstall things in your system . It's a job for your system's package manager. Makefile that has a target (historically called "install") that puts things in appropriate places in a chroot-like manner is just good enough.

I fully agree!

Moreover, I tend to see "make install" not as actual installation command (that's indeed the task of a package manager), but more of an "extract the relevant build results".

With that in mind, I prefer packages that have a clear "build destination" folder which is filled (and updated) by a plain "make", so that "make install" isn't even needed anymore, because installation then boils down to a trivial "cp -R" command.

This, of course, requires a disciplined separation of intermediate build results from the final build results. But that shouldn't be too hard. In fact, this may be simpler than writing down a good "make install" in the first place.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#140

Post by Twirrim »

finchisko wrote: post #117
madeofpalk wrote: post #116

Earlier quoted context omitted.

You mean... like gzip?

yeah, but also the opposite direction client -> server. client will send compressed data to server use LZFSE. Kind of nice extension to HTTP protocol, which only supports compression from server to client.

There are already gzip, lzma etc. compression libraries for javascript you could use if you really wanted to do this.
Post reply on HN