Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Open-Sources its Compression Algorithm LZFSE

infoq.com

Post reply on HN
Print view View original

191–200 of 219 posts

Re: Apple Open-Sources its Compression Algorithm LZFSE

#191

Post by macintux »

wmf wrote: post #180
Negative1 wrote: post #169

I'm not fully up on the latest and greatest in compression technologies but my go to format these days is usually 7zip which I believe is just a container that uses LZMA. For whatever reason *nix people seem to hate it even though I get much better compression with it than tarballs, zlib/zips. Is there a similar container format that will or does use LZFSE? And how much better is it than 7zip/LZMA?

The Unixy version of LZMA is called XZ. Unix people prefer tar over zip, probably for historical/tribal reasons. There are really three different compression "markets"; LZMA/XZ provides strong compression while LZFSE provides moderate or light compression so they don't really compete.

> Unix people prefer tar over zip, probably for historical/tribal reasons.

To this day I refuse to use the z argument to tar. Packaging and compressing are two different operations, and this is why the UNIX gods created pipes.

And also I'm a stubborn old fart.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#192

Post by eridius »

wmf wrote: post #168
lilyball wrote: post #167

Earlier quoted context omitted.

Can you measure energy impact? Apple has specifically said that LZFSE is the right choice when you want lower energy impact, so doing benchmarks of LZFSE against other algorithms is kind of meaningless when you don't measure one of the key metrics. Of course, I'm not sure offhand how to measure energy impact, but Activity Monitor displays energy impact numbers so there must be some way to measure it.

Based on Apple's own presentation it looks like energy efficiency is pretty proportional to performance, so it's very likely that a faster algorithm (e.g. LZ4) would also be more efficient. In modern "braniac" processors, 90% of the energy is spent on overhead and doesn't vary based on instruction mix. Memory accesses consume both energy and time, so fewer are better. (From the previous discussion: https://news.ycomb…

Surely LZ4 is not in fact more energy efficient, otherwise Apple would be pushing LZ4 as the algorithm to use on mobile devices instead of recommending LZFSE.

If you see in that previous discussion, I was asking the same questions there and got no real answer. It looks to me like nobody (outside of Apple) has actually tested the energy efficiency.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#193

Post by johncolanduoni »

steveklabnik wrote: post #189
johncolanduoni wrote: post #186

Earlier quoted context omitted.

Unless I'm mistaken, no_std means no built-in non-manual dynamic allocation (Box, Rc, etc.), unless you use "extern crate alloc", once again requiring the nightly. Some fundamentals one expects from a modern language like Vec are also missing in either case. This is fine if you're using no_std for something where these are anathema anyway (writing bare-metal OSes comes to mind) but a huge limitation for a humble user…

Ah I see. There's two things here: first off, I'm using it in an OSdev context, so I don't expect any allocation to exist, since I haven't actually implemented that yet. And second, I took your comment to mean the language itself, which doesn't lose anything with no_std, but you mean the convenience of the libraries, which makes sense. By the way, you _can_ reintroduce just those things if you want to. no_std means "…

Yes, my issue is just that creating a small library with a C API requires a lot of machinations that are going to turn off anyone who isn't really set on going with Rust. For OS development, standalone binaries, or Rust libraries, I think Rust is in excellent shape as it is.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#194

Post by indolering »

dietrichepp wrote: post #130
tlrobinson wrote: post #60

Well, what's the Weissman score?

The Weissman score is the most moronic compression "metric" ever devised. I put "metric" in quotes because from a mathematical perspective, it is practically gibberish. I'm tired of seeing it mentioned in every HN post on compression.

Silicon Valley gets rave reviews, but a story plot based on a patented compression algorithm....

Re: Apple Open-Sources its Compression Algorithm LZFSE

#195

Post by eru »

rusanu wrote: post #146
eru wrote: post #109

Earlier quoted context omitted.

They seem to like a style that has rather constrained use of `return' (but uses deliberate amounts of goto instead). Not sure why.

I assume is "one return only per function" rule http://programmers.stackexchange.com/questions/118703/where-...

Not quite, there are multiple returns, still.

But might be something related, eg like they are talking about a state machine in the description somewhere, and give at most one return per state, or something like that?

Re: Apple Open-Sources its Compression Algorithm LZFSE

#196

Post by eln1 »

Someone wrote: post #190
espadrine wrote: post #136

I feel like LZFSE is too little, too late. It would be great to have a proper comparison, but Zstd is stable, and offers a superior compression ratio with compression and decompression speeds that seem on par with LZFSE. And Zstd is not proprietary. (This issue is relevant in this regard: https://github.com/lzfse/lzfse/issues/21 ) https://github.com/Cyan4973/zstd Edit: here is a quick comparison I did on Linux with P…

I’m not that comfortable with Xcode, but I just built two versions of this tool, one direct from GitHub, one (called lzfse2) that calls the Compression library on Mac OS X. Typical timings on my system are: > time ./lzfse -encode -i webster -o webster.lzfse real 0m0.945s user 0m0.864s sys 0m0.062s > time ./lzfse2 -encode -i webster -o webster.lzfse2 real 0m0.803s user 0m0.715s sys 0m0.072s > time ./lzfse -decode -i w…

The difference is size probably come from suboptimalities which were repaired after open-sourcing: http://encode.ru/threads/2221-LZFSE-New-Apple-Data-Compressi... The difference in speed is more surprising, it could come from using a different compiler - there can be really large differences.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#197

Post by technion »

svckr wrote: post #17

With energy efficiency as a primary goal I was expecting way more use of explicit SIMD instructions. The InfoQ post mentions xcodebuild, but there is also a Makefile. I really appreciate the presence of a no-nonsense Makefile. No autoconf, no pkgconfig, just plain and simple make. Also, because nobody mentioned it: yes, it compiles on Linux out of the box.

> I really appreciate the presence of a no-nonsense Makefile

As expected whenever a project does this, there's already a PR from someone trying to change that.[0]

[0] https://github.com/lzfse/lzfse/pull/15

Re: Apple Open-Sources its Compression Algorithm LZFSE

#199

Post by jibsen »

hollander wrote: post #57
shmerl wrote: post #24

> LZFSE is only present in iOS and OS X, so it can’t be used when the compressed payload has to be shared to other platforms (Linux, Windows). So now it will be cross platform?

You can bet that there is a Linux version soon, and if it's good enough, it will end up in the default repo. For Windows, it's a different story, but maybe because of the iPhone and iPad, and many MS employees using them, will it be supported somewhere in the not so near future. Anyways, 7zip and Winzip will probably support it soon enough.

It compiles fine with mingw-w64 on Windows. MSVC is a bit more involved due to the GCC extensions used. When LZFSE first came out I made a fork with the changes to compile with MSVC [1], but there wasn't any interest in it.

[1]: https://github.com/jibsen/lzfse/tree/msvc-compatibility

Re: Apple Open-Sources its Compression Algorithm LZFSE

#200

Post by jibsen »

Someone wrote: post #190
espadrine wrote: post #136

I feel like LZFSE is too little, too late. It would be great to have a proper comparison, but Zstd is stable, and offers a superior compression ratio with compression and decompression speeds that seem on par with LZFSE. And Zstd is not proprietary. (This issue is relevant in this regard: https://github.com/lzfse/lzfse/issues/21 ) https://github.com/Cyan4973/zstd Edit: here is a quick comparison I did on Linux with P…

I’m not that comfortable with Xcode, but I just built two versions of this tool, one direct from GitHub, one (called lzfse2) that calls the Compression library on Mac OS X. Typical timings on my system are: > time ./lzfse -encode -i webster -o webster.lzfse real 0m0.945s user 0m0.864s sys 0m0.062s > time ./lzfse2 -encode -i webster -o webster.lzfse2 real 0m0.803s user 0m0.715s sys 0m0.072s > time ./lzfse -decode -i w…

Try compiling with a recent 64-bit GCC and adding `-mtune=native -flto` to the mix.
Post reply on HN