Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Open-Sources its Compression Algorithm LZFSE

infoq.com

Post reply on HN
Print view View original

41–50 of 219 posts

Re: Apple Open-Sources its Compression Algorithm LZFSE

#41

Post by dchest »

shmerl wrote: post #25
DiabloD3 wrote: post #18

Earlier quoted context omitted.

So, worse than LZ4 for what Apple seems to be using it for. Why didn't they just use LZ4? Confusing company, they are.

They are very much into NIH, seemingly out of paranoid fear of patent attacks (though not sure how it can protect them).

This is a baseless speculation. First of all, Apple provides LZ4 in libcompression. Secondly, LZFSE uses Lempel–Ziv algorithm and ANS coder invented by Jarek Duda (https://arxiv.org/abs/1311.2540): https://developer.apple.com/library/ios/documentation/Perfor...

Re: Apple Open-Sources its Compression Algorithm LZFSE

#42

Post by dang »

panic wrote: post #30

The article is essentially a link to https://www.infoq.com/news/2016/07/apple-lzfse-lossless-open... with a bunch of ads on top. Maybe someone could update it to point there instead?

Ok, we changed to that from http://www.appleworld.today/blog/2016/7/6/apples-lzfse-compr....

Re: Apple Open-Sources its Compression Algorithm LZFSE

#43

Post by a_imho »

Someone wrote: post #16
benmarten wrote: post #10

I just quickly tested it, in terms of highest compression ratio it still does not beat xz, e.g. `tar -cf -FILE | xz -c9e > FILE.tar.xz` https://blog.benmarten.me/2016/04/01/Compress-Files-With-Hig...

That's not surprising, given that they went for compression and decompression speed and for energy usage. Their goal seems to have been to be at least as good as zlib at compressing stuff using less energy and doing it faster (that often correlates quite well with energy use on modern CPUs, as it allows them to drop to low energy states faster)

Could you give me some pointers on the actual numbers? My searches came back with nothing. I'm especially interested how they benchmarked the energy consumption.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#44

Post by Pica_soO »

gigatexal wrote: post #31
nodesocket wrote: post #9

If you want to see some crazy C code, check out this file from the GitHub repo: https://github.com/lzfse/lzfse/blob/master/src/lzvn_encode_b...

I wonder why they use goto statements instead of just returning q1 like the statement evaluates to.

No function call overhead. Makes sense as long as you stay in the same state-machine. This thing doesn't have to be pretty. It has to be fast. Who cares for any oo-written implementation that takes half a hour to do the same job?

Re: Apple Open-Sources its Compression Algorithm LZFSE

#45

Post by nicky0 »

pilif wrote: post #29

It's 2016. How can you launch a reasonably high profile open source project with code that looks like this? This fulfills all the TODO list for unreadable code. One character variable names, one character parameter names, full of magic numbers... Yes. This is very performance critical code and I completely see the need to write very optimized code. That's fine. But optimizing code for speed shouldn't imply also optim…

You could offer to rewrite if for them...

It's an implementation of a mathematical algorithm. It doesn't need allTheVariables toBeNamed likeThis. Single letters map to meaningful concepts in the mathematical algorithm.

I don't see how giving the variables longer names would make it more readable. Indeed I think long variable names would obscure the structure.

Code like this has to be looked at in the concept of the algorithm design (which I hope exists...)

Re: Apple Open-Sources its Compression Algorithm LZFSE

#46

Post by msvalkon »

pilif wrote: post #34
johncolanduoni wrote: post #32

Earlier quoted context omitted.

> To release compression code in a non-safe language is risky enough At the moment, what's their real alternative? Rust is the only memory-safe language I can think of that could hope to meet their performance requirements, but even the Rust runtime would be a lot of overhead for this application. That said, I agree this isn't acceptable C code for something that runs on untrusted data while using tons of pointer ari…

You're right about C. C in general, I would find acceptable, because, yes, there aren't that many good alternatives around for this kind of code. But there's nothing stopping you from writing readable C code. That's where my concerns come from.

I don't really understand where the downvotes come from? I find the readability concerns legitimate, and would like to understand why compression algorithm developers feel like this is OK? Is it just the math heavy background? Can't think of any real benefits to this style.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#47

Post by venomsnake »

nodesocket wrote: post #9

If you want to see some crazy C code, check out this file from the GitHub repo: https://github.com/lzfse/lzfse/blob/master/src/lzvn_encode_b...

When a code is rooted deeply in domain, the definition or crazy should take into context is code readable for people in knowledge in the same domain.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#48

Post by B1FF_PSUVM »

nicky0 wrote: post #45
pilif wrote: post #29

It's 2016. How can you launch a reasonably high profile open source project with code that looks like this? This fulfills all the TODO list for unreadable code. One character variable names, one character parameter names, full of magic numbers... Yes. This is very performance critical code and I completely see the need to write very optimized code. That's fine. But optimizing code for speed shouldn't imply also optim…

You could offer to rewrite if for them... It's an implementation of a mathematical algorithm. It doesn't need allTheVariables toBeNamed likeThis. Single letters map to meaningful concepts in the mathematical algorithm. I don't see how giving the variables longer names would make it more readable. Indeed I think long variable names would obscure the structure. Code like this has to be looked at in the concept of the a…

It's funny, because there's a whole couple of generations out there that have been taught that explicitVariableNames will fix their thinking deficit.

I mean, if it weren't sad.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#49

Post by MichaelBurge »

pilif wrote: post #29

It's 2016. How can you launch a reasonably high profile open source project with code that looks like this? This fulfills all the TODO list for unreadable code. One character variable names, one character parameter names, full of magic numbers... Yes. This is very performance critical code and I completely see the need to write very optimized code. That's fine. But optimizing code for speed shouldn't imply also optim…

I feel like spitting on other people's code is a good way to become an unhappy person. And this is coming from someone who's had a few head-scratching gdb sessions when a segfault occasionally appears in similar LZF compression code.

And that's because doing the work is more valuable than being a snoot in the style aristocracy.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#50

Post by nicky0 »

dchest wrote: post #41
shmerl wrote: post #25

Earlier quoted context omitted.

They are very much into NIH, seemingly out of paranoid fear of patent attacks (though not sure how it can protect them).

This is a baseless speculation. First of all, Apple provides LZ4 in libcompression. Secondly, LZFSE uses Lempel–Ziv algorithm and ANS coder invented by Jarek Duda ( https://arxiv.org/abs/1311.2540 ): https://developer.apple.com/library/ios/documentation/Perfor...

Can I just say I really appreciate your contributions on this thread. Answering crass company-bashing with well-referenced and informative replies.
Post reply on HN