Live data from Hacker News

Apple Open-Sources its Compression Algorithm LZFSE

infoq.com

101–110 of 219 posts

Re: Apple Open-Sources its Compression Algorithm LZFSE

#101

If you want to see some crazy C code, check out this file from the GitHub repo: https://github.com/lzfse/lzfse/blob/master/src/lzvn_encode_b...

Excerpt from the link : if (D == D_prev) { if (L == 0) { *q++ = 0xF0 + (x + 3); // XM! } else { *q++ = (L >8 in 0..5 *q++ = (D >> 8) + (L = (1 34) { // Long dist *q++ = (L > 2) + (L

[deleted]

Re: Apple Open-Sources its Compression Algorithm LZFSE

#102
post #17

With energy efficiency as a primary goal I was expecting way more use of explicit SIMD instructions. The InfoQ post mentions xcodebuild, but there is also a Makefile. I really appreciate the presence of a no-nonsense Makefile. No autoconf, no pkgconfig, just plain and simple make. Also, because nobody mentioned it: yes, it compiles on Linux out of the box.

I don't mind pkg-config that much: if anything, it's one of the more sensible portability mechanisms out there.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#103

Earlier quoted context omitted.

Excerpt from the link : if (D == D_prev) { if (L == 0) { *q++ = 0xF0 + (x + 3); // XM! } else { *q++ = (L >8 in 0..5 *q++ = (D >> 8) + (L = (1 34) { // Long dist *q++ = (L > 2) + (L

I've never seen this code base before, but this makes straightforward sense to me as I do keep my hand in with compression software, and I don't anticipate others who works with compression algorithms in general would have any trouble. It looks just like the style of code in all the other fast LZ codebases. They are all in this style. The "non-aligned access OK" comment litter is presumably to silence an LLVM perform…

When implementing complex algorithm, this kind of code is usually easiest to understand.

When you look at the code, you use the paper that describes the algorithm as documentation. Using same short one letter variable names in the code and paper makes understanding much easier.

The thing I hate most is when the the paper uses 1-based numbering and the programming language uses 0-based numbering. We should settle for 0-based numbering when describing algorithms.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#104
post #103

Earlier quoted context omitted.

I've never seen this code base before, but this makes straightforward sense to me as I do keep my hand in with compression software, and I don't anticipate others who works with compression algorithms in general would have any trouble. It looks just like the style of code in all the other fast LZ codebases. They are all in this style. The "non-aligned access OK" comment litter is presumably to silence an LLVM perform…

When implementing complex algorithm, this kind of code is usually easiest to understand. When you look at the code, you use the paper that describes the algorithm as documentation. Using same short one letter variable names in the code and paper makes understanding much easier. The thing I hate most is when the the paper uses 1-based numbering and the programming language uses 0-based numbering. We should settle for…

Arithmetic coding often looks just as dense, because C is not a good vehicle to describe algorithms.

Look at eg https://www.cs.ox.ac.uk/jeremy.gibbons/publications/arith.pd... to see a cleaner alternative.

(This is about describing algorithms in papers. Optimizing for performance after the big-O has been taken care of is a different matter.)

Re: Apple Open-Sources its Compression Algorithm LZFSE

#105

Oh wow dude you watch Silicon Valley also? Great reference! It sure added a lot to the conversation. Please continue making useful comments like this in the future, you really got us thinking with your pop culture reference. (And I sure patted myself on the back for recognizing such witticism)

There's a way to make this point without being an asshole about it.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#106
post #29

It's 2016. How can you launch a reasonably high profile open source project with code that looks like this? This fulfills all the TODO list for unreadable code. One character variable names, one character parameter names, full of magic numbers... Yes. This is very performance critical code and I completely see the need to write very optimized code. That's fine. But optimizing code for speed shouldn't imply also optim…

I agree on the C-bashing. But the single-letter variable names are fine---they correspond to the paper, and are probably very mathematical entities that don't even have real longer names.

(Longer names are for book keeping, not so much for calculation.)

Re: Apple Open-Sources its Compression Algorithm LZFSE

#107

Earlier quoted context omitted.

I wonder why they use goto statements instead of just returning q1 like the statement evaluates to.

No function call overhead. Makes sense as long as you stay in the same state-machine. This thing doesn't have to be pretty. It has to be fast. Who cares for any oo-written implementation that takes half a hour to do the same job?

What does return have to do with OO?

(If you want to be glib, complain about misplaced FP perhaps?)

Re: Apple Open-Sources its Compression Algorithm LZFSE

#108
post #24

> LZFSE is only present in iOS and OS X, so it can’t be used when the compressed payload has to be shared to other platforms (Linux, Windows). So now it will be cross platform?

You can bet that there is a Linux version soon, and if it's good enough, it will end up in the default repo. For Windows, it's a different story, but maybe because of the iPhone and iPad, and many MS employees using them, will it be supported somewhere in the not so near future. Anyways, 7zip and Winzip will probably support it soon enough.

It compiles in Linux as is. There is a generic makefile.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#109

If you want to see some crazy C code, check out this file from the GitHub repo: https://github.com/lzfse/lzfse/blob/master/src/lzvn_encode_b...

I wonder why they use goto statements instead of just returning q1 like the statement evaluates to.

They seem to like a style that has rather constrained use of `return' (but uses deliberate amounts of goto instead).

Not sure why.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#110
post #52
post #45

Earlier quoted context omitted.

You could offer to rewrite if for them... It's an implementation of a mathematical algorithm. It doesn't need allTheVariables toBeNamed likeThis. Single letters map to meaningful concepts in the mathematical algorithm. I don't see how giving the variables longer names would make it more readable. Indeed I think long variable names would obscure the structure. Code like this has to be looked at in the concept of the a…

> You could offer to rewrite if for them... Nope. Because I can't read it. I plainly do not have the information needed to understand what's going on here, nor is the design document part of the source code. As you seem to have no trouble understanding what's going on, can you enlighten me, for example, what's so special about the number 271 that we see M being compared to? > I don't see how giving the variables long…

> I plainly do not have the information needed to understand what's going on here

That's not a problem with the code, that's you not understanding the math that's actually happening. For what it's worth, neither do I, but from what I'm gleaning from other comments here, it's a C implementation of a mathematical proof, so it'd be better to go learn the math than expect to be given friendly code. Besides, "open source" doesn't mean "catered to the lowest common denominator" - they don't write code for you or I to understand without some work, we have to earn the knowledge.

Post reply on HN