Live data from Hacker News

Apple Open-Sources its Compression Algorithm LZFSE

infoq.com

51–60 of 219 posts

Re: Apple Open-Sources its Compression Algorithm LZFSE

#51
post #19

Kind of weak licence, what are you actually allowed to do with this code? Change it? Distribute your changes? https://github.com/lzfse/lzfse/blob/master/LICENSE

Clicked the link expecting to see some obscure license and wryly think "ah, crazy Apple logic again". Turns out it's the BSD license, probably the most well-known and widely used permissive open source license.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#52
post #45
post #29

It's 2016. How can you launch a reasonably high profile open source project with code that looks like this? This fulfills all the TODO list for unreadable code. One character variable names, one character parameter names, full of magic numbers... Yes. This is very performance critical code and I completely see the need to write very optimized code. That's fine. But optimizing code for speed shouldn't imply also optim…

You could offer to rewrite if for them... It's an implementation of a mathematical algorithm. It doesn't need allTheVariables toBeNamed likeThis. Single letters map to meaningful concepts in the mathematical algorithm. I don't see how giving the variables longer names would make it more readable. Indeed I think long variable names would obscure the structure. Code like this has to be looked at in the concept of the a…

> You could offer to rewrite if for them...

Nope. Because I can't read it. I plainly do not have the information needed to understand what's going on here, nor is the design document part of the source code. As you seem to have no trouble understanding what's going on, can you enlighten me, for example, what's so special about the number 271 that we see M being compared to?

> I don't see how giving the variables longer names would make it more readable.

Variables that are part of the algorithm itself probably make sense in their short name (though that's a common problem with math in general. Being more expressive isn't a bad thing), but there are also parameters to public API that are equally short for no gain.

Also, the code is full of magic numbers and I'm sure an algorithm design document would at least give names out to them.

Besides, even if the document was available (it isn't, at least not to the public), there's absolutely zero harm in making the code more accessible or at least reference the spec.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#53
post #29

It's 2016. How can you launch a reasonably high profile open source project with code that looks like this? This fulfills all the TODO list for unreadable code. One character variable names, one character parameter names, full of magic numbers... Yes. This is very performance critical code and I completely see the need to write very optimized code. That's fine. But optimizing code for speed shouldn't imply also optim…

> To release compression code in a non-safe language is risky enough At the moment, what's their real alternative? Rust is the only memory-safe language I can think of that could hope to meet their performance requirements, but even the Rust runtime would be a lot of overhead for this application. That said, I agree this isn't acceptable C code for something that runs on untrusted data while using tons of pointer ari…

Which bits of the Rust runtime(?) do you think are too high overhead for this?

Re: Apple Open-Sources its Compression Algorithm LZFSE

#54

If you want to see some crazy C code, check out this file from the GitHub repo: https://github.com/lzfse/lzfse/blob/master/src/lzvn_encode_b...

I think that there's a scientific paper accompanying that code somewhere. So in order to understand that code one should read the paper and all those names probably just came from paper's algorithms.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#55
post #33

Earlier quoted context omitted.

Blows my mind that people can come up with this stuff. I'm assuming they came up with the mathematical proofs first and translated that into code, so that has something to do with it, correct? It looks a lot like some crypto algorithms which are a nearly direct translation of the mathematical formulas. It's not that it's incredibly difficult to follow, but it's just very "math like".

That is my experience. Maths people are not renowned for their ability to write readable or maintainable code. I recently needed an implementation of the Simplex Noise algorithm (that I could port to Common Lisp). I ended up using this one, which works but the code certainly does nothing to help understanding: https://github.com/josephg/noisejs/blob/master/perlin.js Note that the Javscript implementation is also a po…

Seems pretty well commented. There are definitely worse examples. I think you are underestimating the amount of temporary calculations that most mathematical formulas or algorithms require. It's actually a good thing to see so many vars because the variable names combined with the comments make more sense. Any js packer will most certainly get rid of the redundancy of memory allocations, so I'd say the superfluous var style is harmless.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#56

Earlier quoted context omitted.

Excerpt from the link : if (D == D_prev) { if (L == 0) { *q++ = 0xF0 + (x + 3); // XM! } else { *q++ = (L >8 in 0..5 *q++ = (D >> 8) + (L = (1 34) { // Long dist *q++ = (L > 2) + (L

It's code like this that has an exploit several years in the future given an edge case that is hard to fathom

The variable names hardly seem complex to me.

D clearly refers to some form of "distance". M is "Medium". L is defined before this snippet of code, but I'd imagine it maps to a concept of Long.

And you're left with the variable 'q'.

The real issue is that unless you are comfortable, code involving pointer math can get confusing (all the * and + can be confusing, especially since they are such overloaded symbols in C). The single character variable names (especially when we're talking about what is basically code representation of mathematical formulae) is hardly a huge issue.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#57
post #24

> LZFSE is only present in iOS and OS X, so it can’t be used when the compressed payload has to be shared to other platforms (Linux, Windows). So now it will be cross platform?

You can bet that there is a Linux version soon, and if it's good enough, it will end up in the default repo. For Windows, it's a different story, but maybe because of the iPhone and iPad, and many MS employees using them, will it be supported somewhere in the not so near future. Anyways, 7zip and Winzip will probably support it soon enough.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#58
post #34

Earlier quoted context omitted.

You're right about C. C in general, I would find acceptable, because, yes, there aren't that many good alternatives around for this kind of code. But there's nothing stopping you from writing readable C code. That's where my concerns come from.

I don't really understand where the downvotes come from? I find the readability concerns legitimate, and would like to understand why compression algorithm developers feel like this is OK? Is it just the math heavy background? Can't think of any real benefits to this style.

If you don't understand the underlying mathematical algorithms its using, no amount of explicit varible names are going to help you. If you do, the concise structure makes things straightforward. The code is not meant to be read alone and understood, the papers published along with it need to be understood first.

Re: Apple Open-Sources its Compression Algorithm LZFSE

#59
post #52
post #45

Earlier quoted context omitted.

You could offer to rewrite if for them... It's an implementation of a mathematical algorithm. It doesn't need allTheVariables toBeNamed likeThis. Single letters map to meaningful concepts in the mathematical algorithm. I don't see how giving the variables longer names would make it more readable. Indeed I think long variable names would obscure the structure. Code like this has to be looked at in the concept of the a…

> You could offer to rewrite if for them... Nope. Because I can't read it. I plainly do not have the information needed to understand what's going on here, nor is the design document part of the source code. As you seem to have no trouble understanding what's going on, can you enlighten me, for example, what's so special about the number 271 that we see M being compared to? > I don't see how giving the variables long…

OK but which is more readable ... (I know it's a bit silly, I juts made up names)

} else if (D >= (1 34) {

} else if (DirectWeightingFactor >= HUYGENS_LIMIT || MariachiBand == 0 || (xylemNonce + STANDARD_PZSH_INCREMENT) + MariachiBand > SWIM_RATE_B) {

I guess your opinion differs to mine. I like the one that looks like math.

Post reply on HN