Apple Open-Sources its Compression Algorithm LZFSE

Post by joeblau » Thu, Jul 07, 2016, 8:29 PM UTC

PeCaN wrote: post #156

joeblau wrote: post #79

Earlier quoted context omitted.

I feel like the main thing that makes this look crazy is the variable naming and bit shifting. If someone saw this program with descriptive variable names and array operations, it would probably look less daunting.

Eh. I feel like long variable names would obscure the algorithm. And bit shifting and array operations aren't interchangeable…. C just tends to look like line noise for numerical algorithms sometimes.

Post by PeCaN » Thu, Jul 07, 2016, 8:54 PM UTC

legulere wrote: post #176

PeCaN wrote: post #162

Earlier quoted context omitted.

I agree. The only questionable part is the stuff like *(uint16_t *)q = D; q += 2; // non-aligned access OK *(uint32_t *)q = literal; instead of memcpy(q, &D, sizeof(uint16_t)); q += 2; // non-aligned access OK memcpy(q, &literal, sizeof(uint32_t)); which is better defined behavior (i.e. doesn't violate -fstrict-aliasing) and possibly faster.

Why not test it and do a pull request?

Post by eye2sky » Thu, Jul 07, 2016, 9:31 PM UTC

klodolph wrote: post #154

ausjke wrote: post #139

A quick test resutl(zip a 1.5GB file): lzfse: real 1m44.481s user 1m17.956s sys 0m2.852s lz4: real 0m28.136s user 0m1.200s sys 0m2.240s lz4 is much faster somehow. The final size are very close.

It's not terribly useful for most applications to test compression speed. The only applications I can think of where this is relevant is data backup and archiving. There are two typical speed benchmarks you want to do. For the "compress once, decompress many times" situation, benchmark the time it takes to decompress and ignore compression time. For the "compress once, decompress once" situation, add the compression…

Post by vog » Thu, Jul 07, 2016, 9:44 PM UTC

mschuster91 wrote: post #170

vog wrote: post #126

Earlier quoted context omitted.

I don't trust "make uninstall" anyway. Instead, I usually put self-compiled packages into separate directories, such as: make install INSTALL_PREFIX=/opt/lzfse or make install INSTALL_PREFIX=$HOME/.../lzfse Uninstalling is then as simple as: rm -r /opt/lzfse For convenience, I either add /opt/lzfse/bin to $PATH , or create a symlink from /opt/lzfse/bin/lzfse to /usr/bin/lzfse . More generally, I believe that uninstal…

Good luck with getting pkg-config to recognize your include files automatically, for example. Or the good old "man" utility. Alternative: maintain a HUUUUUGE list of xPATH env variables, and update them every time you recompile something and change the directory in the process. Oh, and hope that other people's Makefiles are intelligent enough to not mess up shit (e.g. use headers from system, and library .so files fr…

Post by steveklabnik » Thu, Jul 07, 2016, 11:20 PM UTC

johncolanduoni wrote: post #186

steveklabnik wrote: post #144

Earlier quoted context omitted.

> as well as a significant chunk of Rust's usability advantage. What bits are you thinking of here? Just curious, as I do a lot of no_std work, and don't feel that way, and am probably blind to it :) Rust 1.10, coming out later today, has a new crate type that removes Rust-specific metadata for dynamic libraries, by the way, making them a bit smaller for this kind of case.

Unless I'm mistaken, no_std means no built-in non-manual dynamic allocation (Box, Rc, etc.), unless you use "extern crate alloc", once again requiring the nightly. Some fundamentals one expects from a modern language like Vec are also missing in either case. This is fine if you're using no_std for something where these are anathema anyway (writing bare-metal OSes comes to mind) but a huge limitation for a humble user…

    #![feature(alloc)]
    #![feature(collections)]
    #![no_std]

    extern crate alloc;
    extern crate collections;

    use alloc::boxed::Box;
    use alloc::rc::Rc;
    use collections::vec::Vec;

    pub fn foo() -> Box {
        Box::new(5)
    }

    pub fn bar() -> Rc {
        Rc::new(5)
    }

    pub fn baz() -> Vec {
        let mut v = Vec::new();

        v.push(5);

        v
    }