41–50 of 107 posts
Earlier quoted context omitted.
no compressor tests the output while compressing as it hurts the performance. you can do it after compressing, though, using `bzip3 -t`.
Right, but I would probably test the output while compressing instead of putting a big USE AT YOUR OWN PERIL sign across the front…
There comes a point where the complexity itself becomes too much of a liability. It's important to be able to trust these algorithms as well as all popular implementations with your data.
One should verify the integrity of stuff like backups or archives anyway, by supplying the end user with a sha1 or better hash of both the compressed/encrypted archive as well as all of the files it contains, and by regularly verifying if both still match.
Has anyone tried doing zstd at the end instead of LZ77 and entropy coding?
Does the idea even make sense? (I'm a layman)
Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data. I am not sure how I feel abo…
I think it boils down to the feelings of the author (of the previous format).
I don't think PKWARE feels bad because ZSTD is a homage to ZIP. Similarly if someone created a follow-up file format to something I've designed, I'd just want credit or a link to my version as a homage and pointer for history continuity, nothing else.
Open source software is designed to be mangled, modified, shared and leapfrogged. If a completely different implementation advertises itself as a newer iteration of a format because it's built on the same theory, I think it's ethical if the developer is not intending to capitalize name. Either case, if the original developer returns to the game, it can create a BZIP4 and points to the diversion as, "hey, somebody liked BZIP2 too much and created this, give him/her a kudos", and continue.
Earlier quoted context omitted.
That's what I took out of it too. Sacrificed a bit of speed and a lot of memory for a smaller output size. edit: ah, bzip3 is parallelizable, while bzip2 isn't. That alone is enough for me to be able to claim 'faster'.
bzip2 can exploit concurrency through pbzip2, can't it?
(context: I have had a situation where files created by pbzip2 on linux were not able to be decompressed with some library on .NET, but using lbzip2, they were. I never looked into the details.)
Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data. I am not sure how I feel abo…
That's funny, 23 MiB/s is exactly what I get for reading systemd logs (on an NVME SSD). Is it supposed to be otherwise? $ sudo journalctl -r | pv -a > /dev/null [22.8MiB/s]
$ dd if=/dev/urandom of=test bs=1G count=1 iflag=fullblock
$ gzip -k test
$ zcat test.gz | pv -a >/dev/null
[ 228MiB/s]
$ sudo journalctl -r | pv -a >/dev/null
[13.1MiB/s]
$ gzip -k adventures-of-huckleberry-finn.txt
$ zcat adventures-of-huckleberry-finn.txt.gz | pv -a >/dev/null
[ 151MiB/s]
Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data. I am not sure how I feel abo…
> I am not sure how I feel about you "stealing" the bzip name. While the author of bzip2 doesn't seem to plan to release a follow-up, I feel it is bad manner to take over a name like this. I think it boils down to the feelings of the author (of the previous format). I don't think PKWARE feels bad because ZSTD is a homage to ZIP. Similarly if someone created a follow-up file format to something I've designed, I'd just…
The original author replaced their own bzip release with bzip2 to avoid a patent issue with arithmetic coding, so this is the first time a third party has done so: https://web.archive.org/web/19980704181204/http://www.muraro...
So if the release of bzip3 is approved by the current maintainer, then I guess it’s fine, but otherwise it makes me uncomfortable to consider using under this name.
It's not entirely clear to me what the selling point is. "Better than bzip2" isn't exactly a convincing sales pitch given bzip2 is mostly of historic interest these days.
Right now the modern compression field is basically covered by xz (if you mostly care about best compression ratio) and zstd (if you want decent compression and very good speed), so when someone wants to pitch a new compression they should tell me where it stands compared to those.
Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data. I am not sure how I feel abo…
> I am not sure how I feel about you "stealing" the bzip name. While the author of bzip2 doesn't seem to plan to release a follow-up, I feel it is bad manner to take over a name like this. I think it boils down to the feelings of the author (of the previous format). I don't think PKWARE feels bad because ZSTD is a homage to ZIP. Similarly if someone created a follow-up file format to something I've designed, I'd just…
I agree in spirit, but I can also see why someone might want their source to be free and mangleable, but still care about trademarks.
(Just imagine Linus Torvalds getting lots of emails with support requests for a hypothetical Linux2 operating system that I wrote, and that he has no relation with. That could become pretty annoying; even if he doesn't mind me taking his source code.)