161–170 of 197 posts
Earlier quoted context omitted.
“Recent” as in the 80s or 90s, sure. Threads are older than unix was when threads were introduced.
I’m not sure I understand. Typically I’ve thought of threads called “threads” being formalized with POSIX threads. Before that, while my memory is vague on this, it was just lumped into multiprocessing under various names.
Earlier quoted context omitted.
"zstd finally has some potential to replace gzip" bzip2 and xz have had the potential to replace gzip for the vast majority of users and use cases since more than a decade - and in many cases they have.
What I'm trying to say is that the excessive focus on cutting-edge technology is holding back progress. gzip is still the default compressor people use when they are not sure about the appropriateness of other compressors in their specific use case, and they don't have the time or energy to find out. To replace it, the a compressor must satisfy two requirements: * It must not be substantially worse than gzip on any r…
Earlier quoted context omitted.
I’m not sure I understand. Typically I’ve thought of threads called “threads” being formalized with POSIX threads. Before that, while my memory is vague on this, it was just lumped into multiprocessing under various names.
Even if you ignore threading models prior to pthreads, pthreads itself dates to 1995.
Earlier quoted context omitted.
That’s fairly new, with threads and processes becoming basically the same. Historically threads didn’t exist, then they were horrifically implemented and non standard, then they standardized and were horrifically implemented, then they were better implemented but the APIs were difficult to use safely, etc etc. Also threads were much more light weight than a process. This shifted with light weight processes, etc.
That's true, but if you're looking that far back, multicore is new too
Earlier quoted context omitted.
What I'm trying to say is that the excessive focus on cutting-edge technology is holding back progress. gzip is still the default compressor people use when they are not sure about the appropriateness of other compressors in their specific use case, and they don't have the time or energy to find out. To replace it, the a compressor must satisfy two requirements: * It must not be substantially worse than gzip on any r…
I think these arguments are a bit contrived. bzip2 made progress on compression ratio. xz (and zstd) made progress on both compression ratio and speed. Neither hovered around the idea of cutting-edge "technology" (they all fall into the same technology: general data compression algorithms) because they aren't niched oddities like e.g. paq. But I don't understand why a successor must trump gzip in both aspects. gzip c…
I work in bioinformatics, where people typically use either gzip or domain-specific compressors. gzip is used for the reasons I mentioned. It works, it's usually good enough, and if people in another organization you've never heard of want to use your compressed files, they can do so without bothering you with support requests.
zstd would be faster and compress better, but because you can't be sure everyone else can use it, you don't even bother thinking about it. The saved computational resources are probably not worth it. On the other hand, anything that makes gzip faster is valuable, as it allows saving computational resources without taking any interoperability risks.
I didn't say the gzip replacement must be better than gzip in every aspect. I said it must not be substantially worse. bzip2 was substantially worse, because it was substantially slower.
I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…
Side note: In the 1990s, everyone in my engineering school had an HP48 calculator. There were a healthy selection of pretty decent games available. One fine day, I finished my physics exam an hour early and so opened up an enjoyable game on my calculator. 45 minutes went by and so I went up and handed in my paper. It was at this point that the professor noted, “were you planning on leaving the second page blank?” Oh.
Teachers were completely stumped. Their initial suspicions were always that someone brought a universal remote control to class, but they would painstakingly search everyone's desks to find nothing. And then after asking everyone to put their hands up in the ai, the TV would still have a mind of its own.
Yeah, your TI-89 was no fun.
I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…
This episode was fascinating. I had heard of LZ4 but not Zstd. It spurred me to make changes to our system at work that are reducing file sizes by as much as 25%. It’s great to have a podcast in which I learn practical stuff!
Depending upon the data, the non-threaded versions of these utilities can have higher performance when run with some kind of dispatcher on multiple files.
The GNU xargs utility is able to do this, and the relevant features are also in busybox.
I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…
One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…
There are also optimizations that only work on today's larger cores, and you have to actively update old code to get the advantages (happily some work is going into that): https://news.ycombinator.com/item?id=32533061 / https://news.ycombinator.com/item?id=32537545
That's not to minimize the clever ideas and amazing implementation work in new stuff. It's more that people were making smart decisions both then and now, more so than you might guess just from comparisons on today's hardware.
I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…
One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…
Earlier quoted context omitted.
gzip has the advantage of being ubiquitous. It's pretty much guaranteed to be available on every modern Unix-alike. And is good enough for most purposes. Zstd is getting there but I personally don't bother with it on a daily basis except in situations where both performance and compression ratio are important, like build artifact pipelines or large archives.
You make it sound like installing zstd is a big deal. Which it is not.