Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

7-Zip for Windows can now use more than 64 CPU threads for compression

7-zip.org

Post reply on HN
Print view View original

191–200 of 220 posts

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#191

Post by ars »

sltkr wrote: post #180
sidewndr46 wrote: post #171

Earlier quoted context omitted.

Here are some examples of what I was doing in one case https://www.hydrogen18.com/blog/apk-the-strangest-format.htm... I was running "zstd --ultra --threads=0" which I assumed was asking it for the absolute maximum

I think your mistake was to use --ultra without a compression level. I redid your experiments with rust-wasm-1.83.0-r0.apk: size perc c.time d.time uncompressed: 290072064 - - gzipped original: 105255109 36.29% - bzip2 -9: 107099379 36.92% 21.1s 11.0s bzip3 -b511: 73539847 25.35% 28.9s 32.0s xz --extreme -9: 71010672 24.48% 142.0s 3.1s lzip -9: 70964413 24.46% 173.5s 5.3s zstd --ultra -22: 48288499 16.64% 155.6s 0.4s…

That was an impressive result, so I tried it on a huge email inbox.

    uncompressed:    1512662084
    xz --extreme -9:  508431572  12:47
    zstd --ultra -21: 508432560  12:44
(-22 ran out of memory.) So at least by me zstd was identical to xz almost to the byte and the second.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#192

Post by psyclobe »

dikei wrote: post #29
d33 wrote: post #10

I worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.

I use ZSTD a ton in my programming work where efficiency matters. But for sharing files with other people, ZIP is still king. Even 7z or RAR is niche. Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger.

zip is such a shit standard, hell there are parts of it that are still undocumented and sharing documents between system zip implementations across mac and windows sometimes fails.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#193

Post by monocasa »

p_ing wrote: post #175
monocasa wrote: post #164

Earlier quoted context omitted.

Microsoft was absolutely wanting to target large servers at the time. They were actively trying to kill off the vendor unices in the 90s.

They successfully killed off vendor unicies in the 90s, but that was thanks to cheap x86 .

That was what stuck, but supporting the big servers was also part of their multifaceted strategy. That's why the alpha, itanium, powerpc, and mips ports existed.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#194

Post by ars »

ars wrote: post #191
sltkr wrote: post #180

Earlier quoted context omitted.

I think your mistake was to use --ultra without a compression level. I redid your experiments with rust-wasm-1.83.0-r0.apk: size perc c.time d.time uncompressed: 290072064 - - gzipped original: 105255109 36.29% - bzip2 -9: 107099379 36.92% 21.1s 11.0s bzip3 -b511: 73539847 25.35% 28.9s 32.0s xz --extreme -9: 71010672 24.48% 142.0s 3.1s lzip -9: 70964413 24.46% 173.5s 5.3s zstd --ultra -22: 48288499 16.64% 155.6s 0.4s…

That was an impressive result, so I tried it on a huge email inbox. uncompressed: 1512662084 xz --extreme -9: 508431572 12:47 zstd --ultra -21: 508432560 12:44 (-22 ran out of memory.) So at least by me zstd was identical to xz almost to the byte and the second.

I got -22 to run:

    zstd --ultra -22: 494517545 14:00
Pretty minor difference.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#195

Post by notepad0x90 »

fsckboy wrote: post #186
notepad0x90 wrote: post #174

Earlier quoted context omitted.

why? and why does it have to be a compromise? You're assuming things because things are already done insecurely. You can authenticate the self-extractor as well as the extracted content. The user gets a nice message "This is a 7zip self-extracting archive sent to you by Bob containing the files below". As an incident responder, I've seen much more of regular archives being used to social engineer users than self-extr…

jar files are zip files, so they don't contain "self extract" code, instead they are associated with already installed extraction code. however, once extracted, jar files do contain executable code, and that is a security issue. the java model pays attention to security, but if code can do something, it can do something bad. if it can't do something, it's not very useful, is it.

the windows kernel executes a self extracting 7z archive, java.exe extracts and executes .jar files. If the 7z self extractor was .net CLR bytecode it would operate very much the same as .JAR files. to your point though, the primary purpose of JAR files is not to compress and transport other files, they're supposed to be executables only. From a user's perspective, abuse potential is the main difference.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#196

Post by Szpadel »

soruly wrote: post #122
Szpadel wrote: post #106

Earlier quoted context omitted.

in my experience using zstd --long --ultra -22 gives marginally better compression ratio than xz -9 while being significantly faster

I think it depends on what you're compressing. I experimented with my data full of hex text xml files. xz -6 is both faster and smaller than zstd -19 by about 10%. For my data, xz -2 and zstd -17 achieve the same compressed size but xz -2 is 3 times faster than zstd -17. I still use xz for archive because I rarely needs to decompress them.

Try combining it with --long

My use cases are usually source code, SQL dumps and log files.

Sometimes xz gave marginally better results, but difference was well below 1%

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#197

Post by xxs »

sidewndr46 wrote: post #171
xxs wrote: post #105

Earlier quoted context omitted.

do you have examples where xz 'blows it away', not just zstd -3?

Here are some examples of what I was doing in one case https://www.hydrogen18.com/blog/apk-the-strangest-format.htm... I was running "zstd --ultra --threads=0" which I assumed was asking it for the absolute maximum

yup, you should have tried just different -NN, and notice. I had a talk on zstd couple of years back, and one of the points was that it was better than xz across the board.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#198

Post by xxs »

danudey wrote: post #163
xxs wrote: post #107

Earlier quoted context omitted.

Most linux distributions have zip support with zstd.

The `zip` command on Ubuntu is 6.0, which was released in 2009 and does not support zstd. It does support bzip2 though!

i meant the "zip support" (as in zlib) with zstd command.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#199

Post by dahrkael »

Am4TIfIsER0ppos wrote: post #91
redeeman wrote: post #81

Earlier quoted context omitted.

there are A LOT of zip files using lzma in the wild. also, how about people learn to use updated software? should newer video compression technologies not be allowed in mkv/mp4. if you cant open it, well.. then stop using 90ies winzip

mkv or mp4 with h264 and aac is good enough. mp3 is good enough. jpeg is good enough. zip with deflate is also good enough.

I started using winrar because winzip wouldnt fit in a floppy disk. so even in the 90s zip wasnt good enough

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#200

Post by birksherty »

abhinavk wrote: post #12
sammy2255 wrote: post #11

Earlier quoted context omitted.

https://github.com/mcmilk/7-Zip-zstd

https://github.com/M2Team/NanaZip It includes the above patches as well as few QoL features.

Tried it, compression is worse compared to 7-Zip-zstd by mcmilk for zstd, for same speed. The removal of text from toolbar icons is enough for me to never use it again. 7zip can change file associations directly with it and very easy. Feels like NanaZip is worse in QoL than 7zip.
Post reply on HN