Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

7-Zip for Windows can now use more than 64 CPU threads for compression

7-zip.org

Post reply on HN
Print view View original

171–180 of 220 posts

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#171

Post by sidewndr46 »

xxs wrote: post #105
sidewndr46 wrote: post #67

Earlier quoted context omitted.

What are you compressing with zstd? I had to do this recently and the "xz" utility still blows it away in terms of compression ratio. In terms of memory and CPU usage, zstd wins by a large margin. But in my case I only really cared about compression ratio

do you have examples where xz 'blows it away', not just zstd -3?

Here are some examples of what I was doing in one case

https://www.hydrogen18.com/blog/apk-the-strangest-format.htm...

I was running "zstd --ultra --threads=0" which I assumed was asking it for the absolute maximum

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#172

Post by sidewndr46 »

vlovich123 wrote: post #86
sidewndr46 wrote: post #67

Earlier quoted context omitted.

What are you compressing with zstd? I had to do this recently and the "xz" utility still blows it away in terms of compression ratio. In terms of memory and CPU usage, zstd wins by a large margin. But in my case I only really cared about compression ratio

people tend to care about decompression speed - xz can be quite slow decompressing super compressed files whereas zstd decompression speed is largely independent of that. People also tend to care about how much time they spend on compression for each incremental % of compression performance and zstd tends to be a Pareto frontier for that (at least for open source algorithms)

I definitely agree, I basically have unlimited time and unlimited CPU for decompressing. Available memory is huge too. The gains from xz were significant enough that I went with it.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#173

Post by martinald »

danudey wrote: post #166
LegionMammal978 wrote: post #150

Earlier quoted context omitted.

It's similarly annoying how many websites take the existence of the lossy format as a license to recompress all WebP uploads, or sometimes other filetypes converted to WebP, even when it causes the filesize to increase. It's like we're returning to ye olden days of JPEG artifacts on every screenshot.

I was thinking about this with YouTube as an example. A lot of people complain about the compression on YouTube videos making things look awful, but I bet there's a reasonable number of high-end content creators out there who would run a native(-ish, probably Electron) app on their local system to do a higher-quality encoding to YouTube's specifications before uploading. In many (most?) cases, it's possible to get be…

AFIAK you can upload any bitrate to youtube as long as the file is So you could upload a crazy high bitrate file to them for a 20 min video which I suspect would be close to "raw" quality.

I don't know how many corners youtube cut on encoding though.

I suspect most of the problem is people exporting 4k at a 'web' bitrate preset (15mbit/s?), which is actually gonna get murdered on the 2nd encode more than encoding quality on youtubes side?

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#174

Post by notepad0x90 »

landl0rd wrote: post #146
notepad0x90 wrote: post #62

Earlier quoted context omitted.

I don't know about, had a dicey situation recently where powershell's compress-archive couldn't handle archives >4GB and had to use 7zip. it is more reliable and you can ship 7za.exe or create self-extracting archives (wish those were more of a thing outside of the windows world).

I understand that security has to compromise for the real world, but a self-extracting archive is possibly one of the worst things one could use in terms of security.

why? and why does it have to be a compromise?

You're assuming things because things are already done insecurely. You can authenticate the self-extractor as well as the extracted content. The user gets a nice message "This is a 7zip self-extracting archive sent to you by Bob containing the files below".

As an incident responder, I've seen much more of regular archives being used to social engineer users than self-extracting archives, because self-extracting is not "content executing". it is better for social engineering for users to establish trust in the payload first by having them manually open the archive. if something "weird" like self-extraction happens first, it might feel less trustworthy.

Oh and by the way, things like PyInstaller or electron apps are already self-extracting and self-executing archives. So are JAR files and android APK's.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#175

Post by p_ing »

monocasa wrote: post #164
p_ing wrote: post #138

Earlier quoted context omitted.

"Server systems" but not server systems that Microsoft targeted. NT4 Enterprise Server (1996) only supported up to 8 sockets (some companies wrote their own HAL to exceed that limit). And 8 sockets was 8 threads with no NUMA back then, not something that would have been an issue for the purposes of this discussion.

Microsoft was absolutely wanting to target large servers at the time. They were actively trying to kill off the vendor unices in the 90s.

They successfully killed off vendor unicies in the 90s, but that was thanks to cheap x86.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#176

Post by p_ing »

monocasa wrote: post #56
Dylan16807 wrote: post #16

Earlier quoted context omitted.

They're either wrong or thinking about windows 7/8/10. That page is quite clear. > starting with Windows 11 and Windows Server 2022 the OS has changed to make processes and their threads span all processors in the system, across all processor groups, by default. > Each process is assigned a primary group at creation, and by default all of its threads' primary group is the same. Each thread's ideal processor is in the…

I mean, it seems it's quite clear that a single process and all of its threads will just be assigned to a single processor group, and it'll take manual work for that process to use more than 64 cores. The difference is just that processes will be assigned a processor group more or less randomly by default, so they'll be balanced on the process level, but not the thread level. Not super helpful for a lot of software s…

https://learn.microsoft.com/en-us/windows/win32/procthread/p...

This explicitly says the feature is automatic and programs will not need to manually adjust their affinity.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#177

Post by redeeman »

landl0rd wrote: post #160
redeeman wrote: post #154

Earlier quoted context omitted.

if they want to open certain files, they will update

No, they're just not going to work with you.

im okay with that. That being said, I have not had a single issue delivering zip files with lzma, and i KNOW that I have gotten MANY from the random sources.

I would also expect people to be able to decode h265 in an mp4 file.

Your proposal seems, to word it bluntly, retarded. You would have mp4 frozen for h264 for ETERNITY, and then invent a new format as replacement? or you would just say "god has bestowed upon the world h264, and it shall be the LAST CODEC EVER!".

get with the program. Things change, you cannot expect to be forwards compatible for ever. Sometimes people have to switch to newer versions of software.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#178

Post by redeeman »

philipwhiuk wrote: post #170
redeeman wrote: post #155

Earlier quoted context omitted.

what software doesnt support OSs that are in active SECURITY support?

https://arstechnica.com/gadgets/2023/07/windows-95-98-and-ot...

"Retro-computing fans can download the final updates released for '90s-era OSes."

Right, and when were these "final updates" made? are you suggesting 95, 98 still sees ACTIVE security support?

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#179

Post by okanat »

lofties wrote: post #4
avidiax wrote: post #2

Why was there a limitation on Windows? I can't find any such limit for Linux.

Windows has a concept of processor groups, that can have up to 64 (hardware) threads. I assume they updated 7zip to support multiple processor groups.

Every operating system, that's relevant in 2025, needs that concept. It is called NUMA. At some point you cannot model the system ignoring the memory affinity / closeness to the cores.

Modern AMD CPUs are literally consist of core groups on chiplets. It is better for an OS to make decisions / expose APIs for cores that are physically so far away from each other that moving data back-and-forth over the RAM, system bus or interconnect has significant time penalties.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#180

Post by sltkr »

sidewndr46 wrote: post #171
xxs wrote: post #105

Earlier quoted context omitted.

do you have examples where xz 'blows it away', not just zstd -3?

Here are some examples of what I was doing in one case https://www.hydrogen18.com/blog/apk-the-strangest-format.htm... I was running "zstd --ultra --threads=0" which I assumed was asking it for the absolute maximum

I think your mistake was to use --ultra without a compression level.

I redid your experiments with rust-wasm-1.83.0-r0.apk:

                            size       perc   c.time  d.time
    uncompressed:      290072064          -        -
    gzipped original:  105255109     36.29%        -  
    bzip2 -9:          107099379     36.92%    21.1s  11.0s
    bzip3 -b511:        73539847     25.35%    28.9s  32.0s
    xz --extreme -9:    71010672     24.48%   142.0s   3.1s
    lzip -9:            70964413     24.46%   173.5s   5.3s
    zstd --ultra -22:   48288499     16.64%   155.6s   0.4s
It's pretty clear zstd blows everything else out of the water by a huge margin. And even though compressing with zstd is slightly slower than xz in this case (by less than 10%), decompression is nearly 8x as fast, and you can probably tweak the compression level to make zstd be both faster and better than xz.
Post reply on HN