7-Zip for Windows can now use more than 64 CPU threads for compression
1–10 of 220 posts
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#2Why was there a limitation on Windows? I can't find any such limit for Linux.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#3Why was there a limitation on Windows? I can't find any such limit for Linux.
Seems like this is a general Windows thing per https://learn.microsoft.com/en-us/windows/win32/procthread/p... - applications that want to run on more than 64 CPUs need to be written with dedicated support for doing so.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#4Why was there a limitation on Windows? I can't find any such limit for Linux.
Windows has a concept of processor groups, that can have up to 64 (hardware) threads. I assume they updated 7zip to support multiple processor groups.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#5Why was there a limitation on Windows? I can't find any such limit for Linux.
[deleted]
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#6Why was there a limitation on Windows? I can't find any such limit for Linux.
The linked Processor Group documentation also says:
> Applications that do not call any functions that use processor affinity masks or processor numbers will operate correctly on all systems, regardless of the number of processors.
I suspect the limitation 7zip encountered was in how it checked how many logical processors a system has, to determine how many threads to spawn. GetActiveProcessorCount can tell you how many logical processors are on the system if you pass ALL_PROCESSOR_GROUPS, but that API was only added in Windows 7 (that said, that was more than 15 years ago, they probably could've found a moment to add and test a conditional call to it).
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#7Why was there a limitation on Windows? I can't find any such limit for Linux.
Maybe WaitForMultipleObjects limit of 64 (MAXIMUM_WAIT_OBJECTS) applies?
An ugly limitation on an API that initially looks superior to Linux equivalents.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#8Why was there a limitation on Windows? I can't find any such limit for Linux.
A lot of synchronization primitives in the NT kernel are based on a register width bit mask of a CPU set, so each collection of 64 hardware threads on 64 bit systems kind of runs in its own instance of the scheduler. It's also unfortunately part of the driver ABI since these ops were implemented as macros and inline functions.
Because of that, transitioning a software thread to another processor group is a manual process that has to be managed by user space.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#9Why was there a limitation on Windows? I can't find any such limit for Linux.
The linked Processor Group documentation also says: > Applications that do not call any functions that use processor affinity masks or processor numbers will operate correctly on all systems, regardless of the number of processors. I suspect the limitation 7zip encountered was in how it checked how many logical processors a system has, to determine how many threads to spawn. GetActiveProcessorCount can tell you how m…
It isn't just detecting the extra logical processors, you have to do work to utilise them. From the linked text:
"If there are more than one processor group in Windows (on systems with more than 64 cpu threads), 7-Zip distributes running CPU threads across different processor groups."
The OS does not do that for you under Windows. Other OSs handle that many cores differently.
> more than 15 years ago, they probably could've found a moment to add and test a conditional call to it
I suspect it hasn't been an issue much at all until recently. Any single block of data worth spinning up that many threads for compressing is going to be very large, you don't want to split something into too small chunks for compression or you lose some benefit of the dynamic compression dictionary (sharing that between threads would add a lot of inter-thread coordination work, killing any performance gain even if the threads are running local enough on the CPU to share cache). Compression is not an inherently parallelizable task, at least not “embarrassingly” so like some processes.
Even when you do have something to compress that would benefit for more than 64 separate tasks in theory, unless it is all in RAM (or on an incredibly quick & low latency drive/array) the process is likely to be IO starved long before it is compute starved, when you have that much compute resource to hand.
Recent improvements in storage options & CPUs (and the bandwidth between them) have presumably pushed the occurrences of this being worthwhile (outside of artificial tests) from “practically zero” to “near zero, but it happens”, hence the change has been made.
Note that two or more 7-zip instances working on different data could always use more than 64 threads between them, if enough cores to make that useful were available.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#10I worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.