I worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.
7-Zip for Windows can now use more than 64 CPU threads for compression
11–20 of 220 posts
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#12I worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.
https://github.com/mcmilk/7-Zip-zstd
https://github.com/M2Team/NanaZip
It includes the above patches as well as few QoL features.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#13I worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.
Not that many people care about zstd; I would assume most 7-zip users care about the convenience of the gui.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#14Earlier quoted context omitted.
The linked Processor Group documentation also says: > Applications that do not call any functions that use processor affinity masks or processor numbers will operate correctly on all systems, regardless of the number of processors. I suspect the limitation 7zip encountered was in how it checked how many logical processors a system has, to determine how many threads to spawn. GetActiveProcessorCount can tell you how m…
It isn't just detecting the extra logical processors, you have to do work to utilise them. From the linked text: "If there are more than one processor group in Windows (on systems with more than 64 cpu threads), 7-Zip distributes running CPU threads across different processor groups." The OS does not do that for you under Windows. Other OSs handle that many cores differently. > more than 15 years ago, they probably c…
Are you sure that if you don't attempt to set any affinities, Windows won't schedule 64+ threads over other processor groups? I don't have any system handy that'll produce more than 64 logical processors to test this, but I'd be surprised if Windows' scheduler won't distribute a process's threads over other processor groups if you exceed the number of cores in the group it launches into.
The referenced text suggests applications will "work", but that isn't really explicit.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#15Earlier quoted context omitted.
The linked Processor Group documentation also says: > Applications that do not call any functions that use processor affinity masks or processor numbers will operate correctly on all systems, regardless of the number of processors. I suspect the limitation 7zip encountered was in how it checked how many logical processors a system has, to determine how many threads to spawn. GetActiveProcessorCount can tell you how m…
It isn't just detecting the extra logical processors, you have to do work to utilise them. From the linked text: "If there are more than one processor group in Windows (on systems with more than 64 cpu threads), 7-Zip distributes running CPU threads across different processor groups." The OS does not do that for you under Windows. Other OSs handle that many cores differently. > more than 15 years ago, they probably c…
That depends on what format you're using. Zip compresses every file separately. Bzip and zstd have pretty small maximum block sizes and gzip doesn't gain much from large blocks anyway. And even when you're making large blocks, you can dump a lot of parallelism into searching for repeat data.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#16Earlier quoted context omitted.
It isn't just detecting the extra logical processors, you have to do work to utilise them. From the linked text: "If there are more than one processor group in Windows (on systems with more than 64 cpu threads), 7-Zip distributes running CPU threads across different processor groups." The OS does not do that for you under Windows. Other OSs handle that many cores differently. > more than 15 years ago, they probably c…
Are you sure that if you don't attempt to set any affinities, Windows won't schedule 64+ threads over other processor groups? I don't have any system handy that'll produce more than 64 logical processors to test this, but I'd be surprised if Windows' scheduler won't distribute a process's threads over other processor groups if you exceed the number of cores in the group it launches into. The referenced text suggests…
They're either wrong or thinking about windows 7/8/10. That page is quite clear.
> starting with Windows 11 and Windows Server 2022 the OS has changed to make processes and their threads span all processors in the system, across all processor groups, by default.
> Each process is assigned a primary group at creation, and by default all of its threads' primary group is the same. Each thread's ideal processor is in the thread's primary group, so threads will preferentially be scheduled to processors on their primary group, but they are able to be scheduled to processors on any other group.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#177-zip is one of the software that I miss since I’ve moved to macOS
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#18Why was there a limitation on Windows? I can't find any such limit for Linux.
A lot of synchronization primitives in the NT kernel are based on a register width bit mask of a CPU set, so each collection of 64 hardware threads on 64 bit systems kind of runs in its own instance of the scheduler. It's also unfortunately part of the driver ABI since these ops were implemented as macros and inline functions. Because of that, transitioning a software thread to another processor group is a manual pro…
Wow. That's surprisingly lame.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#19I worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.
Being a bit faster or efficient won't make most people switch. 7z offers great UX (convenient GUI and support for many formats) that keeps people around.
Re: 7-Zip for Windows can now use more than 64 CPU threads for compression
#207-zip is one of the software that I miss since I’ve moved to macOS
How about PeaZip?