Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

7-Zip for Windows can now use more than 64 CPU threads for compression

7-zip.org

Post reply on HN
Print view View original

51–60 of 220 posts

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#51

Post by justin66 »

cesarb wrote: post #37
dikei wrote: post #29

Earlier quoted context omitted.

I use ZSTD a ton in my programming work where efficiency matters. But for sharing files with other people, ZIP is still king. Even 7z or RAR is niche. Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger.

> Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger. You can use ZSTD with ZIP files too! It's compression method 93 (see https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT which is the official ZIP file specification). Which reveals that "everyone can open a ZIP file" is a lie. Sure, everyone can open a ZIP file, as long as that file uses only a limited subset of the Z…

> Copyright (c) 1989 - 2014, 2018, 2019, 2020, 2022

Mostly it seems nutty that, after all these years, they’re still updating the zip spec instead of moving on to a newer format.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#52

Post by Beretta_Vexee »

d33 wrote: post #24
sammy2255 wrote: post #11

Earlier quoted context omitted.

https://github.com/mcmilk/7-Zip-zstd

Thanks! Any ideas why it didn't get merged? Clearly 7-Zip has some development activity going on and so does this fork...

Working with Igor Pavlov, the creator of 7-zip, does not seem very straightforward (understatement).

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#53

Post by monocasa »

zamadatix wrote: post #45
monocasa wrote: post #40

Earlier quoted context omitted.

There were single image systems with hundreds of cores in the late 90s and thousands of cores in the early 2000s. I absolutely stand by the fact that Intel and AMD didn't pursue high core count systems until that point because they were so focused on single core perf, in part because Windows didn't support high core counts. The end of Denmark scing forced their hand and Microsoft's processor group hack.

Do you have anything to say regarding NUMA for the 90s core counts though? As I said, it's not enough that there were a lot of cores - they have to be monolithically scheduled to matter. The largest UMA design I can recall was the CS6400 in 1993, to go past that they started to introduce NUMA designs.

Windows didn't handle numa either until they created processor groups, and there's all sorts reasons why you'd want to run a process (particularly on Windows which encourages single process high thread count software archs) that spans numa nodes. It's really not that big if a deal for a lot of workloads where your working set fits just fine in cache, or you take the high hatdware thread count approach of just having enough contexts in flight that you can absorb the extra memory latency in exchange for higher throughput.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#54

Post by [deleted] »

cesarb wrote: post #37
dikei wrote: post #29

Earlier quoted context omitted.

I use ZSTD a ton in my programming work where efficiency matters. But for sharing files with other people, ZIP is still king. Even 7z or RAR is niche. Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger.

> Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger. You can use ZSTD with ZIP files too! It's compression method 93 (see https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT which is the official ZIP file specification). Which reveals that "everyone can open a ZIP file" is a lie. Sure, everyone can open a ZIP file, as long as that file uses only a limited subset of the Z…

[deleted]

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#55

Post by Jackson__ »

jorvi wrote: post #25
rf15 wrote: post #13

Earlier quoted context omitted.

Not that many people care about zstd; I would assume most 7-zip users care about the convenience of the gui.

.. but 7-zip has a pretty terrible GUI? Hence why PeaZip is so popular, and J-Zip used to be before it was stuffed with adware.

PeaZip is popular? It seems a lot less tested than 7zip; Last time I tried to use it, it failed to unpack an archive because the password had a quote character or something like that. Never had such crazy issues in 7zip myself.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#56

Post by monocasa »

Dylan16807 wrote: post #16
dwattttt wrote: post #14

Earlier quoted context omitted.

Are you sure that if you don't attempt to set any affinities, Windows won't schedule 64+ threads over other processor groups? I don't have any system handy that'll produce more than 64 logical processors to test this, but I'd be surprised if Windows' scheduler won't distribute a process's threads over other processor groups if you exceed the number of cores in the group it launches into. The referenced text suggests…

They're either wrong or thinking about windows 7/8/10. That page is quite clear. > starting with Windows 11 and Windows Server 2022 the OS has changed to make processes and their threads span all processors in the system, across all processor groups, by default. > Each process is assigned a primary group at creation, and by default all of its threads' primary group is the same. Each thread's ideal processor is in the…

I mean, it seems it's quite clear that a single process and all of its threads will just be assigned to a single processor group, and it'll take manual work for that process to use more than 64 cores.

The difference is just that processes will be assigned a processor group more or less randomly by default, so they'll be balanced on the process level, but not the thread level. Not super helpful for a lot of software systems on windows which had historically preferred threads to processes for concurrency.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#57

Post by delfinom »

jorvi wrote: post #25
rf15 wrote: post #13

Earlier quoted context omitted.

Not that many people care about zstd; I would assume most 7-zip users care about the convenience of the gui.

.. but 7-zip has a pretty terrible GUI? Hence why PeaZip is so popular, and J-Zip used to be before it was stuffed with adware.

I would never trust PeaZip.

The author updates code in the github repo....by drag and drop file uploads. https://github.com/peazip/PeaZip/commits/sources/

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#59

Post by pornel »

justin66 wrote: post #51
cesarb wrote: post #37

Earlier quoted context omitted.

> Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger. You can use ZSTD with ZIP files too! It's compression method 93 (see https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT which is the official ZIP file specification). Which reveals that "everyone can open a ZIP file" is a lie. Sure, everyone can open a ZIP file, as long as that file uses only a limited subset of the Z…

> Copyright (c) 1989 - 2014, 2018, 2019, 2020, 2022 Mostly it seems nutty that, after all these years, they’re still updating the zip spec instead of moving on to a newer format.

The English language is awful, and we keep updating it instead of moving to a newer language.

Some things are used for interoperability, and switching to a newer incompatible thing loses all of its value.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#60

Post by Dylan16807 »

monocasa wrote: post #56
Dylan16807 wrote: post #16

Earlier quoted context omitted.

They're either wrong or thinking about windows 7/8/10. That page is quite clear. > starting with Windows 11 and Windows Server 2022 the OS has changed to make processes and their threads span all processors in the system, across all processor groups, by default. > Each process is assigned a primary group at creation, and by default all of its threads' primary group is the same. Each thread's ideal processor is in the…

I mean, it seems it's quite clear that a single process and all of its threads will just be assigned to a single processor group, and it'll take manual work for that process to use more than 64 cores. The difference is just that processes will be assigned a processor group more or less randomly by default, so they'll be balanced on the process level, but not the thread level. Not super helpful for a lot of software s…

> it'll take manual work for that process to use more than 64 cores.

No it won't.

Post reply on HN