Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

7-Zip for Windows can now use more than 64 CPU threads for compression

7-zip.org

Post reply on HN
Print view View original

71–80 of 220 posts

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#71

Post by immibis »

MYEUHD wrote: post #21
aquir wrote: post #17

7-zip is one of the software that I miss since I’ve moved to macOS

If you're talking about the program you use in the terminal, you can install it via homebrew

No, the GUI. 7-zip integrates well with the shell: select a group of files, right click -> make zip file, and so on. Or right-click a zip file and select extract. If you're accustomed to Linux you might not know what they're talking about.

TortoiseGit (and TortoiseSVN) are similarly convenient. Right click a folder with an SVN repo checked out, and select "SVN update". Right-click an empty space, and select "SVN checkout". SVN was the main distribution method for some modding communities before things like Steam Workshop and Github, specifically because TortoiseSVN made it so convenient. Checkout into your addons folder, and periodically update. What could be simpler?

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#72

Post by sidewndr46 »

Const-me wrote: post #28
zik wrote: post #18

Earlier quoted context omitted.

Wow. That's surprisingly lame.

The NT kernel dates back to 1993. Computers didn’t exceed 64 logical processors per system until around 2014. And doing it back then required a ridiculously expensive server with 8 Intel CPUs. The technical decision Microsoft made initially worked well for over two decades. I don’t think it was lame; I believe it was a solid choice back then.

That was actually the DEC team from what I understand, Microsoft just hired all of their OS engineers when they collapsed

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#73

Post by sidewndr46 »

zamadatix wrote: post #38
monocasa wrote: post #30

Earlier quoted context omitted.

I mean, x86 didn't, but other systems had been exceeding 64 cores since the late 90s. And x86 arguably didn't ship >64 hardware thread systems until then because NT didn't support it.

> And x86 arguably didn't ship >64 hardware thread systems until then because NT didn't support it. If that were the case the above system wouldn't have needed 8 sockets. With NUMA systems the app needs to be scheduling group aware anyways. The difference here really appears when you have a single socket with more than 64 hardware threads, which took until ~2019 for x86.

Why would an application need to be NUMA aware on Linux? Most software I've ever written or looked at has no concept of NUMA. It works just fine.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#74

Post by meepmorp »

sidewndr46 wrote: post #72
Const-me wrote: post #28

Earlier quoted context omitted.

The NT kernel dates back to 1993. Computers didn’t exceed 64 logical processors per system until around 2014. And doing it back then required a ridiculously expensive server with 8 Intel CPUs. The technical decision Microsoft made initially worked well for over two decades. I don’t think it was lame; I believe it was a solid choice back then.

That was actually the DEC team from what I understand, Microsoft just hired all of their OS engineers when they collapsed

Dave Cutler left DEC in 1988 and started working on WINNT at MS, well before the collapse.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#75

Post by zx2c4 »

Beretta_Vexee wrote: post #48
d33 wrote: post #10

I worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.

You are looking for 7-Zip Zstd: https://github.com/mcmilk/7-Zip-Zstd I don't know what your use case is, but it seems to be quite a niche.

I was curious upon seeing this and found the thread where its inclusion was turned down: https://sourceforge.net/p/sevenzip/discussion/45797/thread/a...

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#76

Post by axus »

general1726 wrote: post #26
jorvi wrote: post #25

Earlier quoted context omitted.

.. but 7-zip has a pretty terrible GUI? Hence why PeaZip is so popular, and J-Zip used to be before it was stuffed with adware.

Most people won't use that GUI, but will right click file or folder -> 7-Zip -> Add To ... and it will spit out a file without questions. Granted Windows 11 has started doing the same for its zip and 7zip compressors. Same trick goes for opening archives or executables (Installers) as archives.

Let's chat about Windows 11 right-click menu. I'm pretty sure they hid all the application menu extensions to avoid worst-case performance issues.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#77

Post by zamadatix »

monocasa wrote: post #53
zamadatix wrote: post #45

Earlier quoted context omitted.

Do you have anything to say regarding NUMA for the 90s core counts though? As I said, it's not enough that there were a lot of cores - they have to be monolithically scheduled to matter. The largest UMA design I can recall was the CS6400 in 1993, to go past that they started to introduce NUMA designs.

Windows didn't handle numa either until they created processor groups, and there's all sorts reasons why you'd want to run a process (particularly on Windows which encourages single process high thread count software archs) that spans numa nodes. It's really not that big if a deal for a lot of workloads where your working set fits just fine in cache, or you take the high hatdware thread count approach of just having…

3.1 (1993) - KAFFINITY bitmask

5.0 (1999) - NUMA scheduling

6.1 (2009) - Processor Groups to have the KAFFINITY limit be per NUMA node

Xeon E7-8800 (2011) - An x86 system exceeding 64 total cores is possible (10x8 -> requires Processor Groups)

Epyc 9004 (2022) - KAFFINITY has created an artificial limit for x86 where you need to split groups more granular than NUMA

If x86 had actually hit a KAFFINITY wall then the E7-8800 even would have occured years before processor groups because >8 core CPUs are desirable regardless if you can stick 8 in a single box.

The story is really a bit reverse from the claim: NT in the 90s supported architectures which could scale past the KAFFINITY limit. NT in the late 2000s supported scaling x86 but it wouldn't have mattered until the 2010s. Ultimately KAFFINITY wasn't an annoyance until the 2020s.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#78

Post by Tuldok »

d33 wrote: post #24
sammy2255 wrote: post #11

Earlier quoted context omitted.

https://github.com/mcmilk/7-Zip-zstd

Thanks! Any ideas why it didn't get merged? Clearly 7-Zip has some development activity going on and so does this fork...

7-zip's development is very cathedral. Igor Pavlov doesn't look like he accepts contributions from the public.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#79

Post by throw0101d »

cesarb wrote: post #37
dikei wrote: post #29

Earlier quoted context omitted.

I use ZSTD a ton in my programming work where efficiency matters. But for sharing files with other people, ZIP is still king. Even 7z or RAR is niche. Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger.

> Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger. You can use ZSTD with ZIP files too! It's compression method 93 (see https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT which is the official ZIP file specification). Which reveals that "everyone can open a ZIP file" is a lie. Sure, everyone can open a ZIP file, as long as that file uses only a limited subset of the Z…

> You can use ZSTD with ZIP files too!

Support for which was added in 2020:

> On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number 93, deprecating the previous codec number of 20 as it was implemented in version 6.3.7, released on 1 June.[36][37]

* https://en.wikipedia.org/wiki/Zstd#Usage

So I'm not sure how widely deployed it would be.

Re: 7-Zip for Windows can now use more than 64 CPU threads for compression

#80

Post by Dylan16807 »

monocasa wrote: post #64
Dylan16807 wrote: post #60

Earlier quoted context omitted.

> it'll take manual work for that process to use more than 64 cores. No it won't.

It absolutely will. Your process is only assigned a single processor group at process creation time. The only difference now is that it's by default assigned a random processor group rather than inheriting the parent's. For processes that don't require >64 cores, this means better utilization at the system level. However you're still assigned That's literally why 7-zip is announcing completion of that manual work.

The 7zip code needed to change because it was counting cores by looking at affinity masks, and that limits it to 64.

It also needed to change if you want optimal scheduling, and it needed to change if you want it to be able to use all those cores on something that isn't windows 11.

But for just the basic functionality of using all the cores: >Starting with Windows 11 and Windows Server 2022, on a system with more than 64 processors, process and thread affinities span all processors in the system, across all processor groups, by default

That's documentation for a single process messing with its affinity. They're not writing that because they wrote a function to put different processes on different groups. A single process will span groups by default.

Post reply on HN