Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do Not Use Task Manager for Memory Info

mahdytech.com

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Do Not Use Task Manager for Memory Info

#21

Post by dijit »

AGoodName wrote: post #11

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use? I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB o…

OOOOOOO, something I actually know!

> Now why would you want to know the committed memory over the actual physical RAM in use?

Because in Windows, committed size is relative to physical size. You can commit a lot more than RAM, but watch your page-file grow.

malloc() can fail on Windows for this reason. This is not the same on Linux or any of the BSD's I've tried. :)

I experienced/discovered this August last year.. sometimes understanding a lot about Linux can make you blind to the architectural differences Windows has.

I wrote some cross-platform CPP to show it[0]

[0]: https://gist.github.com/dijit/cb2caa1a40d48e03613f5af0e518d6...

Re: Do Not Use Task Manager for Memory Info

#22

Post by AGoodName »

dijit wrote: post #21
AGoodName wrote: post #11

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use? I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB o…

OOOOOOO, something I actually know! > Now why would you want to know the committed memory over the actual physical RAM in use? Because in Windows, committed size is relative to physical size. You can commit a lot more than RAM, but watch your page-file grow. malloc() can fail on Windows for this reason. This is not the same on Linux or any of the BSD's I've tried. :) I experienced/discovered this August last year.. s…

This doesn't occur if you memory map a file though (barring certain flags that you can set as stated by quotemstr below)

You can legitimately have Windows stating many GB's of committed RAM without actually using that RAM and it's not using the systems pagefile/swap. It's also common for this to occur. Pretty much every program capable of opening large files (GB+) in a non-sequential fashion does this.

Re: Do Not Use Task Manager for Memory Info

#23

Post by dijit »

AGoodName wrote: post #22
dijit wrote: post #21

Earlier quoted context omitted.

OOOOOOO, something I actually know! > Now why would you want to know the committed memory over the actual physical RAM in use? Because in Windows, committed size is relative to physical size. You can commit a lot more than RAM, but watch your page-file grow. malloc() can fail on Windows for this reason. This is not the same on Linux or any of the BSD's I've tried. :) I experienced/discovered this August last year.. s…

This doesn't occur if you memory map a file though (barring certain flags that you can set as stated by quotemstr below) You can legitimately have Windows stating many GB's of committed RAM without actually using that RAM and it's not using the systems pagefile/swap. It's also common for this to occur. Pretty much every program capable of opening large files (GB+) in a non-sequential fashion does this.

Hm, maybe I wasn't clear. It's not actually /using/ the memory when it's committed.

But the sum of your committed memory across all applications must exist in some form on the host system.

So, for example it's a common performance optimisation to double the amount of allocated space when you grow anything in C++, what this means is that you're not actually using the space yet but malloc() and zeroing is kinda slow.

So, you have 128MB of ram which is your programs address space and you just doubled your array from 75MB to 150MB, well, that extra 22MB must exist. Even though you're only using 76MB.. even if the OS shows it as free. (which, it will)

Thems the rules, and I promise you that I have thoroughly tested this; as it was causing a really nice crash on my servers even though we had more than 50% of memory "free"

Re: Do Not Use Task Manager for Memory Info

#24

Post by AGoodName »

dijit wrote: post #23
AGoodName wrote: post #22

Earlier quoted context omitted.

This doesn't occur if you memory map a file though (barring certain flags that you can set as stated by quotemstr below) You can legitimately have Windows stating many GB's of committed RAM without actually using that RAM and it's not using the systems pagefile/swap. It's also common for this to occur. Pretty much every program capable of opening large files (GB+) in a non-sequential fashion does this.

Hm, maybe I wasn't clear. It's not actually /using/ the memory when it's committed. But the sum of your committed memory across all applications must exist in some form on the host system. So, for example it's a common performance optimisation to double the amount of allocated space when you grow anything in C++, what this means is that you're not actually using the space yet but malloc() and zeroing is kinda slow. S…

Memory Mapped files do exist in some form or another though. As the file itself. That's the point of memory mapping files. You can right now memory map every file on your computer. That's TB of files. There will be no physical RAM usage and no swap file usage unless you start to actually work with those files (at which point they will be paged in). This will show as 'committed memory' in task manager.

Your example above isn't memory mapping files. It's just allocating RAM. That does have to exist in RAM or the swap file. But that's not what 'committed memory' above shows. Which is the whole point. The column the article is telling people to use is misleading.

Re: Do Not Use Task Manager for Memory Info

#25

Post by MarkSweep »

saagarjha wrote: post #3

This article seems to have a high-level overview of how virtual memory works, without mentioning it at all…I find the terminology use rather strange too. Saying that the virtual address space as is “reserved by the OS for each process” was a bit confusing to to me.

The article confuses address space and memory reservations as you note. It again makes the confusion when it suggests the virtual size column in Process Explorer is a reflection of the address space[1]. It also suggests that working set, private bytes, and committed memory are the same thing[2]. The article does not even live up to its click-baity title: Task Manager's default memory column of "private working set" is a decent measure of how much memory a process is uniquely using and task manager has the ability to add all the other measures of memory usage that the article mentions.

For a better explanation of virtual memory in Windows, I recommend Mark Russinovich's article[3]. His tool VMMap[4] is useful for visualizing the memory usage of an individual process.

[1]: The reserved memory is really large for 64-bit processes that use Control Flow Gaurd: http://www.alex-ionescu.com/?p=246

[2]: Task Manager and Process Explorer add to the confusion by calling the same memory different things (Process Explorer's "private bytes" number is the same as Task Manager's "commit size" number on Windows 10 1809).

[3]: https://blogs.technet.microsoft.com/markrussinovich/2008/11/...

[4]: https://docs.microsoft.com/en-us/sysinternals/downloads/vmma...

Re: Do Not Use Task Manager for Memory Info

#26

Post by gameswithgo »

AGoodName wrote: post #11

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use? I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB o…

>Task Manager is doing the right thing here

I had a problem on my windows 10 pc for a long time, where I would clearly be running out of ram, task manager would show 100% usage, everything getting slow. However if I added up all of the processes using ram, it was nowhere close.

So something was using up ram that task manager gave me no visibility into. I had to download obscure tools and do some guesswork to figure it out. It would have been really nice if Task Manager could just report it in the first place (it turned out to be a network card driver with a memory leak in it)

Re: Do Not Use Task Manager for Memory Info

#28

Post by fphhotchips »

gameswithgo wrote: post #26
AGoodName wrote: post #11

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use? I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB o…

>Task Manager is doing the right thing here I had a problem on my windows 10 pc for a long time, where I would clearly be running out of ram, task manager would show 100% usage, everything getting slow. However if I added up all of the processes using ram, it was nowhere close. So something was using up ram that task manager gave me no visibility into. I had to download obscure tools and do some guesswork to figure i…

Yep, this is particularly evident if you use Virtualbox a lot - the memory use isn't in the actual process but in one of the drivers. Has also happened to me with a VPN drivers.

Best tool I've seen to start finding where it's going is RAMmap [1].

1. https://docs.microsoft.com/en-us/sysinternals/downloads/ramm...

Re: Do Not Use Task Manager for Memory Info

#29

Post by greglindahl »

dijit wrote: post #21
AGoodName wrote: post #11

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use? I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB o…

OOOOOOO, something I actually know! > Now why would you want to know the committed memory over the actual physical RAM in use? Because in Windows, committed size is relative to physical size. You can commit a lot more than RAM, but watch your page-file grow. malloc() can fail on Windows for this reason. This is not the same on Linux or any of the BSD's I've tried. :) I experienced/discovered this August last year.. s…

> This is not the same on Linux

Linux lets you choose an overcommit policy. https://www.kernel.org/doc/Documentation/vm/overcommit-accou...

Re: Do Not Use Task Manager for Memory Info

#30

Post by saltyshake »

I am surprised by how little thought MS always put into the Task Manager compared to every other process manager for Windows, including MSs own Process Explorer.

Even on Windows Server 2016, it doesn't show the per-process memory correctly for processes using over 128 GB.

Had to learn that the hard way when the overall RAM usage was pretty high but I couldn't find any individual process using that much. Then opened Process Explorer and boom, SQL Server was using over 130 GB (cubes..).

Post reply on HN