Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do Not Use Task Manager for Memory Info

mahdytech.com

Post reply on HN
Print view View original

11–20 of 41 posts

Re: Do Not Use Task Manager for Memory Info

#11

Post by AGoodName »

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use?

I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB of committed memory aren't something you should care about. I'd be scared if someone looked at the committed memory use of a program that correctly uses mmap and caused someone to exclaim "OMG this is uses TB of RAM!".

Task Manager is doing the right thing here. It's showing you want's actually paged in and in use right now.

Re: Do Not Use Task Manager for Memory Info

#12

Post by quotemstr »

AGoodName wrote: post #11

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use? I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB o…

> committed memory is that in use + memory that's being paged out

That's not actually true though. You can see an increase in a process's commit charge without anything new being written to the pagefile. Commit is a check that the kernel writes to applications; you're confusing that check for cash in a wallet. You can also have commit without any virtual address space to blame for it through section handles or other tricks. It's complicated.

Re: Do Not Use Task Manager for Memory Info

#13

Post by AGoodName »

quotemstr wrote: post #12
AGoodName wrote: post #11

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use? I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB o…

> committed memory is that in use + memory that's being paged out That's not actually true though. You can see an increase in a process's commit charge without anything new being written to the pagefile. Commit is a check that the kernel writes to applications; you're confusing that check for cash in a wallet. You can also have commit without any virtual address space to blame for it through section handles or other…

I'm not going to dispute that but just want to highlight it doesn't change the fact that committed RAM can show as extremely high just by working with memory mapped files.

Memory mapping a GB log file for example will absolutely show GB's of committed memory but in reality you'll only have the last page in actual physical RAM.

Re: Do Not Use Task Manager for Memory Info

#14

Post by quotemstr »

AGoodName wrote: post #13
quotemstr wrote: post #12

Earlier quoted context omitted.

> committed memory is that in use + memory that's being paged out That's not actually true though. You can see an increase in a process's commit charge without anything new being written to the pagefile. Commit is a check that the kernel writes to applications; you're confusing that check for cash in a wallet. You can also have commit without any virtual address space to blame for it through section handles or other…

I'm not going to dispute that but just want to highlight it doesn't change the fact that committed RAM can show as extremely high just by working with memory mapped files. Memory mapping a GB log file for example will absolutely show GB's of committed memory but in reality you'll only have the last page in actual physical RAM.

Right. When you memory-map a file, what you've essentially done is add a temporary new pagefile to the system (your mapped file), and when you work with memory backed by that file, it's no different from working with "anonymous" memory backed by the system-wide pagefile.

Re: Do Not Use Task Manager for Memory Info

#15

Post by stdgy »

quotemstr wrote: post #9

The bigger problem is that there's no good number that captures the memory impact of a process on modern unified-memory page-cache-ful architectures. Practically nobody gets this right, and the author of the article is himself glosses over some important details. Every byte of address space is either allocated and backed by some memory object ("reserved" memory) or it's unallocated. Commit charge is a measure of the…

Lovely post, but I now realize that I am rather ignorant of the interplay between processes and the kernel with regards to memory allocation.

Any ideas on cool stuff to read to remedy this?

Re: Do Not Use Task Manager for Memory Info

#16

Post by rococode »

fencepost wrote: post #10

For versions of Windows before 10, I'd agree with this 100%. Windows 10 (or maybe 8-8.1?) pulled in a lot of functionality from Process Explorer and is much improved. For developers Process Explorer (and ProcMon and a few other utils) is likely an improvement, but frankly if you're doing Windows development you should already have learned about them and probably some of Nir Sofer's tools as well (nirsoft.net). For 90…

Here's the link to Process Explorer: https://docs.microsoft.com/en-us/sysinternals/downloads/proc...

It also has an option to replace Task Manager so that it comes up when you do Ctrl+Shift+Esc, etc.

Re: Do Not Use Task Manager for Memory Info

#17

Post by _cs2017_ »

quotemstr wrote: post #12
AGoodName wrote: post #11

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use? I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB o…

> committed memory is that in use + memory that's being paged out That's not actually true though. You can see an increase in a process's commit charge without anything new being written to the pagefile. Commit is a check that the kernel writes to applications; you're confusing that check for cash in a wallet. You can also have commit without any virtual address space to blame for it through section handles or other…

I understand commit can happen without physical page file being written to. But you say commit can happen even without any increase in virtual address space usage? That seems strange, could you explain how it can happen?

Re: Do Not Use Task Manager for Memory Info

#18

Post by quotemstr »

_cs2017_ wrote: post #17
quotemstr wrote: post #12

Earlier quoted context omitted.

> committed memory is that in use + memory that's being paged out That's not actually true though. You can see an increase in a process's commit charge without anything new being written to the pagefile. Commit is a check that the kernel writes to applications; you're confusing that check for cash in a wallet. You can also have commit without any virtual address space to blame for it through section handles or other…

I understand commit can happen without physical page file being written to. But you say commit can happen even without any increase in virtual address space usage? That seems strange, could you explain how it can happen?

Change a PAGE_READONLY mapping to a PAGE_WRITECOPY one. The commit charge is billed at VirtualProtect time, and the protection can fail if you run out of commit. The kernel doesn't have to commit anything for a PAGE_READONLY mapping because all pages in such a mapping are guaranteed to be clean and trivially evictable. Not so once you introduce the possibility of COW faults. Reserving a big range of address space and committing it a little bit at a time is a very common pattern.

Another thing: create a 1GB section. Map it, and fill it up. Unmap it. Map it again. What you wrote is still there. Between the map and remap, you have commit without corresponding address space.

Post reply on HN