Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do Not Use Task Manager for Memory Info

mahdytech.com

Post reply on HN
Print view View original

1–10 of 41 posts

Re: Do Not Use Task Manager for Memory Info

#3

Post by saagarjha »

This article seems to have a high-level overview of how virtual memory works, without mentioning it at all…I find the terminology use rather strange too. Saying that the virtual address space as is “reserved by the OS for each process” was a bit confusing to to me.

Re: Do Not Use Task Manager for Memory Info

#5

Post by [deleted] »

saagarjha wrote: post #3

This article seems to have a high-level overview of how virtual memory works, without mentioning it at all…I find the terminology use rather strange too. Saying that the virtual address space as is “reserved by the OS for each process” was a bit confusing to to me.

[deleted]

Re: Do Not Use Task Manager for Memory Info

#9

Post by quotemstr »

The bigger problem is that there's no good number that captures the memory impact of a process on modern unified-memory page-cache-ful architectures. Practically nobody gets this right, and the author of the article is himself glosses over some important details. Every byte of address space is either allocated and backed by some memory object ("reserved" memory) or it's unallocated. Commit charge is a measure of the amount of memory that the system has guaranteed will be available in the worst case, should all faultable address ranges be faulted, but that's not the same thing as memory actually being used. For example, if you MapViewOfFile a 1GB file PAGE_READONLY, you burn very little commit --- just enough for the page metadata --- but if you change page protections to PAGE_WRITECOPY, then you incur an extra 1GB of commit charge, even though you still haven't faulted anything into memory or reserved more address space --- and that's because you could legally COW-fault every page in that file, and the kernel has to commit (thus the word) to providing that memory in the worst case.

I work on Linux these days, which is even more confusing, because thanks to overcommit, most people don't distinguish these different kinds of memory allocation, even though the distinction between commit and reserved memory exists on Linux too. (The kernel just lies about satisfying commit charges unless you tell it not to lie to you. Most people are happy with overcommit's optimism.)

Anyway, the key thing to realize about modern virtual memory subsystems is that "memory consumption" is an incoherent concept. You can derive lots of different numbers from memory management statistics, but each of these numbers is useful for a specific purpose. There is no one number that will give you an accurate measure of the impact of a particular process in all scenarios. People constantly say, "Look: just give me a number that I can plot on a dashboard and drive down over time". No such thing exists.

Task manager has to pick one of these numbers to show users by default, and its choice, roughly equivalent to Linux Private_Dirty, probably isn't terrible, since it's a decent proxy for how much RAM you get back if you kill the process. I don't think total commit charge is as good a choice, since with a large pagefile (which everyone should have) total commit can be much larger than total resident memory. Linux PSS is another popular choice, since (unlike Private_Dirty) it reflects the impact of a program's use of shared memory, but PSS behaves in perverse ways --- e.g., starting an instance of memory-hungry process can make PSS decrease because some pages in this program are distributed across more processes, increasing the denominator in the PSS calculation.

Are you worried about running out of page file space? Yes, you want to look at commit. Are you wondering why you're seeing a large number of page faults starting a game? Commit won't help you, but RSS might. It really depends on the situation.

I wouldn't take the advice in the article at face value. If you want to understand the impact a particular program has on the system's memory behavior, you need to understand how the virtual memory system actually behaves, and that's non-trivial.

Re: Do Not Use Task Manager for Memory Info

#10

Post by fencepost »

For versions of Windows before 10, I'd agree with this 100%. Windows 10 (or maybe 8-8.1?) pulled in a lot of functionality from Process Explorer and is much improved.

For developers Process Explorer (and ProcMon and a few other utils) is likely an improvement, but frankly if you're doing Windows development you should already have learned about them and probably some of Nir Sofer's tools as well (nirsoft.net). For 90% of people (even developers) you probably don't need what Process Explorer provides.

Side note, in Process Explorer if you turn on the lower pane (View menu or Ctrl-L) you can view all handles that a process has open, including file handles. That can be useful for identifying unrecognized processes.

Post reply on HN