Hmm, that java process has 0.227 TB of mapped virtual memory and 3090% CPU..
and they run it as root
Linux Performance Analysis
11–20 of 84 posts
Re: Linux Performance Analysis
#12The real gem for me in that article (particularly since I already knew about those linux commands) was the USE method [1] and then consequently finding the TSA method [2] on the same linked site.
Re: Linux Performance Analysis
#13that's a pretty awesome summary. to that i would add: * `iftop` that allows me to quickly check which network streams are hogging the machine (this is a little like sar -n DEV 1 but much more detailed!) * `tcpdump -c 100 -vv`- poor man's alternative to iftop or systat if they're not available locally * tail those logfiles, with systemd it's even easier with journald as all logfiles can be checked at once: `journalctl…
I'll add to your list with htop. Its a little nicer than plain ole top.
Re: Linux Performance Analysis
#14Hmm, that java process has 0.227 TB of mapped virtual memory and 3090% CPU..
You sound like you doubt the readings. This is netflix, I assume they have some pretty hefty kit backing up their service. For a massive server the CPU reading might not be unusual. Maybe it has 32+ CPU cores and a multi-threaded java app is spinning most of them. Also remember that on a heavily loaded system the task that is reading CPU use is itself competeing for time. Timing issues and other monitoring vagueries…
The 233GB is VIRT, meaning virtual memory. It need not all be backed by physical memory. For example, if you mmap a file, and then access only a small portion.
The RES column shows how much memory is resident (eg, currently backed by physical memory), and that is a much more reasonable 12GB.
Re: Linux Performance Analysis
#15Hmm, that java process has 0.227 TB of mapped virtual memory and 3090% CPU..
You sound like you doubt the readings. This is netflix, I assume they have some pretty hefty kit backing up their service. For a massive server the CPU reading might not be unusual. Maybe it has 32+ CPU cores and a multi-threaded java app is spinning most of them. Also remember that on a heavily loaded system the task that is reading CPU use is itself competeing for time. Timing issues and other monitoring vagueries…
Probably r3.8xlarge.
Re: Linux Performance Analysis
#16Hmm, that java process has 0.227 TB of mapped virtual memory and 3090% CPU..
You sound like you doubt the readings. This is netflix, I assume they have some pretty hefty kit backing up their service. For a massive server the CPU reading might not be unusual. Maybe it has 32+ CPU cores and a multi-threaded java app is spinning most of them. Also remember that on a heavily loaded system the task that is reading CPU use is itself competeing for time. Timing issues and other monitoring vagueries…
The r3.8xlarge instance type has 32 vCPUs and 244GiB RAM
Re: Linux Performance Analysis
#17Earlier quoted context omitted.
You sound like you doubt the readings. This is netflix, I assume they have some pretty hefty kit backing up their service. For a massive server the CPU reading might not be unusual. Maybe it has 32+ CPU cores and a multi-threaded java app is spinning most of them. Also remember that on a heavily loaded system the task that is reading CPU use is itself competeing for time. Timing issues and other monitoring vagueries…
Probably r3.8xlarge.
jinx!
Re: Linux Performance Analysis
#18I'm wondering if there's a way to automate this kind of analysis to give you condensed, interesting points of a system performance status. À la powertop for power management.
Agreed, considering they've built Vector (mentioned by bgregg in the second sentence - https://github.com/Netflix/vector) on top of Performance Co-Pilot. While PCP doesn't yet have all the wrappers to mimic each sysstat output in a fully compat manner. The underlying mechanisms to remotely fetch that data (using the tools Vector is already built on), is already there.
Re: Linux Performance Analysis
#19that's a pretty awesome summary. to that i would add: * `iftop` that allows me to quickly check which network streams are hogging the machine (this is a little like sar -n DEV 1 but much more detailed!) * `tcpdump -c 100 -vv`- poor man's alternative to iftop or systat if they're not available locally * tail those logfiles, with systemd it's even easier with journald as all logfiles can be checked at once: `journalctl…
I'll add to your list with htop. Its a little nicer than plain ole top.
And `glances` is pretty great for big-picture stuff, all-in-one. It's not as precise on CPU/memory as htop, but you get network bandwidth, disk I/O, disk usage and more.
Re: Linux Performance Analysis
#20Hmm, that java process has 0.227 TB of mapped virtual memory and 3090% CPU..
Note that's virtual memory. For determining real RAM usage for programs, I find https://github.com/pixelb/ps_mem extremely useful