Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Performance Analysis

techblog.netflix.com

Post reply on HN
Print view View original

51–60 of 84 posts

Re: Linux Performance Analysis

#51

Post by baldfat »

digi_owl wrote: post #38
jandrese wrote: post #30

Earlier quoted context omitted.

Can you imagine a world where the Windows System Event Viewer was this useful? It's amazing how much time one useful error message can save. Also, I'm a little disappointed that the author didn't drill down into what the actual problem was in his example. That java process is of course suspicious, but that might just be the video server and expected behavior.

dmesg and being able to run most things in a terminal to check what its doing (grumble, skype, grumble) is likely what has drawn many to Linux in the first place. And a likely source of misgivings regarding recent developments in the ecosystem...

journalctl is a good tool. I really like SystemD but wish we still had plain text logs.

Re: Linux Performance Analysis

#54

Post by jerf »

Is there any way to analyze memory bandwidth usage?

I've had a couple of programs over the years that I wondered if we were just hitting memory bandwidth limitations but I couldn't find any way to prove that, or even particularly gather evidence.

Re: Linux Performance Analysis

#55

Post by richardwhiuk »

Abundnce10 wrote: post #49

I had one of my EC2 instances (Ubuntu 14.04) lock up on Monday morning. I use it to run a Ruby on Rails app and to do a handful of ETL jobs. The website wasn't able to load and I wasn't able to SSH into the box. So, I went into the AWS console and rebooted the instance. This has happened 3 times in the last 6 months. It looks like the CPU spikes up to almost 100% around the same time when the instance locks up. I'd l…

Check the log files that got written before you killed the instance?

Re: Linux Performance Analysis

#56

Post by CTrox »

Abundnce10 wrote: post #49

I had one of my EC2 instances (Ubuntu 14.04) lock up on Monday morning. I use it to run a Ruby on Rails app and to do a handful of ETL jobs. The website wasn't able to load and I wasn't able to SSH into the box. So, I went into the AWS console and rebooted the instance. This has happened 3 times in the last 6 months. It looks like the CPU spikes up to almost 100% around the same time when the instance locks up. I'd l…

I can recommend atop[1] for that. It runs every ten minutes by default and writes lots of information to /var/log/atop/atop_YYYYMMDD. With that you can examine what happend before the crash, just open a file with atop -r /var/log/atop/atop_YYYYMMDD.

[1] http://linux.die.net/man/1/atop

Re: Linux Performance Analysis

#57

Post by perlgeek »

Abundnce10 wrote: post #49

I had one of my EC2 instances (Ubuntu 14.04) lock up on Monday morning. I use it to run a Ruby on Rails app and to do a handful of ETL jobs. The website wasn't able to load and I wasn't able to SSH into the box. So, I went into the AWS console and rebooted the instance. This has happened 3 times in the last 6 months. It looks like the CPU spikes up to almost 100% around the same time when the instance locks up. I'd l…

One thing we do at $work is collecting each server's static with https://collectd.org/ and often the graphs are pretty revealing. Sometimes it's the disk activity that kills the server, sometimes it's the memory usage, or number of processes, or a disk running full.

For things like databases, you can gather more specific information as well, for example number of queries and connections, failed transactions and the likes.

If you have a hunch what it might be, you could try to counter-act, for example limiting memory, number of processes, or use cgroups to limit IO rate (see for examplehttp://unix.stackexchange.com/questions/48138/how-to-throttl...).

Also leave a ssh connection with something like htop or atop open, and look at it once it freezes up.

Re: Linux Performance Analysis

#58

Post by Splines »

samstave wrote: post #46

This is yet another awesome contribution from Netflix. Just curious though, anyone know of a similar type of writeup or post regarding windows? I have an MSSQL box I want to kill with fire - but I would like to be able to measure its perf with as much insight as I might on a comparable linux box. What is the best way to measure disk IO perf on a windows box, specifically?

xperf/wpr. Launch wprui, select First Level Triage and Disk I/O and File I/O, make sure logging mode is set to Memory, then click Start.

Let it run for a minute, click Stop, then open up the generated ETL in WPA.

From there it's hard to tell you what to do in WPA. Documentation isn't the greatest, but this link is close to what you're trying to do: http://blogs.msdn.com/b/sql_pfe_blog/archive/2013/03/19/trou...

Drag in the relevant graphs into your current view, and understand how the yellow bar aggregates data. MSSQL probably has event providers - you might want to look those up and add them to wprui when you generate your ETL.

Re: Linux Performance Analysis

#59

Post by zantana »

baldfat wrote: post #51
digi_owl wrote: post #38

Earlier quoted context omitted.

dmesg and being able to run most things in a terminal to check what its doing (grumble, skype, grumble) is likely what has drawn many to Linux in the first place. And a likely source of misgivings regarding recent developments in the ecosystem...

journalctl is a good tool. I really like SystemD but wish we still had plain text logs.

One improvement (which may or may not be related to systemd, but is in newer kernels) is that dmesg has timestamps enabled by default which makes it much easier and `dmesg -H` is really nice as well.

Re: Linux Performance Analysis

#60

Post by anon3423423422 »

jandrese wrote: post #31
i_have_to_speak wrote: post #3

Hmm, that java process has 0.227 TB of mapped virtual memory and 3090% CPU..

I'm guessing that the java process is their video server. It's probably having to DRM and bandwidth regulate hundreds of HD streams. Unfortunately, this might mean that all of the stuff he talked about in here didn't actually help him solve the problem because none of it had any visibility inside of their Java VM.

Some of their recommendation algorithms run on large boxes in java.
Post reply on HN