Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Performance Analysis

techblog.netflix.com

Post reply on HN
Print view View original

61–70 of 84 posts

Re: Linux Performance Analysis

#61

Post by odaymansour »

jerf wrote: post #54

Is there any way to analyze memory bandwidth usage? I've had a couple of programs over the years that I wondered if we were just hitting memory bandwidth limitations but I couldn't find any way to prove that, or even particularly gather evidence.

I've identified a memory bandwidth issue in the past by keeping an eye on truss/strace output and "counting" mem operations.

After that, I compiled and ran a tiny executable called Stream[1] and got the numbers I needed in order to explain why one machine was twice as slow as another.

[1] http://www.cs.virginia.edu/stream/

Re: Linux Performance Analysis

#62

Post by sciurus »

CTrox wrote: post #56
Abundnce10 wrote: post #49

I had one of my EC2 instances (Ubuntu 14.04) lock up on Monday morning. I use it to run a Ruby on Rails app and to do a handful of ETL jobs. The website wasn't able to load and I wasn't able to SSH into the box. So, I went into the AWS console and rebooted the instance. This has happened 3 times in the last 6 months. It looks like the CPU spikes up to almost 100% around the same time when the instance locks up. I'd l…

I can recommend atop[1] for that. It runs every ten minutes by default and writes lots of information to /var/log/atop/atop_YYYYMMDD. With that you can examine what happend before the crash, just open a file with atop -r /var/log/atop/atop_YYYYMMDD. [1] http://linux.die.net/man/1/atop

A nice atop tutorial: https://lwn.net/Articles/387202/

Re: Linux Performance Analysis

#63

Post by samstave »

Splines wrote: post #58
samstave wrote: post #46

This is yet another awesome contribution from Netflix. Just curious though, anyone know of a similar type of writeup or post regarding windows? I have an MSSQL box I want to kill with fire - but I would like to be able to measure its perf with as much insight as I might on a comparable linux box. What is the best way to measure disk IO perf on a windows box, specifically?

xperf/wpr. Launch wprui, select First Level Triage and Disk I/O and File I/O, make sure logging mode is set to Memory, then click Start. Let it run for a minute, click Stop, then open up the generated ETL in WPA. From there it's hard to tell you what to do in WPA. Documentation isn't the greatest, but this link is close to what you're trying to do: http://blogs.msdn.com/b/sql_pfe_blog/archive/2013/03/19/trou... Drag…

Thank you!!

Re: Linux Performance Analysis

#64

Post by JD557 »

unfletch wrote: post #40
tveita wrote: post #37

The link redirects to the blog frontpage for me, but https://media.netflix.com/en/tech-blog/linux-performance-ana... works.

I think Netflix may have just rolled out a blog redesign. (The original link worked for me at first, but now redirect me to media.netflix.com.) Too bad they broke old links though. Hopefully that's temporary.

It seems that they also removed the RSS feed, which is a shame.

Re: Linux Performance Analysis

#65

Post by alinspired »

Great list with "standard" tools that you'd find on most systems by default. From my experience for virtualized servers you might also look at:

- system and stolen cpu to show virtualization overhead and VMs starvation for CPU (in vmstat)

- interrupts and context switches which might indicate that VMs might be running non-optimized OSs or non-paravirtualized drivers (vmstat)

- abusing VMs/Containers (platform specific), ie for KVM: virsh vcpuinf, virsh dominfo

- socket summary and dig from there: ss -s

Re: Linux Performance Analysis

#66

Post by danieltillett »

luckydude wrote: post #35

This is pretty cool because back in the dot com days I got paid huge amounts of money to do what they did in the first 60 seconds. I started at $1000/day and the phone rang off the hook. Talked to a friend and he said "double your rates" and I said "WTF? I'm not worth $2000/day, that's crazy". He said "double your rates". So I did, phone still rang like crazy. I got up to $4000/day and had 5 days/week work. Went to $…

Did the money dry up because you got too expensive or because the dot com bubble burst?

Re: Linux Performance Analysis

#67

Post by cakes »

Aissen wrote: post #4

I'm wondering if there's a way to automate this kind of analysis to give you condensed, interesting points of a system performance status. À la powertop for power management.

There is - software like Splunk, SysTrack, etc. get towards that in various ways.

Re: Linux Performance Analysis

#70

Post by simoncion »

zantana wrote: post #59
baldfat wrote: post #51

Earlier quoted context omitted.

journalctl is a good tool. I really like SystemD but wish we still had plain text logs.

One improvement (which may or may not be related to systemd, but is in newer kernels) is that dmesg has timestamps enabled by default which makes it much easier and `dmesg -H` is really nice as well.

  dmesg -T
Has been available for years to print out human-readable timestamps. Check to see if dmesg is an alias to that on your system. Also, -IIRC- dmesg has printed system-uptime timestamps with every line since at least the early 2000's.

One caveat: the times printed by dmesg -T will be incorrect if the system has suspended to RAM or disk.

Check dmesg(1) for more info! :D

Post reply on HN