Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Performance Analysis

techblog.netflix.com

Post reply on HN
Print view View original

31–40 of 84 posts

Re: Linux Performance Analysis

#31

Post by jandrese »

i_have_to_speak wrote: post #3

Hmm, that java process has 0.227 TB of mapped virtual memory and 3090% CPU..

I'm guessing that the java process is their video server. It's probably having to DRM and bandwidth regulate hundreds of HD streams. Unfortunately, this might mean that all of the stuff he talked about in here didn't actually help him solve the problem because none of it had any visibility inside of their Java VM.

Re: Linux Performance Analysis

#32

Post by JoshTriplett »

> Pidstat is a little like top’s per-process summary, but prints a rolling summary instead of clearing the screen. This can be useful for watching patterns over time, and also recording what you saw (copy-n-paste) into a record of your investigation.

top has a batch mode: "top -b" will print all processes periodically to the terminal, without starting the curses-style UI, providing the same "rolling summary".

Re: Linux Performance Analysis

#33

Post by brendangregg »

Good to see people sharing extra commands!

I had written some follow on commands at the end of the article, but trimmed them to make it shorter. They were:

  # CPU
  perf record -F 99 -a -g -- sleep 10; perf report -n --stdio   # and flamegraphs
  execsnoop       # from perf-tools
  turbostat

  # Memory
  cat /proc/meminfo
  slabtop

  # Disk
  df -h
  iosnoop         # from perf-tools
  pidstat -d
  iotop

  # Networking
  netstat -s
  tcpdump
  tcpretrans      # from perf-tools
Where perf-tools is https://github.com/brendangregg/perf-tools.

Re: Linux Performance Analysis

#35

Post by luckydude »

This is pretty cool because back in the dot com days I got paid huge amounts of money to do what they did in the first 60 seconds. I started at $1000/day and the phone rang off the hook. Talked to a friend and he said "double your rates" and I said "WTF? I'm not worth $2000/day, that's crazy". He said "double your rates". So I did, phone still rang like crazy. I got up to $4000/day and had 5 days/week work. Went to $8000/day and got about 3-4 more days and then it dried up.

And that is how I made payroll for BitKeeper in the early days.

I think it's cool that this stuff is far more common knowledge these days. All I was doing was figuring out if it was CPU, network, disks, file system, vm system, memory. 99% of the time I knew what the problem was in a few minutes, all the real effort was tracking down what program (or programs) were causing it. Funny enough, most people didn't care what cause was, they just wanted to know what to buy to make it not an issue. Crazy amounts of money being thrown at Sun equipment.

Re: Linux Performance Analysis

#36

Post by luckydude »

STRML wrote: post #19
agentgt wrote: post #13

Earlier quoted context omitted.

I'll add to your list with htop. Its a little nicer than plain ole top.

And `glances` is pretty great for big-picture stuff, all-in-one. It's not as precise on CPU/memory as htop, but you get network bandwidth, disk I/O, disk usage and more.

No love for dstat? Glances looks cool though, thanks for that one.

Re: Linux Performance Analysis

#38

Post by digi_owl »

jandrese wrote: post #30
baldfat wrote: post #28

> Don’t miss this step! dmesg is always worth checking. Best advice given in the whole article. Many times I go to check something that "can't figure out what is wrong." dmseg | tail and then the swearing begins on their end.

Can you imagine a world where the Windows System Event Viewer was this useful? It's amazing how much time one useful error message can save. Also, I'm a little disappointed that the author didn't drill down into what the actual problem was in his example. That java process is of course suspicious, but that might just be the video server and expected behavior.

dmesg and being able to run most things in a terminal to check what its doing (grumble, skype, grumble) is likely what has drawn many to Linux in the first place.

And a likely source of misgivings regarding recent developments in the ecosystem...

Re: Linux Performance Analysis

#40

Post by unfletch »

tveita wrote: post #37

The link redirects to the blog frontpage for me, but https://media.netflix.com/en/tech-blog/linux-performance-ana... works.

I think Netflix may have just rolled out a blog redesign. (The original link worked for me at first, but now redirect me to media.netflix.com.) Too bad they broke old links though. Hopefully that's temporary.
Post reply on HN