Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two frequently used system calls are ~77% slower on AWS EC2

blog.packagecloud.io

Post reply on HN
Print view View original

91–100 of 101 posts

Re: Two frequently used system calls are ~77% slower on AWS EC2

#91

Post by lathiat »

nneonneo wrote: post #36

The title is misleading. 77% slower sounds like the system calls take 1.77x the time on EC2. In fact, the results indicate that the normal calls are 77% faster - in other words, EC2 gettimeofday and clock_gettime calls take nearly 4.5x longer to run on EC2 than they do on ordinary systems. This is a big speed hit. Some programs can use gettimeofday extremely frequently - for example, many programs call timing functio…

Yeah like the php Xdebug extension. In 5.3 at least even with it just loaded and nothing enabled it called gettimeofday 1000s of times and would add seconds to web app render times for me (also on Xen with slow gettimeofday)

Re: Two frequently used system calls are ~77% slower on AWS EC2

#93

Post by tropo »

stouset wrote: post #71
klodolph wrote: post #44

Earlier quoted context omitted.

77% faster is not correct either. "Speed" would probably by ops/s. 4.5x longer = 350% slower.

Even this is confusing as hell. Just say the native calls take 22% of the time they do on EC2. Or that the EC2 calls take 450% of the time of their native counterparts. "Faster" and "slower" when going with percentages are ripe with confusion. Please don't use them.

This is what decibels are for.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#94

Post by klodolph »

stouset wrote: post #71
klodolph wrote: post #44

Earlier quoted context omitted.

77% faster is not correct either. "Speed" would probably by ops/s. 4.5x longer = 350% slower.

Even this is confusing as hell. Just say the native calls take 22% of the time they do on EC2. Or that the EC2 calls take 450% of the time of their native counterparts. "Faster" and "slower" when going with percentages are ripe with confusion. Please don't use them.

I can't agree. Speed is usually units/time, and everyone knows that 100 mph is 2x as fast as 50 mph, or 100% faster.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#95

Post by jankedeen »

teddyuk wrote: post #31

How common are get time calls so that they would actually be an issue? I've worked on quite a few systems and can't think of a time where an api for getting the time would have been called so much that it would affect performance?

Any application code that includes logging of any sort is going to grab time. All of my code (quite a private set and scientific in nature) calls time() at critical points with identifiers so I can easily investigate issues.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#96

Post by jankedeen »

peterwwillis wrote: post #88

> All programmers deploying software to production environments should regularly strace their applications in development mode and question all output they find. Or, instead, you could just not do that. Then you could go back to being productive, instead of wasting time tracking down unstable small tweaks for edge cases that you can barely notice after looping the same syscall 5 million times in a row. When will peop…

Crapulent and without merit.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#97

Post by amr_01 »

brendangregg wrote: post #33
brendangregg wrote: post #10

Yes, this is why we (Netflix) default to tsc over the xen clocksource. I found the xen clocksource had become a problem a few years ago, quantified using flame graphs, and investigated using my own microbenchmark. Summarized details here: https://www.slideshare.net/brendangregg/performance-tuning-e...

This reminds me: I should give an updated version of that talk for 2017...

Please do, I would be very interested in this!

Re: Two frequently used system calls are ~77% slower on AWS EC2

#98

Post by jankedeen »

brendangregg wrote: post #33
brendangregg wrote: post #10

Yes, this is why we (Netflix) default to tsc over the xen clocksource. I found the xen clocksource had become a problem a few years ago, quantified using flame graphs, and investigated using my own microbenchmark. Summarized details here: https://www.slideshare.net/brendangregg/performance-tuning-e...

This reminds me: I should give an updated version of that talk for 2017...

I've been in a couple positions recently where they mention your name and I look at your work and think to myself..here is a sysadmin with modest skills who (by exposure) has become notably vocal and somewhat adept at scale computing. In general if a company mentions Netflix or Brendan Gregg I flinch. Just an FYI.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#99

Post by bdarfler »

brendangregg wrote: post #32
aluminussoma wrote: post #14

Earlier quoted context omitted.

Can you share if you needed to do anything to deal with time drift issues when using tsc? For my own systems, incorrect timestamps would cause a lot of issues.

Well, it's been a few years and we haven't switched it back. :) We have had a number of clock issues, and one of the first things I try is taking and instance and switching it back to xen for a few days, but those issues have not turned out to be the clocksource. Usually NTP. AWS can comment more about the state (safety/risk) of these clocksources (given they have access to all the SW/HW internals).

Looks like AWS recommends TSC as well.

https://www.slideshare.net/AmazonWebServices/cmp402-amazon-e...

Re: Two frequently used system calls are ~77% slower on AWS EC2

#100

Post by brendangregg »

jankedeen wrote: post #98
brendangregg wrote: post #33

Earlier quoted context omitted.

This reminds me: I should give an updated version of that talk for 2017...

I've been in a couple positions recently where they mention your name and I look at your work and think to myself..here is a sysadmin with modest skills who (by exposure) has become notably vocal and somewhat adept at scale computing. In general if a company mentions Netflix or Brendan Gregg I flinch. Just an FYI.

Sorry to make you flinch! I'm curious what of my work you were looking at; on this thread I had mentioned this:

https://www.slideshare.net/brendangregg/performance-tuning-e...

I think it's a pretty good summary, and includes work from my team and some original work of my own.

Is there something I could change in it that would make it more helpful for you?

Post reply on HN