Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Finds NUMA Up to 20% Slower for Gmail and Websearch

highscalability.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch

#11

Post by scott_s »

mtdewcmu wrote: post #7

I'm having a little trouble making sense of this: "For example, bigtable benefits from cache sharing and would prefer 100 % remote accesses to 50% remote. Search-frontend prefers spreading the threads to multiple caches to reduce cache contention and thus also prefers 100 % remote accesses to 50% remote." Let me see if I've got this straight: * bigtable benefits from scheduling related threads on the same cpu so they…

It's not that remote accesses are good, it's that trying to induce them can harm cache usage elsewhere. If the author at High Scalability will allow me another quibble, I'd say that actually, memory locality is still King. It's just that we have to be very careful about trying to improve it; if you try to improve locality in one place (say, induce local accesses from a socket to main memory), you may end up harming it somewhere else (more total number of accesses to main memory because now the cache is thrashing).

The NUMA bit comes in when you said "scheduling related threads on the same cpu" and "threads spread over many cpus". If you schedule related threads on the same socket (cpu), you're more likely to get local accesses. If your threads share data, then that's two good things: local memory accesses, and good cache usage. But if your threads use different data, then the fact that you have local memory accesses may not matter because you may have many more cache misses because the threads are interfering with each other.

A simpler way to think about it: shorter access to main memory does not help you if you end up doing many more total accesses.

Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch

#12

Post by mtdewcmu »

jamesaguilar wrote: post #9
mtdewcmu wrote: post #7

I'm having a little trouble making sense of this: "For example, bigtable benefits from cache sharing and would prefer 100 % remote accesses to 50% remote. Search-frontend prefers spreading the threads to multiple caches to reduce cache contention and thus also prefers 100 % remote accesses to 50% remote." Let me see if I've got this straight: * bigtable benefits from scheduling related threads on the same cpu so they…

I'm not familiar with this research, but it's possible that sequential accesses to memory would lead to prefetching, in which case going half-local half-remote could actually lead to a slowdown versus all-remote. Another hypothesis is if the memory ends up having to be migrated from one cpu cache to the other, then back. It's better if it's always in the remote cache than if it's getting flipped between the two. I'm…

Ah. So a remote access may be coming directly from the cache of a different CPU? That's something I didn't consider, and definitely adds another wrinkle.

I sense that the article is saying things in confusing ways, perhaps because that's the way computer architects speak (it always struck me as counterintuitive and confusing to measure a cache by its miss rate rather than its hit rate) or maybe it's this article.

Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch

#13

Post by mtdewcmu »

mckilljoy wrote: post #8

I like reading these analyses, although I'm afraid headlines like this oversimplify things and give off the wrong impression. There isn't anything inherently wrong with NUMA, it just isn't useful in this situation. No technology is a 'silver bullet'. Every workload has a different set of considerations that require a different set of technology to optimize.

The way it's portrayed is extremely misleading. The headline misses the point -- actually, the article didn't really have a point. It sounds like they didn't get the results they wanted from the project, but tried to make the best of it by highlighting what they did get, which is a jumble of facts that are incoherent and self-contradictory. It's sort of interesting to read, because they did honest research, asked good questions and followed the data, and there is plenty of value in negative results.

The way I read the outcome, NUMA seems to do what it's supposed to. The premise was that remote memory accesses are a performance killer, and forcing threads onto fewer cpus should be a big win. But NUMA came out looking pretty good. Leaving it alone looks like an excellent policy. Consider that google brought in a team of experts for the sole purpose of figuring out how to beat the default behavior of NUMA.

Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch

#14

Post by chad_walters »

The title is not just misleading -- it is just plain wrong.

NUMA was 15% better for Gmail and 20% better for the Web search frontends, as indicated by the reductions (improvements) in CPI for these workloads.

There were some workloads where NUMA did degrade performance, such as BigTable accesses (12% regression).

Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch

#15

Post by rpearl »

Given Google's ability to obtain processors before they are available to the public, and given that this paper refers to AMD's Barcelona processors, the results published here are probably approximately seven years out-of-date, and it's not clear whether they're still relevant now.

Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch

#16

Post by b0b0b0b »

scott_s wrote: post #11
mtdewcmu wrote: post #7

I'm having a little trouble making sense of this: "For example, bigtable benefits from cache sharing and would prefer 100 % remote accesses to 50% remote. Search-frontend prefers spreading the threads to multiple caches to reduce cache contention and thus also prefers 100 % remote accesses to 50% remote." Let me see if I've got this straight: * bigtable benefits from scheduling related threads on the same cpu so they…

It's not that remote accesses are good , it's that trying to induce them can harm cache usage elsewhere. If the author at High Scalability will allow me another quibble, I'd say that actually, memory locality is still King. It's just that we have to be very careful about trying to improve it; if you try to improve locality in one place (say, induce local accesses from a socket to main memory), you may end up harming…

Do the bigtable performance characteristics look kind of like cache line ping ponging? My intuition for scenario 3 outperforming scenario 2 (100% remote vs 50% local + 50% remote) is that there are more mutations of data and therefore more interconnect traffic is required to maintain coherency across sockets.

Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch

#17

Post by mgurlitz »

scott_s wrote: post #3

One quibble: the author at High Scalability refers to the authors of the paper collectively as "Google," but the lead authors, Lingjia Tang and Jason Mars, are professors at UC San Diego. Of course, they must have collaborated with Google and they may have done the work while doing summer internships in 2011 (CVs are at http://www.lingjia.org/ and http://jasonmars.org/ ).

A little background on the relationship of the authors to Google: http://www.wired.com/wiredenterprise/2013/05/google-jason-ma...

Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch

#19

Post by kmasters »

scott_s wrote: post #3

One quibble: the author at High Scalability refers to the authors of the paper collectively as "Google," but the lead authors, Lingjia Tang and Jason Mars, are professors at UC San Diego. Of course, they must have collaborated with Google and they may have done the work while doing summer internships in 2011 (CVs are at http://www.lingjia.org/ and http://jasonmars.org/ ).

A bit of a frustrating read. What exactly is a "performance swing"? Does that mean performance increased or decreased? I find the writing does not match the precision required to cover a highly technical topic.
Post reply on HN