Google Finds NUMA Up to 20% Slower for Gmail and Websearch
highscalability.com
Google Finds NUMA Up to 20% Slower for Gmail and Websearch
1–10 of 19 posts
Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch
#2Up to 20% slower than what?
(Than SMP systems, I guess, but the OP does not say.)
Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch
#3One quibble: the author at High Scalability refers to the authors of the paper collectively as "Google," but the lead authors, Lingjia Tang and Jason Mars, are professors at UC San Diego. Of course, they must have collaborated with Google and they may have done the work while doing summer internships in 2011 (CVs are at http://www.lingjia.org/ and http://jasonmars.org/).
Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch
#4One quibble: the author at High Scalability refers to the authors of the paper collectively as "Google," but the lead authors, Lingjia Tang and Jason Mars, are professors at UC San Diego. Of course, they must have collaborated with Google and they may have done the work while doing summer internships in 2011 (CVs are at http://www.lingjia.org/ and http://jasonmars.org/ ).
Thanks for pointing out the error Scott. It should be fixed now.
Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch
#5Up to 20% slower than what? (Than SMP systems, I guess, but the OP does not say.)
Than keeping memory accesses chip local (I guess via thread pinning). The comparison was done on the same hardware platform.
Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch
#6Up to 20% slower than what? (Than SMP systems, I guess, but the OP does not say.)
Than keeping memory accesses chip local (I guess via thread pinning). The comparison was done on the same hardware platform.
I am not surprised, I have seen slowdowns of 40%. NUMA leads to an annoying bimodal timing, where some runs are fast and others slow.
Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch
#7I'm having a little trouble making sense of this:
"For example, bigtable benefits from cache sharing and would prefer 100 % remote accesses to 50% remote. Search-frontend prefers spreading the threads to multiple caches to reduce cache contention and thus also prefers 100 % remote accesses to 50% remote."
Let me see if I've got this straight:
* bigtable benefits from scheduling related threads on the same cpu so they can share a cache, I'm guessing because multiple threads work on the same data simultaneously
* search benefits from having its threads spread over many cpus, probably because the threads are unrelated to each other and not sharing data, so they like to have their own caches
I'm not sure I understand how this relates to NUMA, or why remote accesses are ever a good thing. Maybe it requires a more sophisticated understanding of computer architecture than what I have.
Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch
#8I like reading these analyses, although I'm afraid headlines like this oversimplify things and give off the wrong impression. There isn't anything inherently wrong with NUMA, it just isn't useful in this situation.
No technology is a 'silver bullet'. Every workload has a different set of considerations that require a different set of technology to optimize.
Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch
#9I'm having a little trouble making sense of this: "For example, bigtable benefits from cache sharing and would prefer 100 % remote accesses to 50% remote. Search-frontend prefers spreading the threads to multiple caches to reduce cache contention and thus also prefers 100 % remote accesses to 50% remote." Let me see if I've got this straight: * bigtable benefits from scheduling related threads on the same cpu so they…
I'm not familiar with this research, but it's possible that sequential accesses to memory would lead to prefetching, in which case going half-local half-remote could actually lead to a slowdown versus all-remote. Another hypothesis is if the memory ends up having to be migrated from one cpu cache to the other, then back. It's better if it's always in the remote cache than if it's getting flipped between the two.
I'm pretty sure it goes without saying that 100% local is always better, assuming you're not trading anything else away (like accessible CPU on other nodes).
Re: Google Finds NUMA Up to 20% Slower for Gmail and Websearch
#10Specifically: "in multicore multisocket machines, there is often a tradeoff between optimizing NUMA performance by clustering threads close to the memory nodes to increase the amount of local accesses and optimizing for cache performance by spreading threads to reduce the cache contention"
I.e. the performance benefit from socket-local memory accesses may not be worth having all the threads using that memory on that socket's CPUs, because they'll each get too little a share of the cache.