It'd be interesting to know some info on the hardware back end, such as number of servers, storage system, etc. Also, how many servers does a typical query touch?
MySQL at Facebook
21–30 of 42 posts
Re: MySQL at Facebook
#22"It is OK if a query is slow as long as it is always slow" I find this enlightening. I think it is just one instance of difference in emphasis between Google and Facebook. Google, a technology oriented company, would minimize the average response time, while Facebook, a people oriented company would minimize the unpredictability (by minimizing the variance)
For what it's worth, Jeff Dean of Google has famously emphasized 95th and 99th percentile performance in preference to average or median performance over the years. The realization that edge cases more powerfully determine user perception of performance than average cases is a deep one, and is not original to Facebook or Google; Dean and colleagues were making some of the same points when they were at DEC's CRC pre-G…
Do you know of any research supporting that? It sounds plausible to me but a quick search turned up nothing.
Re: MySQL at Facebook
#23Does PayPal use MySQL?
Everyone is using MySQL. But I attended a talk by one of their DBAs where he said that they large OLTP server, the one that processes the payments is Oracle. Single-instance, because Oracle RAC can't give them the low latency they needed. They switch the the biggest machine IBM can give them every few month.
>They switch the the biggest machine IBM can give them every few month.
Yea, almost all this "web scale" scalability can achieve those numbers because they can partition, relax transactional requirements, and scale out. For system-wide true transactions, scaling up is still the only option, it seems.
Although, H-Store/VoltDB claims some impressive numbers if your OLTP app fits the requirements.
Re: MySQL at Facebook
#24"It is OK if a query is slow as long as it is always slow" I find this enlightening. I think it is just one instance of difference in emphasis between Google and Facebook. Google, a technology oriented company, would minimize the average response time, while Facebook, a people oriented company would minimize the unpredictability (by minimizing the variance)
>> "It is OK if a query is slow as long as it is always slow" I'm having trouble understanding the motivation. If a slow query is always slow, then I'm always going to be kept waiting for that page/data. It seems logical to worry about the queries that 100% of the time keeps users waiting rather than the queries that keep users waiting Does anyone care to explain why this is a good idea (for Facebook at least)?
It's all about user experience. If a feature loads for you in 300ms, then it will feel unbearably slow if it later take 700ms. If the feature always loaded at 700ms, your expectation of 700ms would never be disappointed. The theory is that a consistent 700ms is more appealing than an inconsistent 300-700ms, even though the median is better in terms of raw performance.
Re: MySQL at Facebook
#25Earlier quoted context omitted.
For what it's worth, Jeff Dean of Google has famously emphasized 95th and 99th percentile performance in preference to average or median performance over the years. The realization that edge cases more powerfully determine user perception of performance than average cases is a deep one, and is not original to Facebook or Google; Dean and colleagues were making some of the same points when they were at DEC's CRC pre-G…
Do you know of any research supporting that? It sounds plausible to me but a quick search turned up nothing.
There is some social psychology and operations research around this -- not necessarily as it relates to human/machine interactions, but certainly as it relates to such quasi-analogous situations as call centers, lines, speed vs. variability of service at McDonald's, and so forth. The premise being practiced by Facebook takes a lot of cues from behavioral science data that is generalizable to many different domains.
McDonald's, to give a famous example, proved analytically a few years back that -- beyond a certain threshold of commonly expected service times -- their customers would rather get semi-slow service on a consistent basis than highly variable service. In a perfect world, of course, average wait times are as short as possible in addition to a minimization of variance. But when you're at the level of acceptability, there are diminishing returns on speed increases and increasing returns on reduced variance.
Re: MySQL at Facebook
#26I doubt this HN post could have a more boring title. The actual article is very interesting. Basically Facebook's DB developers talk about what it takes to perform 13 million queries per second, and a lot of the useful tips they've learned along the way to make scaling simpler. There's a full video of the talk available at http://www.livestream.com/facebookevents/video?clipId=flv_cc...
"Stuff about data at some company"
Re: MySQL at Facebook
#27The concept of variance reduction is not even new with computing - In manufacturing they have six sigma. I believe the famous GE CEO's quote was "Our Customers Feel the Variance, Not the Mean" (Google it).
This is talked about here: http://www.mysqlperformanceblog.com/2010/06/07/performance-o...
Re: MySQL at Facebook
#28"They figure out why the response time for the worst query is bad and then fix it." This can reap huge benefits and doesn't need to be difficult. Just enable the slow query log in MySQL, use the EXPLAIN command to analyze the results, then add indexes where appropriate. I was able to fix poorly indexed tables in a vendor's application with dramatic results. In one case, a twenty-minute(!) query was reduced to less th…
While your premise is correct it's not always that simple. More indexes can slow down updates and inserts. A tables overall usage pattern needs to be examined before just adding more indexes. You might have fixed the 20 minute report query while slowing down some other more critical query that loaded or updated data.
Yes, this is a good point and I actually noted it in my issue tracker while troubleshooting. I made sure that inserts & updates weren't negatively affected to an unacceptable degree after adding the indexes. The overall win was huge, because it eliminated wait times for a sizable staff and client base.
Re: MySQL at Facebook
#29I doubt this HN post could have a more boring title. The actual article is very interesting. Basically Facebook's DB developers talk about what it takes to perform 13 million queries per second, and a lot of the useful tips they've learned along the way to make scaling simpler. There's a full video of the talk available at http://www.livestream.com/facebookevents/video?clipId=flv_cc...
How is the title boring? What could be more telling? Having "Facebook" in the title makes it clear we're going to be talking about astronomical amounts of data. "[insert any db] at Facebook" would be an interesting title in my opinion. The fact that it's MySQL makes it even more interesting, given the shift of scalability interest to MongoDB, etc.
Maybe boring isn't the right word. Vague is probably better. "MySQL at Facebook" is a facebook page, a launchpad user, a twitter user, a O'Reilly MySQLConf talk, etc.
Re: MySQL at Facebook
#30Earlier quoted context omitted.
Do you know of any research supporting that? It sounds plausible to me but a quick search turned up nothing.
There is some social psychology and operations research around this -- not necessarily as it relates to human/machine interactions, but certainly as it relates to such quasi-analogous situations as call centers, lines, speed vs. variability of service at McDonald's, and so forth. The premise being practiced by Facebook takes a lot of cues from behavioral science data that is generalizable to many different domains. M…
Do you have a link to this research or study from McDonalds?