3. Hive is Facebook's data warehouse, with 300 petabytes of data in 800,000 tables. Facebook generates 4 new petabyes of data and runs 600,000 queries and 1 million map-reduce jobs per day.
So 4 PB per day, but only 300 PB total?
Facebook's Top Open Data Problems
21–30 of 65 posts
Re: Facebook's Top Open Data Problems
#223. Hive is Facebook's data warehouse, with 300 petabytes of data in 800,000 tables. Facebook generates 4 new petabyes of data and runs 600,000 queries and 1 million map-reduce jobs per day. So 4 PB per day, but only 300 PB total?
Was wondering the same thing. My guess is that some also gets removed each day as well, but it seems unlikely.
Re: Facebook's Top Open Data Problems
#23This is slightly off topic, but has any experienced an increase in "fake" toasts from facebook mobile? It seems if I haven't used facebook mobile in a few days or I don't respond to their toasts about very minor people in my life uploading a photo, I tend to start getting toasts that say "You have 5 notifications, 3 pokes and 2 messages.", then I open the app and it takes me to an unknown error page.
Am I being too cynical in thinking that Facebook is intentionally misleading its users in an attempt to bump up their metrics? It interests me that they are seeing jumps in their mobile users (and consequently, ad sales) at the same time that I have been receiving more notifications than ever. Interestingly, the slowdown in fake toast notifications coincided with their quarterly earnings report that show mobile ads accounting for an increasingly large portion of revenue and also mentions an increase in mobile user usage.
Comparing Q1 with Q2 with Q3, Q2-Q3 showed double the increase in ad revenue percent from mobile (59% to 62% to 66%). Maybe this is just all anecdotal evidence, but it seems like these sort of fake notifications should either not be sent out (failure of the system that keeps track of what user receives what toasts) or there was a conscious effort to send these notifications....
Re: Facebook's Top Open Data Problems
#24This is slightly off topic, but has any experienced an increase in "fake" toasts from facebook mobile? It seems if I haven't used facebook mobile in a few days or I don't respond to their toasts about very minor people in my life uploading a photo, I tend to start getting toasts that say "You have 5 notifications, 3 pokes and 2 messages.", then I open the app and it takes me to an unknown error page. Am I being too c…
It's been doing this for me via email lately and it's really annoying.
Re: Facebook's Top Open Data Problems
#253. Hive is Facebook's data warehouse, with 300 petabytes of data in 800,000 tables. Facebook generates 4 new petabyes of data and runs 600,000 queries and 1 million map-reduce jobs per day. So 4 PB per day, but only 300 PB total?
Was wondering the same thing. My guess is that some also gets removed each day as well, but it seems unlikely.
Think of it like monitoring data. You may collect one-second data on 500 counters per system over 1000 systems, but then you will do a weekly or monthly roll-up where you dump some of the granularity to save space, and after a year you have aggregates that are basically daily trend lines. The more you collect smaller percentage you actually keep.
Re: Facebook's Top Open Data Problems
#26I'm really curious how they handle paging if they're only using memcached. E.G. If a a photo node has 10,000 comment nodes (and thus 10,000 edges linking the photo to the comments), chances are you only want to display the most recent 50 comments. Are all of the 10,000 edges stored in memcached under one key and then paged on the application servers? Are they stored in chunks under multiple keys? How is cache consistency maintained if somebody makes a new comment (maintaining the time ordering seems tricky and expensive)?
This is a problem I'm actively trying to solve for a project, so if somebody knows the answer, please get in touch!
Re: Facebook's Top Open Data Problems
#27I'm really curious how they handle paging if they're only using memcached. E.G. If a a photo node has 10,000 comment nodes (and thus 10,000 edges linking the photo to the comments), chances are you only want to display the most recent 50 comments. Are all of the 10,000 edges stored in memcached under one key and then paged on the application servers? Are they stored in chunks under multiple keys? How is cache consist…
That's what TAO (mentioned in the article) is for
Re: Facebook's Top Open Data Problems
#28I'm really curious how they handle paging if they're only using memcached. E.G. If a a photo node has 10,000 comment nodes (and thus 10,000 edges linking the photo to the comments), chances are you only want to display the most recent 50 comments. Are all of the 10,000 edges stored in memcached under one key and then paged on the application servers? Are they stored in chunks under multiple keys? How is cache consist…
That's what TAO (mentioned in the article) is for
Yes, I've read the TAO paper and referenced it heavily for my project. I'm curious about the implementation specifics of the caching layer.
EDIT: More specifically, how do you efficiently cache your data such that an assoc_range query can be answered from cached without O(n) operations on your application servers. Memcached can't answer a query like "give me 50 items starting from position 0 in the list" as far as i'm aware, so you'd need to pull the whole list to the application server and slice it up there. When you consider that you want the 50 most recent items too, maintaining sorted lists adds extra complexity.
Re: Facebook's Top Open Data Problems
#29Earlier quoted context omitted.
Simple scales better than complex. What they need is key->value caching and for that memcached is a perfect match. I'm not saying that Redis is bad, but when all you need is key-value memory caching, redis isn't needed.
when all you need is key-value memory caching, redis isn't needed. Balderdash. memcached is actively hostile to modern infrastructure and it's not being actively developed except for routine maintenance. The first time someone stores an entire data structure in a memcache key is the moment you've lost. Playing "read blob, deserialize, update, serialize, write blob" is just dumb when you can avoid it. Other moments of…
You shouldn't be doing read-modify-write cycles in memcached. It's use is as a demand-filled, look-aside cache. Modern open source memcached has slab reallocation that works quite well. It certainly beats a malloc() heap, which will become highly fragmented and inefficient.
Re: Facebook's Top Open Data Problems
#30Still using Memcache wow.
DevOps here using their mcrouter tool [1] in production. It's a phenomenal swiss army knife for using memcached. [1] https://github.com/facebook/mcrouter
Aw shucks.
Mcrouter was the first piece of software I worked on at facebook. It's nice to see that you like it! (Though the only thing that lives on from me is the umbrella protocol)