11–20 of 65 posts
I strongly dislike Facebook the product, and to lesser extent Facebook the company, but I'm continually impressed with Facebook's approach to engineering in the open. I find this an interesting dichotomy. Would I want to work there? I still don't think so, but my opinion on that front is getting less strong over time.
Still using Memcache wow.
Earlier quoted context omitted.
This is the correct answer. Source: I am a FB employee.
Thanks. So, how much does the 4PB of data go down to when it goes into hive? My guess would be something like 200TB. Is it zipped (or lz4/lzo/zopfli/lzma whatever)? Or is it just "distilled"?
Most of it likely gets tossed into a program to determine if somebody actually needs to do anything, or if something is actually breaking.
For example raw user interaction doesn't really grow. While the event is likely ~1kb or so of raw data, at the end your just incrementing a 64bit counter.
This is a baseless exaggerated post but should shed some light on.
Still using Memcache wow.
Earlier quoted context omitted.
Whats wrong with Memcached? Why are you so surprised?
I would have thought that it's too simple for their workload / needs, something like Redis would have been better?
So ~650M daily active users..4PB of data warehouse created each day, that means ~7MB of new data on each active user per day. Given that its data warehouse, I'm going to guess its not images, seems like a lot to me. I guess it shouldn't surprise anyone that every interaction on and off the site, is heavily tracked.
Earlier quoted context omitted.
I would have thought that it's too simple for their workload / needs, something like Redis would have been better?
Simple scales better than complex. What they need is key->value caching and for that memcached is a perfect match. I'm not saying that Redis is bad, but when all you need is key-value memory caching, redis isn't needed.
Balderdash. memcached is actively hostile to modern infrastructure and it's not being actively developed except for routine maintenance.
The first time someone stores an entire data structure in a memcache key is the moment you've lost. Playing "read blob, deserialize, update, serialize, write blob" is just dumb when you can avoid it.
Other moments of great failure with memcached: needing to implement client-side or proxy-based replication, having the memcached process refuse to cache more things because a slabs become broken, the first time your developers don't understand memcached LRU is per-size category and not global, ....
Just Say No to Memcached.
Something does not add up about hive: They say it has 300 PB, and it generates 4PB per day - which means, at this rate, all data was generated within the last 75 days.