Twitter says they fixed the Memcache calcification problem. Dormando disagrees. • Hacker News Board

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#11

Post by shadowfiend » Wed, Jul 11, 2012, 6:15 AM UTC

timaelliott wrote: post #8

achompas wrote: post #6

Earlier quoted context omitted.

Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging. >42M uniques last month.[0] Are you really going to assert Twitter hasn't dealt with amazing or challenging hurdles in getting this far? [0] http://siteanalytics.compete.com/twitter.com/ EDIT: this ignores t…

Yes. In 2008/2009, another engineer and myself built an ad-platform that received around 500M impressions per day, 5M clicks per day. And it wasn't just recording a tweet or publishing out to followers. We took the user input query, had to do some keyword/relevancy targeting, geofiltering, matching to advertisers and deliver back a large result set of adverts. All within 100ms. Our platform was also apache, mod_php,…

You understand that 400M tweets a day is the number of tweets posted to their system, right? That speaks not at all to the consumption of those tweets, which is the metric you're using for your ad platform.

Additionally, they don't just deal with 160 characters, because again, somehow you're still talking about data being posted, and not data being consumed. Data is consumed off their site via polling APIs, streaming APIs, and a website, all of which are pushing those 400M tweets a day out to plenty of consumers.

They may not have as ridiculous a scale as they act like they do. But let's be clear: it is nowhere near as trivial as you make it out to be, either. Armchair quarterbacking is always easy, because you aren't exposed to the complexity that arises when you've spent a few months and years hitting the corner cases of the problem you're commenting on.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#12

Post by InclinedPlane » Wed, Jul 11, 2012, 6:21 AM UTC

timaelliott wrote: post #4

alttab wrote: post #2

I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?

Twitter's attitude is laughable. Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.

What does content size matter? The challenge is that every single page of content except for each individual tweet is utterly unique for every user. That defeats the vast majority of straightforward caching implementations. You can't cache fully rendered pages ever because the chance that one random timeline view at a given time will be identical to any other view (even by the same person at a different time) is pretty much as close to zero as possible. Every view is dynamically generated content from up to several hundred or thousand different streams of data and needs to be put in order and have all of the per-user metadata set correctly.

Once you start looking into the actual mathematical constraints of the problem of twitter you realize that it's a scaling nightmare. Hundreds of millions of updates per day and tens of thousands of views per second (billions per day). There's only a few people in the world who have the right to look down on stats like that.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#13

Post by chewxy » Wed, Jul 11, 2012, 6:24 AM UTC

timaelliott wrote: post #8

achompas wrote: post #6

Earlier quoted context omitted.

Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging. >42M uniques last month.[0] Are you really going to assert Twitter hasn't dealt with amazing or challenging hurdles in getting this far? [0] http://siteanalytics.compete.com/twitter.com/ EDIT: this ignores t…

Yes. In 2008/2009, another engineer and myself built an ad-platform that received around 500M impressions per day, 5M clicks per day. And it wasn't just recording a tweet or publishing out to followers. We took the user input query, had to do some keyword/relevancy targeting, geofiltering, matching to advertisers and deliver back a large result set of adverts. All within 100ms. Our platform was also apache, mod_php,…

Add to that, the challenges of sub-60ms RTB. All the fun!

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#14

Post by burningout » Wed, Jul 11, 2012, 6:32 AM UTC

InclinedPlane wrote: post #12

timaelliott wrote: post #4

Earlier quoted context omitted.

Twitter's attitude is laughable. Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.

What does content size matter? The challenge is that every single page of content except for each individual tweet is utterly unique for every user. That defeats the vast majority of straightforward caching implementations. You can't cache fully rendered pages ever because the chance that one random timeline view at a given time will be identical to any other view (even by the same person at a different time) is pret…

Again, as the parent poster also posted, I think you have never worked on large data. Twitter is like a big mailbox, only that every mail only has 160 bytes. This has been solved 10 years ago.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#15

Post by marcusf » Wed, Jul 11, 2012, 6:39 AM UTC

timaelliott wrote: post #8

achompas wrote: post #6

Earlier quoted context omitted.

Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging. >42M uniques last month.[0] Are you really going to assert Twitter hasn't dealt with amazing or challenging hurdles in getting this far? [0] http://siteanalytics.compete.com/twitter.com/ EDIT: this ignores t…

Yes. In 2008/2009, another engineer and myself built an ad-platform that received around 500M impressions per day, 5M clicks per day. And it wasn't just recording a tweet or publishing out to followers. We took the user input query, had to do some keyword/relevancy targeting, geofiltering, matching to advertisers and deliver back a large result set of adverts. All within 100ms. Our platform was also apache, mod_php,…

So you had 500m reads on a relatively static data set + 5m writes on an unrelated log? Sounds like a fun problem, but I agree I doesn't sound like rocket science. On the other hand, it also doesn't sound like Twitter, having 400m writes per day, and 400*x million reads on that very dynamic data set. Just seems that's a slightly harder problem.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#16

Post by jasonwatkinspdx » Wed, Jul 11, 2012, 6:42 AM UTC

InclinedPlane wrote: post #5

alttab wrote: post #2

I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?

Admittedly I am not well versed in Rails at present. However, the question in regard to twitter isn't so much whether Rails can scale but whether it could scale when twitter needed it too, which was a fair number of years ago. Did Mongrel or Unicorn even exist back in, say, 2007?

Mongrel did.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#17

Post by Morg » Wed, Jul 11, 2012, 6:44 AM UTC

InclinedPlane wrote: post #12

timaelliott wrote: post #4

Earlier quoted context omitted.

Twitter's attitude is laughable. Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.

What does content size matter? The challenge is that every single page of content except for each individual tweet is utterly unique for every user. That defeats the vast majority of straightforward caching implementations. You can't cache fully rendered pages ever because the chance that one random timeline view at a given time will be identical to any other view (even by the same person at a different time) is pret…

Why do you people find it necessary to comment on stuff you do not understand at all ?

This is 2012, it would take me about 1 day to provide you with a working twitter viewer that is 100% cachable and for which all tweets are cachable too. All that requires is a one page app for the viewer, and ajax calls for tweets to view like api/tweets/978486971987169876 which will be cachable for everyone.

Content size matters a whole lot because it linearly dictates the size of your server pool. Serving a million small udp packets is pretty standard for any quake server. Serving a million youtube videos is another story.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#18

Post by achompas » Wed, Jul 11, 2012, 6:46 AM UTC

burningout wrote: post #14

InclinedPlane wrote: post #12

Earlier quoted context omitted.

What does content size matter? The challenge is that every single page of content except for each individual tweet is utterly unique for every user. That defeats the vast majority of straightforward caching implementations. You can't cache fully rendered pages ever because the chance that one random timeline view at a given time will be identical to any other view (even by the same person at a different time) is pret…

Again, as the parent poster also posted, I think you have never worked on large data. Twitter is like a big mailbox, only that every mail only has 160 bytes. This has been solved 10 years ago.

Wait, see my comment below. Twitter received 15B (yes, B) API calls/day last July. How does that compare to your typical email client?

I don't want to argue that Twitter is astoundingly hard, but serving ~170K requests/sec can't really be that trivial, even if they're 160 bytes (they're not, since Twitter sends metadata, logs those messages, tracks service metrics, etc. for those messages)

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#19

Post by jasonwatkinspdx » Wed, Jul 11, 2012, 6:49 AM UTC

timaelliott wrote: post #4

alttab wrote: post #2

I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?

Twitter's attitude is laughable. Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.

Your assumptions are laughable. Tweets have considerable metadata that pushes them far beyond 160 bytes. Fragmenting this into a secondary object is counterproductive due to the constant factor of 2 requests vs one larger payload.

Someone who's been around the block a few times understands that it's difficult to make pronouncements without informed observation. That you are not willing to extend twitter's engineering staff the benefit of the doubt considering your lack of visibility into their measurements speaks loudly.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#20

Post by jasonwatkinspdx » Wed, Jul 11, 2012, 6:51 AM UTC

burningout wrote: post #14

InclinedPlane wrote: post #12

Earlier quoted context omitted.

What does content size matter? The challenge is that every single page of content except for each individual tweet is utterly unique for every user. That defeats the vast majority of straightforward caching implementations. You can't cache fully rendered pages ever because the chance that one random timeline view at a given time will be identical to any other view (even by the same person at a different time) is pret…

Again, as the parent poster also posted, I think you have never worked on large data. Twitter is like a big mailbox, only that every mail only has 160 bytes. This has been solved 10 years ago.

If you don't understand that the request distribution matters more than payload size, you aren't even seeing the problems.

I encourage you to analyze infrastructure for a twitter style app using inbox duplication. Once you model this against hardware costs you'll learn something about how utterly expensive write amplification is in a hot data set that must be backed by ram due to availability requirements.