Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

github.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#21

Post by InclinedPlane »

timaelliott wrote: post #8
achompas wrote: post #6

Earlier quoted context omitted.

Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging. >42M uniques last month.[0] Are you really going to assert Twitter hasn't dealt with amazing or challenging hurdles in getting this far? [0] http://siteanalytics.compete.com/twitter.com/ EDIT: this ignores t…

Yes. In 2008/2009, another engineer and myself built an ad-platform that received around 500M impressions per day, 5M clicks per day. And it wasn't just recording a tweet or publishing out to followers. We took the user input query, had to do some keyword/relevancy targeting, geofiltering, matching to advertisers and deliver back a large result set of adverts. All within 100ms. Our platform was also apache, mod_php,…

You are off by nearly 2 orders of magnitude from twitter's scale. They have billions of views per day and each of those views is a stream comprised of hundreds of different sub-streams.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#22

Post by achompas »

timaelliott wrote: post #8
achompas wrote: post #6

Earlier quoted context omitted.

Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging. >42M uniques last month.[0] Are you really going to assert Twitter hasn't dealt with amazing or challenging hurdles in getting this far? [0] http://siteanalytics.compete.com/twitter.com/ EDIT: this ignores t…

Yes. In 2008/2009, another engineer and myself built an ad-platform that received around 500M impressions per day, 5M clicks per day. And it wasn't just recording a tweet or publishing out to followers. We took the user input query, had to do some keyword/relevancy targeting, geofiltering, matching to advertisers and deliver back a large result set of adverts. All within 100ms. Our platform was also apache, mod_php,…

See my edit. Your ad impressions reached approximately 3% of Twitter's daily request load last year. Note that those requests can serve up to 200 tweets + metadata.

This doesn't account for Twitter's budding ad service, which one can assume has some of the same functionality (targeted advertising, information retrieval) as traditional ad networks.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#23

Post by InclinedPlane »

burningout wrote: post #14
InclinedPlane wrote: post #12

Earlier quoted context omitted.

What does content size matter? The challenge is that every single page of content except for each individual tweet is utterly unique for every user. That defeats the vast majority of straightforward caching implementations. You can't cache fully rendered pages ever because the chance that one random timeline view at a given time will be identical to any other view (even by the same person at a different time) is pret…

Again, as the parent poster also posted, I think you have never worked on large data. Twitter is like a big mailbox, only that every mail only has 160 bytes. This has been solved 10 years ago.

If you treat twitter like a big mailbox, things will work "ok". It's not the worst approach ever, that's for certain. But end-user perceptible performance would be a fraction of what twitter has today.

P.S. How many images does twitter serve up per day at present? That's a tad more than 160 characters of data.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#24

Post by chewxy »

marcusf wrote: post #15
timaelliott wrote: post #8

Earlier quoted context omitted.

Yes. In 2008/2009, another engineer and myself built an ad-platform that received around 500M impressions per day, 5M clicks per day. And it wasn't just recording a tweet or publishing out to followers. We took the user input query, had to do some keyword/relevancy targeting, geofiltering, matching to advertisers and deliver back a large result set of adverts. All within 100ms. Our platform was also apache, mod_php,…

So you had 500m reads on a relatively static data set + 5m writes on an unrelated log? Sounds like a fun problem, but I agree I doesn't sound like rocket science. On the other hand, it also doesn't sound like Twitter, having 400m writes per day, and 400*x million reads on that very dynamic data set. Just seems that's a slightly harder problem.

Adserving is not really static. Cachebusters are named so for good reason. Nowadays ad server developers are clever enough to separate click tracking and impression tracking (the non-Enterprise version of OpenX still deserves a lot of ಠ_ಠ though).

In an RTB environment, there is an additional constraint of having to serve up your ad (or decision) within 60ms (Google ADX sets a hard limit of 80ms), and the fastest best bid wins.

I don't think that's a less hard problem compared to Twitter, especially at high volumes. You can't just say "scale sideward!".

That said, the first link was totally misleading. I was actually quite shocked to see that Twitter only had 42M uniques per month, because a typical ad network does a lot more

EDIT: ah.. 15B requests/day makes more sense. Wtf is with the wrong stats?

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#25

Post by jasonwatkinspdx »

InclinedPlane wrote: post #23
burningout wrote: post #14

Earlier quoted context omitted.

Again, as the parent poster also posted, I think you have never worked on large data. Twitter is like a big mailbox, only that every mail only has 160 bytes. This has been solved 10 years ago.

If you treat twitter like a big mailbox, things will work "ok". It's not the worst approach ever, that's for certain. But end-user perceptible performance would be a fraction of what twitter has today. P.S. How many images does twitter serve up per day at present? That's a tad more than 160 characters of data.

Another key difference is that email users generally contribute directly to their provider's infrastructure costs in providing email as a service. Email infrastructure (and the user experience) is fragmented, and global funding generally scales with global load.

Instead twitter must monetize via advertising of some form, and so the percentage of folks who do not respond to ads acts as a really strong factor in your cost calculations. In this sense, email software has it easy, and can be extremely wasteful in the resources it consumes.

It's not just that the availability expectations of twitter are higher than email, it's also that the economic base of the infrastructure is far more sparse.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#27

Post by InclinedPlane »

jasonwatkinspdx wrote: post #25
InclinedPlane wrote: post #23

Earlier quoted context omitted.

If you treat twitter like a big mailbox, things will work "ok". It's not the worst approach ever, that's for certain. But end-user perceptible performance would be a fraction of what twitter has today. P.S. How many images does twitter serve up per day at present? That's a tad more than 160 characters of data.

Another key difference is that email users generally contribute directly to their provider's infrastructure costs in providing email as a service. Email infrastructure (and the user experience) is fragmented, and global funding generally scales with global load. Instead twitter must monetize via advertising of some form, and so the percentage of folks who do not respond to ads acts as a really strong factor in your c…

And yet another key difference is that there are few email server installations that support half a billion users. Saying that the scaling problem is "solved" because all you have to do is copy, say, gmail, is kind of silly.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#28

Post by achompas »

chewxy wrote: post #24
marcusf wrote: post #15

Earlier quoted context omitted.

So you had 500m reads on a relatively static data set + 5m writes on an unrelated log? Sounds like a fun problem, but I agree I doesn't sound like rocket science. On the other hand, it also doesn't sound like Twitter, having 400m writes per day, and 400*x million reads on that very dynamic data set. Just seems that's a slightly harder problem.

Adserving is not really static. Cachebusters are named so for good reason. Nowadays ad server developers are clever enough to separate click tracking and impression tracking (the non-Enterprise version of OpenX still deserves a lot of ಠ_ಠ though). In an RTB environment, there is an additional constraint of having to serve up your ad (or decision) within 60ms (Google ADX sets a hard limit of 80ms), and the fastest bes…

Are you talking about 15B vs. the visits chart I linked? If so, the 15B number comes from API calls, which do not have to happen through the website (think of all the Twitter clients).

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#29

Post by SoftwareMaven »

alttab wrote: post #2

I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?

Regardless of the Twitter/Memcache controversy, bad carpenters do blame their tools. Good carpenters succeed in spite of their tools.

Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

#30

Post by chewxy »

achompas wrote: post #28
chewxy wrote: post #24

Earlier quoted context omitted.

Adserving is not really static. Cachebusters are named so for good reason. Nowadays ad server developers are clever enough to separate click tracking and impression tracking (the non-Enterprise version of OpenX still deserves a lot of ಠ_ಠ though). In an RTB environment, there is an additional constraint of having to serve up your ad (or decision) within 60ms (Google ADX sets a hard limit of 80ms), and the fastest bes…

Are you talking about 15B vs. the visits chart I linked? If so, the 15B number comes from API calls, which do not have to happen through the website (think of all the Twitter clients).

Requests are requests.15B is a gigantic amount.
Post reply on HN