Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
1–10 of 39 posts
Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
#2I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?
Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
#3That response from manjuraj was kind of evil
Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
#4I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?
Twitter's attitude is laughable. Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.
Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
#5I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?
Admittedly I am not well versed in Rails at present. However, the question in regard to twitter isn't so much whether Rails can scale but whether it could scale when twitter needed it too, which was a fair number of years ago. Did Mongrel or Unicorn even exist back in, say, 2007?
Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
#6I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?
Twitter's attitude is laughable. Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.
Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.
>42M uniques last month.[0] Are you really going to assert Twitter hasn't dealt with amazing or challenging hurdles in getting this far?
[0] http://siteanalytics.compete.com/twitter.com/
EDIT: this ignores that twitter.com is not the only Twitter client--they served 15B (!!) requests/day (!!!) as of a year ago.
Not to mention metadata, instrumentation for services, logging, DB backups, and managing configuration of all of those distributed resources. Are we still talking about the ease of 160B?
http://www.readwriteweb.com/hack/2011/07/twitter-serves-more...
Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
#7Just to clarify things since I helped write the initial blog post about twemcache... in our original post, we had an error regarding the slab calcification problem we mentioned, this problem ONLY applied to our v1.4.4 fork of memcached. After speaking with the upstream maintainers, we learned that recent memcached versions have addressed some of these problems. These are the type of conversations we want to have.
At the time we adopted memcached, that's the version we went with and made sure it worked well in our production environment as we scaled as a company. We also open sourced twemproxy [https://github.com/twitter/twemproxy] which is a lightweight proxy for memcached which has worked well for us in combination with twemcache and may work well for others too.
We just want to reiterate that twemcache has worked well for our unique environment and any teams evaluating memcached should try all their try all their options, just like any other piece of software you adopt in your stack.
One of the reasons of open sourcing our work was to share our ideas with the memcached community to see what worked well for us and help everyone. For example, this is also how we treat our work with our MySQL fork [https://github.com/twitter/mysql] which we maintain in the open and have signed an OCA with Oracle to help get work pushed upstream so everyone benefits in the long run.
Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
#8Earlier quoted context omitted.
Twitter's attitude is laughable. Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.
Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging. >42M uniques last month.[0] Are you really going to assert Twitter hasn't dealt with amazing or challenging hurdles in getting this far? [0] http://siteanalytics.compete.com/twitter.com/ EDIT: this ignores t…
Yes.
In 2008/2009, another engineer and myself built an ad-platform that received around 500M impressions per day, 5M clicks per day. And it wasn't just recording a tweet or publishing out to followers. We took the user input query, had to do some keyword/relevancy targeting, geofiltering, matching to advertisers and deliver back a large result set of adverts. All within 100ms.
Our platform was also apache, mod_php, memcached, mysql and rabbitmq. So definitely not the most optimal of platforms by any means. We had two colos with ~20 servers (dell r410s) at each facility.
Twitter just recently announced 400M tweets/day. I'm not trying to brag about my experiences, because looking back now we made numerous amateur mistakes, but just showing that Twitter's "scale" is a joke compared to everyday challenges at any large internet ad network.
Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
#9I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?
Twitter's attitude is laughable. Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.
Decent amount of traffic ?
Sorry but the only thing laughable is that comment.
Re: Twitter says they fixed the Memcache calcification problem. Dormando disagrees.
#10I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?
There may be a difference between scaling well and scaling 9th on alexa well.
Sometimes bad carpenters blame their tools, but sometimes the tools weren't planned for something of that scale.