Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A decade of major cache incidents at Twitter

danluu.com

Post reply on HN
Print view View original

11–20 of 27 posts

Re: A decade of major cache incidents at Twitter

#11

Post by tluyben2 »

acdha wrote: post #9
YEwSdObPQT wrote: post #8

Earlier quoted context omitted.

It ultimately depends at what scale. I think most people to be fair are talking about building a clone of what was present back in the mid 2000s. You could build a twitter clone that could handle a few hundred users in a few weeks with modern tech stacks. The fact that there are at least 3 twitter clones that are less well put together with a decent amount of users handling in the load proves that it is possible.

“A few weeks” sounds a lot longer than a weekend, and I’d also consider the history: Twitter itself was built quickly using a modern stack. Rails was highly productive, the problem is that the concept of the service makes scaling non-trivial. We have more RAM and SSDs now so you could get further but those aren’t magic.

Twitter was down all the time for hours after the launch. I don’t think I have an issue coding something that goes down when it overloads with functionality twitter had when it launched in a weekend. Most work for that kind of project goes into interpreting the specs/your business colleagues and fixing the mishaps; here you don’t have that.

Re: A decade of major cache incidents at Twitter

#12

Post by rightbyte »

jsty wrote: post #2

Major incidents aside, I always think that cache-related bugs are some of the most likely to go undetected since if you don't test for them end-to-end, they're really not that easy to spot & diagnose. An article sticking around too long on the home page. Semi-stale data creeping into your pipeline. Someone's security token being accepted post-revocation. All really hard to spot unless (1) you're explicitly looking, o…

Microsoft has some serious problem with token caching. I changed job last month and for two three weeks I could log into my old work account for a split second before being thrown out. (By habit visited the page). I could see the news feed and mails but not long enought to see if they were stale.

Re: A decade of major cache incidents at Twitter

#13

Post by tluyben2 »

terom wrote: post #6

Required reading for all of the "I could code up Twitter in a weekend" -types. The long listen queue -> multiple queued up retries feedback loop is a classic: https://datatracker.ietf.org/doc/html/rfc896 TCP/IP "congestion collapse" and the 1986 Internet meltdown [various sources]

Those remarks are always made at launch, not later on. Dropbox and twitter, both of which people said this about, were rather trivial at launch especially with modern tooling. They also, and especially twitter, had growing pains. Twitter defo prioritised move fast and break things.

Obviously copying decades of improvements and scaling lessons you cannot copy unless someone made a product of those parts and you can use those.

Re: A decade of major cache incidents at Twitter

#14

Post by acdha »

tluyben2 wrote: post #11
acdha wrote: post #9

Earlier quoted context omitted.

“A few weeks” sounds a lot longer than a weekend, and I’d also consider the history: Twitter itself was built quickly using a modern stack. Rails was highly productive, the problem is that the concept of the service makes scaling non-trivial. We have more RAM and SSDs now so you could get further but those aren’t magic.

Twitter was down all the time for hours after the launch. I don’t think I have an issue coding something that goes down when it overloads with functionality twitter had when it launched in a weekend. Most work for that kind of project goes into interpreting the specs/your business colleagues and fixing the mishaps; here you don’t have that.

Don’t forget that you’re talking 2006, so you need to be doing a lot more infrastructure work: no containers, shared hosting environments are less stable but bare metal costs a fair amount to get started, you’re using something like cfengine instead of Chef/Ansible if you aren’t setting everything up by hand, you have 10% of the RAM and no SSDs, CDNs are an expensive premium service, etc. Then think about what that means browser-wise: you can do a bit on the client side but server side rendering is a necessity and you’re still going to be burning time on browser compatibility to an extent which can be hard to remember now. HTML5 hasn’t happened yet so you’re building more stuff yourself, too.

I’m not saying there’s nothing they could have done better, just that there’s an awful lot they couldn’t have avoided at least without building a very different app.

Re: A decade of major cache incidents at Twitter

#16

Post by formerly_proven »

spoonjim wrote: post #15

“ On Nov 8, a user changed their name from tigertwo to Woflstar_Bachi.” Horrifically inappropriate inclusion of PII in this post. Didn’t someone at legal go through this?

It's a current, public profile:

> Wolfstar_Bachi @tigertwo

> Wolfstar is an online and social media PR agency that specialises in helping some of the world’s best companies to communicate more effectively.

Re: A decade of major cache incidents at Twitter

#17

Post by tluyben2 »

acdha wrote: post #14
tluyben2 wrote: post #11

Earlier quoted context omitted.

Twitter was down all the time for hours after the launch. I don’t think I have an issue coding something that goes down when it overloads with functionality twitter had when it launched in a weekend. Most work for that kind of project goes into interpreting the specs/your business colleagues and fixing the mishaps; here you don’t have that.

Don’t forget that you’re talking 2006, so you need to be doing a lot more infrastructure work: no containers, shared hosting environments are less stable but bare metal costs a fair amount to get started, you’re using something like cfengine instead of Chef/Ansible if you aren’t setting everything up by hand, you have 10% of the RAM and no SSDs, CDNs are an expensive premium service, etc. Then think about what that m…

Yes, I agree with that. Containers though, as far as some of the advantages; we have been using chroots for deployment since the early 2000s which is not the same but deployment/compatibility wise it was pretty good. It allows you to have the same small Linux image and deployment everywhere as well and you could move most zipped images from machine to machine with vastly different kernels. I still use chroots now on my Pandora handheld which has an ancient kernel but I run modern software on it in a chroot. No overhead too.

I think what they could have done better is rails; it was not good enough then. Php would’ve been far less hassle. But he, they made it!

Re: A decade of major cache incidents at Twitter

#18

Post by daper »

jsty wrote: post #2

Major incidents aside, I always think that cache-related bugs are some of the most likely to go undetected since if you don't test for them end-to-end, they're really not that easy to spot & diagnose. An article sticking around too long on the home page. Semi-stale data creeping into your pipeline. Someone's security token being accepted post-revocation. All really hard to spot unless (1) you're explicitly looking, o…

I categorize this as bugs caused by data inconsistency because od data duplication. That includes:

- Using asynchronous database replication and reading data from database slaves - Duplicating same data over multiple database tables (possibly for performance reasons) - Having additional system that duplicates some data. For example: in the middle of rewriting some legacy system - a process that was split into phases so functionality between new and old systems overlap for some period of time.

Based on my experience I always assume that inconsistency is unavoidable when the same information is stored in more than one place.

Re: A decade of major cache incidents at Twitter

#19

Post by acdha »

tluyben2 wrote: post #17
acdha wrote: post #14

Earlier quoted context omitted.

Don’t forget that you’re talking 2006, so you need to be doing a lot more infrastructure work: no containers, shared hosting environments are less stable but bare metal costs a fair amount to get started, you’re using something like cfengine instead of Chef/Ansible if you aren’t setting everything up by hand, you have 10% of the RAM and no SSDs, CDNs are an expensive premium service, etc. Then think about what that m…

Yes, I agree with that. Containers though, as far as some of the advantages; we have been using chroots for deployment since the early 2000s which is not the same but deployment/compatibility wise it was pretty good. It allows you to have the same small Linux image and deployment everywhere as well and you could move most zipped images from machine to machine with vastly different kernels. I still use chroots now on…

I used chroots, too, and it was useful but much harder to maintain than a container. Automation wasn't impossible, of course, but that was also complicated by concerns about bloating each chroot with copies of all of the system libraries & config files.

I used PHP in that era. It could be faster but then you're in the classic developer productivity tradeoff between, say, hand-coded SQL calls versus using an ORM, etc. The PHP frameworks which were comparable productivity-wise to Rails were also a lot closer to Rails performance-wise since they also had heavy abstractions, and they tended to have even more creative ways to create security holes. (I am feeling very old remembering arguing against enabling register_globals circa 1998)

Re: A decade of major cache incidents at Twitter

#20

Post by Kavelach »

dspillett wrote: post #7
mprovost wrote: post #5

"There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton https://martinfowler.com/bliki/TwoHardThings.html

There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors.

There are only two hard problems in Computer Science: there's one joke, and it's not even funny
Post reply on HN