Twitter was down

Post by Balgair » Thu, Jul 11, 2019, 9:15 PM UTC

degenerate wrote: post #363

gonehome wrote: post #335

Earlier quoted context omitted.

Yep, this also matches what I've heard through the grapevine. Pushing bad regex to production, chaos monkey code causing cascading network failure, etc. They're just different accidents for different reasons. Maybe it's summer and people are taking more vacation?

I actually like the summer vacation hypothesis. Makes the most sense to me - backup devs handling some things they are not used to.

Post by jfountain2015 » Thu, Jul 11, 2019, 9:21 PM UTC

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

Post by bin0 » Thu, Jul 11, 2019, 9:24 PM UTC

vkou wrote: post #324

rossdavidh wrote: post #300

Earlier quoted context omitted.

The assumption is that Russia, China, Iran are less dependent on Google, Twitter, etc., in part because some of them aren't allowed to operate in those countries, in part because some of them are much less dominant in those markets. 'Us' means 'people who might care that Twitter (or whoever) is down'.

Google, Twitter, Reddit, Facebook, etc, all legally operate in Russia.

Post by frogpelt » Thu, Jul 11, 2019, 9:27 PM UTC

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

Post by liberte82 » Thu, Jul 11, 2019, 9:29 PM UTC

bdd wrote: post #331

kwizzt wrote: post #312

Earlier quoted context omitted.

How does the fact you worked at those companies relate to #4? Edit: I misread the parent and my question doesn't make a lot of sense. Please ignore it :)

> How does the fact you worked at those companies relate to #4? For Facebook I worked on the incident, previous Wednesday. 9.5 hours of pain... And for my past employers, I still have friends there texting the root causes with facepalm emojis.