Twitter was down • Hacker News Board

Re: Twitter was down

#181

Post by JaRail » Thu, Jul 11, 2019, 7:41 PM UTC

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

First, I think our general uptime metrics are trending upwards. Recovery times tend to be much shorter as well.

Big services are bigger, more mission-critical parts can fail.

Continuous development culture is designed with failure as part of the process. We don't spend time looking for obscure issues when they'll be easier to find by looking at metrics. This is fine when a staggered deployment can catch an issue with a small number of users. It's bad when that staggered deployment creates a side-effect that isn't fixed by rolling it back. Much harder to fix corrupted metadata, etc.

Automated systems can propagate/cascade/snowball mistakes far more quickly than having to manually apply changes.

We notice errors more now. Mistakes are instantly news.

Re: Twitter was down

#182

Post by daveguy » Thu, Jul 11, 2019, 7:42 PM UTC

pcora wrote: post #4

google, apple, microsoft, facebook.. and now twitter? I keep asking the same, when is amazon's outage day?

Hm. If you were going to plan a worldwide internet outage (or the appearance of one) you could test your ability to take down individual services first and then take them all down at once.

Re: Twitter was down

#183

Post by sho » Thu, Jul 11, 2019, 7:42 PM UTC

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

[flagged]

Re: Twitter was down

#184

Post by yyyk » Thu, Jul 11, 2019, 7:42 PM UTC

idlewords wrote: post #114

dubcanada wrote: post #106

Earlier quoted context omitted.

"I would intercept these rewards and put them in my backpack for the bus ride home, in order to avoid creating perverse incentives for the operations team. But did anyone call me 'hero'?" Wait so you stole rewards for a team that was spending time (I assume extra or stressful) on something you didn't do or have any part in. And you want a cookie? I mean I get it, the company was probably not great in it's infancy. Bu…

I don't want a cookie; I want more $24/bottle Belgian beer.

Wouldn't that have made you the one with a "perverse incentive"?

Re: Twitter was down

#185

Post by idlewords » Thu, Jul 11, 2019, 7:42 PM UTC

ParadisoShlee wrote: post #175

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

Is that beer story satire?

No, it is true.

Re: Twitter was down

#186

Post by sandworm101 » Thu, Jul 11, 2019, 7:42 PM UTC

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

>> he hit Control-C, and Twitter came back up.

Monolithic architecture. When I did security work I fought this every day. Moving away from it is a nightmare of technical debt and heated debate about who should control what. I'm reminded of a story from the early days of MSN. The legend goes that in the late 90s MSN ran out of one cabinet, a single server. The server had redundant power supplies, but only one physical plug.

Re: Twitter was down

#187

Post by narrator » Thu, Jul 11, 2019, 7:42 PM UTC

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

6) White House social media conference just started.

https://www.10tv.com/article/trump-hosts-white-house-summit-...

Re: Twitter was down

#188

Post by Zebfross » Thu, Jul 11, 2019, 7:42 PM UTC

colpabar wrote: post #166

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

Can someone explain the joke (about the beer) because I genuinely don't understand edit: pretty please

Sounds like the guy was bragging about his SQL skills to avoid locking the database but ended up locking the database anyway (thus, people running around)

Re: Twitter was down

#189

Post by kyledrake » Thu, Jul 11, 2019, 7:42 PM UTC

rvz wrote: post #104

0xFFFE wrote: post #85

Earlier quoted context omitted.

I share your sentiment to a certain extent, but I believe Twitter is a necessary evil. There should be an alternative to main stream media for people.

I found Gab [0] to be quite a viable alternative to Twitter due to its decentralised nature thanks to Mastodon so that downtimes like this cannot happen on the platform. [0] - https://gab.ai

I would take an extremely centralized Twitter to a vaguely federated neo-nazi platform a billion times over.

Re: Twitter was down

#190

Post by moret1979 » Thu, Jul 11, 2019, 7:42 PM UTC

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

One possibility on 5) Too many KPIs and quarter goals to be reached, too many corners cut.