Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

251–260 of 544 posts

Re: Twitter was down

#251

Post by Raphmedia »

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

> I would intercept these rewards and put them in my backpack for the bus ride home, in order to avoid creating perverse incentives for the operations team. But did anyone call me 'hero'?

Wait, I don't understand.

Why would anyone call you hero?

Are you suggesting that the team would deliberately crash the app to receive beers and that by stealing them you stopped this from happening?

Free drinks and free food is the standard here to reward teams when they spend extra unpaid time away from their families.

All of the posts asking the same question are being down voted. Am I missing something?

You said you were a contractor at the time. Unless you were on the management team I fail to see how this was your responsibility to choose what happened.

Re: Twitter was down

#252

Post by alanbernstein »

dmix wrote: post #228
elamje wrote: post #154

Earlier quoted context omitted.

The brain is the greatest pattern matcher in the world. While it is unlikely all of these companies would have major outages in a month, be wary that the subconscious is constantly generating narratives to explain statistical anomalies. Interesting theories nonetheless:)

The more the conspiracy grows the faster these otherwise minor stories shoot to the top of HN and add to the pattern. It fuels itself.

"Minor" seems inappropriate. Can you remember another time when so many high-profile websites/services have had outages in so short a time span?

Re: Twitter was down

#253

Post by baud147258 »

colpabar wrote: post #166
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

Can someone explain the joke (about the beer) because I genuinely don't understand edit: pretty please

Each time the ops team brought Twitter back up, they receive good beer. So it would also mean that each time Twitter goes down, they could expect to receive the beer. Without idleword's actions, they would have an incentive (good beer) for having Twitter keep going down and not doing work to improve the stability.

Re: Twitter was down

#254

Post by t0astbread »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

Or maybe it's because the internet is tendentially becoming just a few companies' data centers? Afaik Twitter moved to GCP a few months ago. Maybe this is another Google outage?

Re: Twitter was down

#255

Post by whatshisface »

lordnacho wrote: post #242
whatshisface wrote: post #69

I remember once we were at three outages, someone posted that they thought three was a reasonably-sized random cluster given the rate at which services go down. How many outages have we had in the last 30 days, how many do we have per month on average, and how strongly can we reject the null hypothesis? The formula for computing how unlikely this is is the Poisson distribution: `λ^k * e^-λ / k!`, where λ is the avera…

The outages might not be independent. Chances are these services are cross integrated at some level.

They are, but they're going down on different days. Whatever effect is left over could be accounted for by looking at the postmortems and not counting "we were down because AWS was down."

Re: Twitter was down

#257

Post by nthj »

tonystubblebine wrote: post #233
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

To take the history of direct queries at Twitter even further back, I built a web interface at Odeo for the CEO to run direct queries against the database (and save them so he could re-run them). There were some basic security precautions, but this was totally cowboy. That Odeo team was filled with best practices aficionados and the management (including me) was a bit cowardly about being clear that "WE ARE FAILING H…

Just a quick note not directed at OP but for any other engineers that may be unaware, these days AWS makes provisioning a read replica painless, and you can point the CEO to up-to-the-minute data while essentially firewalling the queries from customer operations.

Re: Twitter was down

#258

Post by whatshisface »

elamje wrote: post #154
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

The brain is the greatest pattern matcher in the world. While it is unlikely all of these companies would have major outages in a month, be wary that the subconscious is constantly generating narratives to explain statistical anomalies. Interesting theories nonetheless:)

Saying that it's an illusory pattern without checking the statistics is no more scientific than saying it's a conspiracy without checking the statistics.

Re: Twitter was down

#259

Post by colpabar »

slackfan wrote: post #247
treis wrote: post #236

Earlier quoted context omitted.

If the ops team got beer every time the servers went down (as a reward for fixing them) then they'd have an incentive for the servers to go down.

What the hell are all of you smoking, some moderately expensive alcohol is nowhere near enough reward to take down a service.

If it was a sure thing that the ops engineers were doing that, then sure, it'd be kinda funny. Otherwise it just seems like a dick move.

Re: Twitter was down

#260

Post by kgraves »

jbverschoor wrote: post #17

What’s the fallback for twitter if twitter is down, or worse: stops

Any decentralised social media alternative that is not Twitter, like Mastodon.

Anything else is centralised and inherently evil. Even the orange website.

Post reply on HN