Live data from Hacker News

Twitter was down

status.twitterstat.us

271–280 of 544 posts

Re: Twitter was down

#271

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

>It has now reached the point where even large companies cannot maintain high reliability.

Waiting for this to be backed up by statistics.

Re: Twitter was down

#272
post #154

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

The brain is the greatest pattern matcher in the world. While it is unlikely all of these companies would have major outages in a month, be wary that the subconscious is constantly generating narratives to explain statistical anomalies. Interesting theories nonetheless:)

> be wary that the subconscious is constantly generating narratives to explain statistical anomalies

This comes up all the time in sports. Let's take pool for example. There are various guesstimates floating around, and I do not have access to detailed tournament statistics, but I have heard that in games where sinking a ball on the break is an advantage, for decent players there's maybe a 75% chance that a ball will go down.

So once in every four breaks, you won't sink a ball. How often do you fail twice in a row? Once in every sixteen breaks. Failing three times in a row? Once in every 64 breaks. Four times in a row? Once in every 256 breaks.

What about five straight breaks without sinking a ball? Once in every 1,024 breaks. That's a lot of breaks. But wait up a moment.

Let's ask, "If you miss a break, what're the odds of it becoming a streak of five misses in a row?" The answer is, "One in every 256 streaks of misses will be a streak of five or more misses." 1/256 is not particularly rare, if you play often enough to sink a ball on the break 75% of the time.

What is the point of knowing that a streak of five misses in a row is rare but not that rare? Well, if you miss five in a row, do you chalk your cue for break number six as usual? Or do you tell yourself that your break isn't working, and start adjusting your stance, aim, spin, &c?

If you start adjusting everything when you get a streak of five misses in a row, you may just make things worse. You have to pay enough attention to your distribution of misses to work out whether a streak of five misses in a row is just the normal 1/256 streaks, or if there really is something amiss.

The brain is a great pattern matcher, but it sucks at understanding statistics.

---

The flip side of this, of course, is that if you upgrade your brain well enough to understand statistics, you can win a lot of money.

If a pro misses five in a row, feel free to wager money that they'll sink a ball on their next break. Your friends may actually give you odds, even though the expectation of winning is 75-25 in your favour.

Re: Twitter was down

#273
post #85

In a slightly better alternate universe it stays down.

I share your sentiment to a certain extent, but I believe Twitter is a necessary evil. There should be an alternative to main stream media for people.

I follow mostly "main stream media" on Twitter. It should be an aggregator of professional news, not "alternative" facts.

Re: Twitter was down

#274

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

The other possibility is intern season (I'm 99.99% joking)

Re: Twitter was down

#275
post #236

Earlier quoted context omitted.

Can someone explain the joke (about the beer) because I genuinely don't understand edit: pretty please

If the ops team got beer every time the servers went down (as a reward for fixing them) then they'd have an incentive for the servers to go down.

We all understand the perverse incentives joke, I think what's confusing people here is whether there's some other hidden joke they're missing that suggests not to take OP at his word that yes, he did make off with someone else's gift, which is generally considered a dick move.

Re: Twitter was down

#276

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

Can someone explain the joke (about the beer) because I genuinely don't understand edit: pretty please

He took the beer because he wanted it. "Perverse incentives" are an excuse, because nobody is going to kill their production servers and all the panic that entails for like $10 worth of beer.

Re: Twitter was down

#277

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

Taking down Twitter could be a huge boon for the economy, though.

Re: Twitter was down

#278

Earlier quoted context omitted.

Twitter is especially weird for this since it's often a platform where people talk about downtimes. I don't see this downtime mentioned on Reddit and I don't know of other sites where it might be discussed, so if Hacker News happened to go down at the same time, where would I go to talk about it with online strangers and find out if it's just me? Nowhere, I guess, I'd just wait it out with no extra insights on what's…

This gets pretty rowdy: https://downdetector.com/status/twitter

For a simpler view of things, there's also https://downforeveryoneorjustme.com

Re: Twitter was down

#279

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

#4

I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4.

#1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the new shiny things. The sinusoidal pattern of exposure will continue.

Re: Twitter was down

#280

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

To take the history of direct queries at Twitter even further back, I built a web interface at Odeo for the CEO to run direct queries against the database (and save them so he could re-run them). There were some basic security precautions, but this was totally cowboy. That Odeo team was filled with best practices aficionados and the management (including me) was a bit cowardly about being clear that "WE ARE FAILING H…

Why not have it run against a replicated copy? I did that in the past, works amazingly, they can f* up all they want without any implications.
Post reply on HN