Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

441–450 of 544 posts

Re: Twitter was down

#441

Post by icebraining »

gcbw2 wrote: post #428
bdd wrote: post #279

Earlier quoted context omitted.

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

since all of them happen in high profile business hours, i'd guess either #1 or #5. For #4 to be the actual cause, outages out of business hours would be more prevalent and longer.

Of course it went down during business hours, that's when people are deploying stuff. It's known that services are more stable during the weekends too.

Re: Twitter was down

#442

Post by meristem »

johngalt wrote: post #437
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

5) Operational reliability is both difficult and unsexy. The fancy new feature, increasing traffic, or adding AI to something will generate headlines, accolades, and positive attention. Not having outages is something everyone expects by default. This goes double for work that prevents outages. No one wins awards for what doesn't happen. How many medals are pinned on the guys installing fire sprinklers?

Corollary: Work that prevents outages--or safe work--is SO unsexy it does not get noticed, but work that causes outages is postmortem-ed to death (pun intended).

Re: Twitter was down

#443

Post by jrockway »

bobthepanda wrote: post #366
kenhwang wrote: post #364

Earlier quoted context omitted.

I'm more partial to the summer interns hypothesis.

Rule one of having interns and retaining your sanity is that interns get their own branch to muck around in.

Rule one of having a useful intern experience is to get them writing production code as quickly as possible. They check in their first change? Get that thing into production immediately. (If it's going to destabilize the system, why did you approve the CL? You two probably pair programmed the whole thing together.)

Re: Twitter was down

#445

Post by daveheq »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

I don't believe it's too complex, I believe people are getting lazy. Complexity can be handled by automation, but too often people just want to rush things out for a buck instead of planning out a good product.

Re: Twitter was down

#446

Post by hn_throwaway_99 »

JaRail wrote: post #181
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

First, I think our general uptime metrics are trending upwards. Recovery times tend to be much shorter as well. Big services are bigger, more mission-critical parts can fail. Continuous development culture is designed with failure as part of the process. We don't spend time looking for obscure issues when they'll be easier to find by looking at metrics. This is fine when a staggered deployment can catch an issue with…

> We notice errors more now. Mistakes are instantly news.

Heck, just look at Twitter itself from its original "Fail Whale" days where there was so much downtime, to now where even this relatively small amount of downtime is the top story on HN for hours.

Re: Twitter was down

#447

Post by root_axis »

rossdavidh wrote: post #115
idlewords wrote: post #105

Earlier quoted context omitted.

Write a script to fire random events and you will notice they sometimes cluster in ways that look like a pattern.

Oh, sure. But Apple, Google, Cloudflare, Stripe, Slack, Microsoft, we're getting to more than five even...

The logic of the GP still applies though. Sites have outages every day so it is inevitable that some large sites will fail around the same time. Also, we know that Cloudflare and Twitter outages were attributed to configuration changes, probably others have benign explanations as well.

Re: Twitter was down

#448

Post by Diederich »

bdd wrote: post #279
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

I've also worked at a couple of the companies involved.

This is the correct analysis on every level.

Re: Twitter was down

#449

Post by hn_throwaway_99 »

Dylan16807 wrote: post #440
hn_throwaway_99 wrote: post #200

Earlier quoted context omitted.

> Monolithic architecture. This particular problem had nothing to do with a monolithic architecture. Your app can be a monolith, but that still doesn't mean your BI team can't have a separate data warehouse or at least separate read replicas to run queries against.

It's not "nothing to do with". You're correct that a monolithic architecture does not imply that a single read query will lock the entire database. But it is a prerequisite .

Not really. I've seen more than one microservice-architected (admittedly, poorly) systems where, instead of the whole DB freezing up, just the one DB would freeze, but then all of the other microservices that talked to the frozen microservice didn't correctly handle the error responses, so now you had corruption strewn over multiple databases and services.

So, while true the failure mode would be different, "one bad query fucking up your entire system" is just as possible with microservices.

Post reply on HN