Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

461–470 of 544 posts

Re: Twitter was down

#461

Post by vokep »

colpabar wrote: post #166
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

Can someone explain the joke (about the beer) because I genuinely don't understand edit: pretty please

He's taking home the special expensive beer and not telling them about it because he cares about the health and well being of his team so much, and yet they wouldn't even consider him a hero for this, how ungrateful they are!

Re: Twitter was down

#462

Post by cfors »

KuhlMensch wrote: post #456

Years ago I read an amazing article (from HN) about how (complex) config, rather than code ends up being the cause of outages at scale. I always reflect on that, when designing almost anything these days.

I would love to see that article. That isn't surprising in the slightest to me.

Just a quick nitpick. A bad config more often than not in my experience is opening up a code path that is riddled with bad code, whether it was not vetted with the proper testing or the wrong environment.

But to your point, I think most people would agree that configuration changes are almost never reviewed with the granularity of a code change. Yes, we may do our due diligence with an approved PR and vetting the configuration and testing the change before deploying it. But, reviewing a PR with a bad config change in json or yaml doesn't necessarily tell you about the code paths that it will open up which makes it much harder to reason about the consequences that a potential bad config push would do.

We should always be reflecting about how adding knobs (configuration) to our programs greatly increases the complexity of the service.

Re: Twitter was down

#463

Post by jmharvey »

kenhwang wrote: post #364
gonehome wrote: post #335

Earlier quoted context omitted.

Yep, this also matches what I've heard through the grapevine. Pushing bad regex to production, chaos monkey code causing cascading network failure, etc. They're just different accidents for different reasons. Maybe it's summer and people are taking more vacation?

I'm more partial to the summer interns hypothesis.

I agree with this, but to be clear, the "summer interns hypothesis" is not "summer interns go around breaking stuff," it's "the existing population of engineers has finite resources, and when the interns and new grads show up, a lot of those resources go toward onboarding/mentoring the new people, so other stuff gets less attention."

Re: Twitter was down

#464

Post by abstract7 »

My guess is that the whales have been securing parts of their codebases from internal leaks or something related but for security. Workflow disruptions. It may be bad code bitting them weeks or more after they pushed it.

There has been many embarrassing and controversial leaks this year. Allegations of uneven TOS enforcement. Hence the WH Social Media Summit. Could also be security related combo ahead of the elections that also is a bit sensitive for low-trust devs.

Imagine code getting pushed that only a smaller subset of devs are privy to. Possibly pushing obsfucated code or launching services outside of the standard pipeline.

Remember that the spectre and meltdown patches for the Linux kernal was a nightmare because the normal open and free-to-discuss-and-review workflow was broken. That applies too in these situations with large codebases that internally are 'open-source'.

Re: Twitter was down

#465

Post by rossdavidh »

root_axis wrote: post #447
rossdavidh wrote: post #115

Earlier quoted context omitted.

Oh, sure. But Apple, Google, Cloudflare, Stripe, Slack, Microsoft, we're getting to more than five even...

The logic of the GP still applies though. Sites have outages every day so it is inevitable that some large sites will fail around the same time. Also, we know that Cloudflare and Twitter outages were attributed to configuration changes, probably others have benign explanations as well.

Sure, but "configuration changes" does not exclude several of these options. For example, is it harder to predict/deal with the consequences of configuration changes than it used to be?

Re: Twitter was down

#466

Post by rossdavidh »

AnIdiotOnTheNet wrote: post #173
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

It's #1. The real question is how this isn't blindingly obvious to everyone.

One possible answer: it's hard to admit that what you've worked really hard at becoming an expert in, might have been a mistake.

Re: Twitter was down

#467

Post by teraflop »

mlevental wrote: post #435
nthj wrote: post #257

Earlier quoted context omitted.

Just a quick note not directed at OP but for any other engineers that may be unaware, these days AWS makes provisioning a read replica painless, and you can point the CEO to up-to-the-minute data while essentially firewalling the queries from customer operations.

how?

First Google result for "aws read replicas": https://aws.amazon.com/rds/details/read-replicas/

> Using the AWS Management Console, you can easily add read replicas to existing DB Instances. Use the "Create Read Replica" option corresponding to your DB Instance in the AWS Management Console.

Re: Twitter was down

#468

Post by foobiekr »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

4.

And imho all that’s really happening is people are noticing the outages more. This is a good thing. For years too much of the mental model has been “{ cloud, google, Facebook, aws, xxx } never goes down!”

That’s been unhealthy. It’s a good thing.

Re: Twitter was down

#469

Post by uxp »

gcbw2 wrote: post #432
wtallis wrote: post #204

Earlier quoted context omitted.

"Celebrities die 2.7183 at a time": http://ssp.impulsetrain.com/celebrities.html

So the only take away is that now the population at large notices tech companies outages as much as they notice celebrity deaths?

"population at large"

This thread is linked to a status page run by Twitter, on a programming and technology news site. I'm not really seeing how most people that exist in the western/1st world are noticing this. Is there a CNN article, or FoxNews segment on how tech companies are having outages?

Re: Twitter was down

#470

Post by dillonmckay »

hn_throwaway_99 wrote: post #446
JaRail wrote: post #181

Earlier quoted context omitted.

First, I think our general uptime metrics are trending upwards. Recovery times tend to be much shorter as well. Big services are bigger, more mission-critical parts can fail. Continuous development culture is designed with failure as part of the process. We don't spend time looking for obscure issues when they'll be easier to find by looking at metrics. This is fine when a staggered deployment can catch an issue with…

> We notice errors more now. Mistakes are instantly news. Heck, just look at Twitter itself from its original "Fail Whale" days where there was so much downtime, to now where even this relatively small amount of downtime is the top story on HN for hours.

So, when it went down, was there a Fail Whale displayed during this most recent incident?
Post reply on HN