Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

471–480 of 544 posts

Re: Twitter was down

#471

Post by gjs278 »

bdd wrote: post #279
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

it sounds like you’re the common factor between the outages. where else have you worked, maybe we can predict the next failure

Re: Twitter was down

#472

Post by uber-employee »

Balgair wrote: post #381
degenerate wrote: post #363

Earlier quoted context omitted.

I actually like the summer vacation hypothesis. Makes the most sense to me - backup devs handling some things they are not used to.

So, a reverse Eternal-September? It'll get better once everyone is back from August vacations?

No, because it’ll only get better until next summer.

Re: Twitter was down

#473

Post by mruts »

rossdavidh wrote: post #143
idlewords wrote: post #105

Earlier quoted context omitted.

Write a script to fire random events and you will notice they sometimes cluster in ways that look like a pattern.

You know, it would be cool if you found stats on the downtime metrics of these various high-profile recent outages, and calculated the odds of having such a cluster. Statistics is hard, though, and avoiding a "Texas Bulls-eye" would be hard.

I mean, we can assume the downtime variance follows a normal distribution. It should pretty easy to calculate P<.05 with just a little bit of data.

Re: Twitter was down

#474

Post by ziftface »

tonystubblebine wrote: post #233
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

To take the history of direct queries at Twitter even further back, I built a web interface at Odeo for the CEO to run direct queries against the database (and save them so he could re-run them). There were some basic security precautions, but this was totally cowboy. That Odeo team was filled with best practices aficionados and the management (including me) was a bit cowardly about being clear that "WE ARE FAILING H…

Thanks for sharing. Out of curiosity, why was the answer no? Was the issue the downtime or something more subtle?

Re: Twitter was down

#475

Post by willart4food »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

all of the above!

Re: Twitter was down

#477

Post by gsich »

iamtheworstdev wrote: post #458
bdd wrote: post #279

Earlier quoted context omitted.

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

Faangt = Facebook amazon Apple Netflix Google tesla?

Gmafia

Re: Twitter was down

#479

Post by tonystubblebine »

ziftface wrote: post #474
tonystubblebine wrote: post #233

Earlier quoted context omitted.

To take the history of direct queries at Twitter even further back, I built a web interface at Odeo for the CEO to run direct queries against the database (and save them so he could re-run them). There were some basic security precautions, but this was totally cowboy. That Odeo team was filled with best practices aficionados and the management (including me) was a bit cowardly about being clear that "WE ARE FAILING H…

Thanks for sharing. Out of curiosity, why was the answer no? Was the issue the downtime or something more subtle?

I think in the end he lost faith over retention. We got a lot of traffic and new users but didn't keep any of it. He was already suspicious that iTunes was going to kill us and so the stats were the nail in that coffin. He was right. We were ten years too early to podcasting.

Re: Twitter was down

#480

Post by _jal »

hexrcs wrote: post #362
bdd wrote: post #279

Earlier quoted context omitted.

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

Life in tech is like a Quentin Tarantino movie.

...except everyone is sitting at desks typing, there's no blood or surf rock or chases or self-indulgent soliloquies, and the cursing is much less creative?
Post reply on HN