Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

531–540 of 544 posts

Re: Twitter was down

#531

Post by rossdavidh »

newsbinator wrote: post #424
mastratton3 wrote: post #411

Earlier quoted context omitted.

lol yes, whats the quote on "Don't assume bad intention when incompetence is to blame"? After seeing how people write code in the real world, I'm actually surprised there aren't more outages.

Hanlon's Razor: https://en.wikipedia.org/wiki/Hanlon%27s_razor "Never attribute to malice that which is adequately explained by stupidity."

Ah, but that's a lot of big corps being more stupid in the last month than last year? If it's two or three more, that's normal variation. We're now at something more like 7 or 8 more. The industry didn't get that much stupider in the last year.

Re: Twitter was down

#533

Post by foobarbecue »

iamtheworstdev wrote: post #458
bdd wrote: post #279

Earlier quoted context omitted.

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

Faangt = Facebook amazon Apple Netflix Google tesla?

Twitter, not tesla

Re: Twitter was down

#534

Post by papito »

humanfactor wrote: post #349
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

1/2) These are web apps. Big web apps but web apps none the less. We know what can go wrong theres nothing really new here. How would you quantify "too many pieces to make work". Is 1000 too many , 10000 ???? There are millions pieces of data on your harddrive and they work fine. In general the idea of variance can be solved with redundancy. Maybe there are not enough backups at twitter. 5/4) Incompetent people lead…

Scientists are squares but they also have a brain. That's why they are usually not conservatives. Conservatives are not a party, it's a herd of paranoid people who tune into Fox News every night to be told what to be afraid of next, but it's definitely not engineers or scientists.

Re: Twitter was down

#535

Post by rco8786 »

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

I worked there for a bit. Sometime around 2014 I dropped a production DB table (via a fat finger, shouldn’t have even been possible in hindsight). It wasn’t consumer facing but the internal effect made it look like all of Twitter was simultaneously down. Mass hysteria there for 20 min or so.

Re: Twitter was down

#536

Post by gcbw2 »

uxp wrote: post #469
gcbw2 wrote: post #432

Earlier quoted context omitted.

So the only take away is that now the population at large notices tech companies outages as much as they notice celebrity deaths?

"population at large" This thread is linked to a status page run by Twitter, on a programming and technology news site. I'm not really seeing how most people that exist in the western/1st world are noticing this. Is there a CNN article, or FoxNews segment on how tech companies are having outages?

Yes, fox news even suggest it was part of a large coordinated censorship effort on the POTUS :D

https://www.foxnews.com/tech/twitter-suffers-widespread-outa...

quote from that url: "The outage came as President Trump was hosting a social media summit with right-wing personalities and tech industry critics who've accused Twitter and other websites of having an anti-conservative bias."

Re: Twitter was down

#537

Post by rossdavidh »

mruts wrote: post #473
rossdavidh wrote: post #143

Earlier quoted context omitted.

You know, it would be cool if you found stats on the downtime metrics of these various high-profile recent outages, and calculated the odds of having such a cluster. Statistics is hard, though, and avoiding a "Texas Bulls-eye" would be hard.

I mean, we can assume the downtime variance follows a normal distribution. It should pretty easy to calculate P<.05 with just a little bit of data.

What you say could be true, but I don't know that we can assume it. If downtime requires several things to happen (cascading errors), but those things interact somehow (problem with one makes another more likely), I could imagine it might not be normally distributed. Disclaimer: I Am Not A Statistician.

Re: Twitter was down

#538

Post by rossdavidh »

sebastianconcpt wrote: post #376
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

I'd normally go for #4 but hypotheses #3 is starting to be a more plausible explanation for the timely "coincidence".

A friend of mine who is retired military told me there is a saying that "once is bad luck, twice is a coincidence, but three times is enemy action". Doesn't necessarily mean it's true, of course.

Re: Twitter was down

#540

Post by selestify »

mlinsey wrote: post #522
cmroanirgo wrote: post #506

Earlier quoted context omitted.

Yes, but it always seems to come down to a very small change with far reaching consequences. For this ongoing twitter outage, it's due to an "internal configuration change"... and yet the change has wide reaching consequences. It seems that something is being lost over time. In the old days of running on bare metal, yes servers failed for various reasons, then we added resiliency techniques whose sole purpose was to…

I remember all these services being far less reliable in the past. The irony of us talking about the bygone era of stability in the context of Twitter is particularly hilarious. I do think that internet services in general are much more mission critical, and the rate of improvement hasn’t necessarily kept up. It used to be not particularly newsworthy if an AWS EBS outage took out half the consumer internet several ti…

Which app was down for 7 days?
Post reply on HN