Live data from Hacker News

Twitter was down

status.twitterstat.us

291–300 of 544 posts

Re: Twitter was down

#291

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

Or maybe it's because the internet is tendentially becoming just a few companies' data centers? Afaik Twitter moved to GCP a few months ago. Maybe this is another Google outage?

Less likely since it looks fine from GCP status page.

Hmm, it seems that Twitter already figured it out, configuration change issues again.

Re: Twitter was down

#292
post #265

Earlier quoted context omitted.

> I would intercept these rewards and put them in my backpack for the bus ride home, in order to avoid creating perverse incentives for the operations team. But did anyone call me 'hero'? Wait, I don't understand. Why would anyone call you hero? Are you suggesting that the team would deliberately crash the app to receive beers and that by stealing them you stopped this from happening? Free drinks and free food is the…

> Are you suggesting that the team would deliberately crash the app to receive beers https://en.wikipedia.org/wiki/Perverse_incentive

Yes, the cobra effect exists. Should this mean that everyone needs to stop all forms of positive reinforcement? I don't believe so.

I doubt anyone would risk a comfortable job at Twitter against a few bottles of beers. Even if they are really fancy, that's what... $20-50?

If this had been worded as a "Haha, I stole the bad team's beer" I would have laughed.

However, worded as "where is my reward for being smart and stopping the cobra effect?" that's just an humble brag and plain unfunny.

Re: Twitter was down

#293
post #279

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

Only one thing to add:

Tech debt is accrued in amounts where every VC fund would get wet pants if tech debt was worth dollars paid out.

Re: Twitter was down

#294

Earlier quoted context omitted.

Or maybe it's because the internet is tendentially becoming just a few companies' data centers? Afaik Twitter moved to GCP a few months ago. Maybe this is another Google outage?

Less likely since it looks fine from GCP status page. Hmm, it seems that Twitter already figured it out, configuration change issues again.

Probably because we all use Kubernetes and YAML files and 100% of configuration failures are "oh shit, I used two spaces instead of 4, we're fucked".

Re: Twitter was down

#295
post #236

Earlier quoted context omitted.

If the ops team got beer every time the servers went down (as a reward for fixing them) then they'd have an incentive for the servers to go down.

What the hell are all of you smoking, some moderately expensive alcohol is nowhere near enough reward to take down a service.

Operant conditioning is a thing and it works.

While I and you would not do this I’m afraid that it would somehow find a way to work in this case too.

Re: Twitter was down

#296

Earlier quoted context omitted.

Write a script to fire random events and you will notice they sometimes cluster in ways that look like a pattern.

You know, it would be cool if you found stats on the downtime metrics of these various high-profile recent outages, and calculated the odds of having such a cluster. Statistics is hard, though, and avoiding a "Texas Bulls-eye" would be hard.

Already did that here: https://news.ycombinator.com/item?id=20356610

Re: Twitter was down

#297
No one is missing out on anything important. Mostly noise. For folks with my same mindset, I do believe I've cracked the code on most social networks, as far as what makes them worthwhile at all.

Twitter- it's the police scanners. Find them for your city, it's really the best way to know what's going on around you. Better than the papers, which can't report on everything or hide stuff for business's financial interests.

Instagram- is pretty much only useful for models, whatever sort you prefer. If you like models and it brightens your day to see a beautiful woman, as it does for me, it gives Instagram a purpose other than the noise it shares with most social networks. If it makes you happy and smile, it's a good thing. No, I'm not into pornography or anything risque. Though if I were, that would probably be ok, I simply value keeping a little imagination and mystery in my life and don't watch it. Nor are the models that I follow doing it as far as I know, but that's their decision. They mostly survive off product placement and payment for additional photos. Nothing wrong with innocent modeling, just like the olden days of pin-up girls and I hope more people support them in their endeavors.

Facebook- this one is better understood by most people, hence the popularity, but it's definitely the whitepages aspect of it. I use the instant messaging more than anything, as it's difficult to have an index of your old friend's emails until you're in touch with them again. Also, people just don't keep up on emails and maintain inbox zero very well.

Youtube- this, other than RSS feeds (through Firefox's Livemarks extension) is my main source of information. I'm not into cat videos, but I certainly love learning about astrophysics and other topics from Youtubers that are more knowledgeable than I am.

Re: Twitter was down

#298

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

Lots of people on vacation this time of year. Would be interesting to see if there is a seasonal component to the reliability of these services.

"Don't forget to occasionally do that thing I mentioned in passing 2 weeks ago, under my breath, during a klaxon alarm test. Otherwise the errors will DDoS the whole cluster. See you in a week, goodluck!"

Nah - that would never happen.

Re: Twitter was down

#299
post #104

Earlier quoted context omitted.

I found Gab [0] to be quite a viable alternative to Twitter due to its decentralised nature thanks to Mastodon so that downtimes like this cannot happen on the platform. [0] - https://gab.ai

Is this the neo-nazi Twitter alternative?

Yes

Re: Twitter was down

#300
post #211

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

I (don't) like how you exclude Russia, China, Iran and somebody from your definition of 'us'.

The assumption is that Russia, China, Iran are less dependent on Google, Twitter, etc., in part because some of them aren't allowed to operate in those countries, in part because some of them are much less dominant in those markets. 'Us' means 'people who might care that Twitter (or whoever) is down'.
Post reply on HN