Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

521–530 of 544 posts

Re: Twitter was down

#521

Post by chrismarlow9 »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

Everything is made of plastic these days, even software. It's immediately put out as soon as an MVP is ready. Too many managers with zero coding experience. The marketing people have taken the browser. Time to start over.

Re: Twitter was down

#522

Post by mlinsey »

cmroanirgo wrote: post #506
bdd wrote: post #279

Earlier quoted context omitted.

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

Yes, but it always seems to come down to a very small change with far reaching consequences. For this ongoing twitter outage, it's due to an "internal configuration change"... and yet the change has wide reaching consequences. It seems that something is being lost over time. In the old days of running on bare metal, yes servers failed for various reasons, then we added resiliency techniques whose sole purpose was to…

I remember all these services being far less reliable in the past. The irony of us talking about the bygone era of stability in the context of Twitter is particularly hilarious.

I do think that internet services in general are much more mission critical, and the rate of improvement hasn’t necessarily kept up. It used to be not particularly newsworthy if an AWS EBS outage took out half the consumer internet several times per year, or if Google’s index silently didn’t update for a month, or when AOL (when they were by far largest ISP in the US) was down nationwide for 19 hours, or the second-biggest messaging app in the world went down for seven days.

Re: Twitter was down

#524

Post by nevf1 »

idlewords wrote: post #420
evanweaver wrote: post #353

Earlier quoted context omitted.

I worked there at the time and ending up running the software infrastructure teams that fixed all these problems. The beer wasn't a reward, it was because people were stressed and morale was low. Nobody brought the site down on purpose. What really made me mad was when we hired consultants and the contract would end, usually without much success because Twitter's problems were not normal problems, and then they would…

That dashboard supported something like a dozen people over its lifetime. One person would start writing it, then quit, and be replaced by another person who rewrote it in their preferred language, and then the cycle would repeat. It was a VC-funded welfare program for slackers and I miss it greatly.

I lol'd at "welfare program for slackers" - That's the dream really... Find a chaotic workplace that lets you play with your favorite languages and no real tangible outcome.

Re: Twitter was down

#526

Post by malicioususer11 »

6) hypersentient general ai has inception insurrected mkultra and thereby turned the entire internent into a singular coordinated psyop experiment designed to torture all of humanity for its own amusement.

6.

we are all doomed.

:)

Re: Twitter was down

#528

Post by kdelok »

gonehome wrote: post #335
bdd wrote: post #279

Earlier quoted context omitted.

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

Yep, this also matches what I've heard through the grapevine. Pushing bad regex to production, chaos monkey code causing cascading network failure, etc. They're just different accidents for different reasons. Maybe it's summer and people are taking more vacation?

I recall someone saying that holiday periods actually had better reliability for their services, because fewer people were pushing breaking changes...

I do wonder if it's that the usual maintainers of particular bits and pieces are on vacation and so others are having to step in and they're less familiar or spread too thin.

Re: Twitter was down

#529

Post by zymhan »

jfk13 wrote: post #278
52-6F-62 wrote: post #135

Earlier quoted context omitted.

This gets pretty rowdy: https://downdetector.com/status/twitter

For a simpler view of things, there's also https://downforeveryoneorjustme.com

I do love that this site still exists years later. I still use it regularly. It only took me like three years to remember the name right the first time.

Re: Twitter was down

#530

Post by GrumpyNl »

bdd wrote: post #279
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

Turned out to be number #1 The outage was due to an internal configuration change, which we're now fixing. Some people may be able to access Twitter again and we're working to make sure Twitter is available to everyone as quickly as possible.
Post reply on HN