Twitter was down • Hacker News Board

Re: Twitter was down

#411

Post by mastratton3 » Thu, Jul 11, 2019, 9:58 PM UTC

bdd wrote: post #279

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

lol yes, whats the quote on "Don't assume bad intention when incompetence is to blame"?

After seeing how people write code in the real world, I'm actually surprised there aren't more outages.

Re: Twitter was down

#412

Post by thfuran » Thu, Jul 11, 2019, 10:03 PM UTC

t0astbread wrote: post #386

ionforce wrote: post #315

Earlier quoted context omitted.

> tendentially Is this a word? You don't mean tangentially? I'm having a crisis right now.

Is it not? Sorry if I got it wrong, English isn't my first language. dict.cc (my source of truth for English vocab) says it's a word: https://www.dict.cc/?s=tendentially

It's apparently a word but I'd say it's quite uncommon. I played around with google ngram viewer and had a hard time coming up with a word that is less common. But I finally came up with "astrophotographic".

E: "unsurpassingly" is way down there too

Re: Twitter was down

#413

Post by charliebrownau » Thu, Jul 11, 2019, 10:04 PM UTC

Gab & parler are online

Re: Twitter was down

#414

Post by SllX » Thu, Jul 11, 2019, 10:08 PM UTC

papito wrote: post #273

0xFFFE wrote: post #85

Earlier quoted context omitted.

I share your sentiment to a certain extent, but I believe Twitter is a necessary evil. There should be an alternative to main stream media for people.

I follow mostly "main stream media" on Twitter. It should be an aggregator of professional news, not "alternative" facts.

Twitter is where the professionalism in the mainstream media goes to die. Between headlines you’re likely either getting insider baseball, a circlejerk or a crowbar in the face with “mainstream” written on the side in sharpie. If all you have are the headlines, then you’ve chosen Twitter to be your alternative “RSS” client. A valid choice for most and not one I would criticize either, but not entirely what it is or “should” be.

Re: Twitter was down

#415

Post by tschellenbach » Thu, Jul 11, 2019, 10:09 PM UTC

brokensegue wrote: post #360

tschellenbach wrote: post #14

Really curious which part of their infrastructure was the root cause.

eh, it's a boring story source: work there

someone tripped over a cable?

Re: Twitter was down

#416

Post by twic » Thu, Jul 11, 2019, 10:11 PM UTC

thomasjudge wrote: post #199

qrush wrote: post #88

Earlier quoted context omitted.

Me too! :)

What is the new fail mascot called? It looks like a cartoonish-alien with a PacMan/snipper hand and another hand that looks like a burning fuse standing next to a bomb with a fuse lit that is split open so it also looks like a PacMan

It's a robot whose hand has fallen off.

Re: Twitter was down

#417

Post by danellis » Thu, Jul 11, 2019, 10:11 PM UTC

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

#4.

When things are random, they cluster.

Re: Twitter was down

#418

Post by imagetic » Thu, Jul 11, 2019, 10:13 PM UTC

Who cares?

Re: Twitter was down

#419

Post by MegaButts » Thu, Jul 11, 2019, 10:15 PM UTC

GuiA wrote: post #344

captn3m0 wrote: post #328

Earlier quoted context omitted.

Can you clarify what redefining problems would mean (with an eg)?

Think of computer vision tasks. Until modern deep learning approaches came around, it was built on brittle, explicitly defined pipelines that could break entirely if something minor about the input data changed. Then the great deep learning wave of 201X happened, replacing dozens/hundreds of carefully defined steps with a more flexible, generalizable approach. The new approach still has limitations and failure cases,…

That's not redefining the problem, so much as applying a new technology to solve the same problem. Usually using the flashy new technology decreases reliability due to immature tooling, lack of testing, and just general lack of knowledge of the new approach.

Also deep learning, while incredibly powerful and useful, is not the magic cure-all to all of computer vision's problems and I have personally seen upper management's misguided belief in this ruin a company (by which I mean they can no longer retain senior staff, they have never once hit a deadline, every single one of their metrics is not where they want it to be, and a bunch of other stuff I can't say without breaking anonymity).

Re: Twitter was down

#420

Post by idlewords » Thu, Jul 11, 2019, 10:16 PM UTC

evanweaver wrote: post #353

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

I worked there at the time and ending up running the software infrastructure teams that fixed all these problems. The beer wasn't a reward, it was because people were stressed and morale was low. Nobody brought the site down on purpose. What really made me mad was when we hired consultants and the contract would end, usually without much success because Twitter's problems were not normal problems, and then they would…

That dashboard supported something like a dozen people over its lifetime. One person would start writing it, then quit, and be replaced by another person who rewrote it in their preferred language, and then the cycle would repeat.

It was a VC-funded welfare program for slackers and I miss it greatly.