Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

311–320 of 544 posts

Re: Twitter was down

#311

Post by kgraves »

azhenley wrote: post #169
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

This gives me hope that one day I will be able to run a startup. The big tech companies aren't too different than the rest of us after all...

Agreed, the only thing that a showstopper for me is the money and talent, It is still a struggle to find talented people who want to work for a startup.

Re: Twitter was down

#312

Post by kwizzt »

bdd wrote: post #279
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

How does the fact you worked at those companies relate to #4?

Edit: I misread the parent and my question doesn't make a lot of sense. Please ignore it :)

Re: Twitter was down

#313

Post by tonystubblebine »

NKCSS wrote: post #280
tonystubblebine wrote: post #233

Earlier quoted context omitted.

To take the history of direct queries at Twitter even further back, I built a web interface at Odeo for the CEO to run direct queries against the database (and save them so he could re-run them). There were some basic security precautions, but this was totally cowboy. That Odeo team was filled with best practices aficionados and the management (including me) was a bit cowardly about being clear that "WE ARE FAILING H…

Why not have it run against a replicated copy? I did that in the past, works amazingly, they can f * up all they want without any implications.

This was 2005. We had dedicated servers in our own cage. I can't remember if we already had replicas. It seems plausible. But actually spinning up a new one would have required more work and convincing than I wanted to do.

Re: Twitter was down

#315

Post by ionforce »

t0astbread wrote: post #254
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

Or maybe it's because the internet is tendentially becoming just a few companies' data centers? Afaik Twitter moved to GCP a few months ago. Maybe this is another Google outage?

> tendentially

Is this a word? You don't mean tangentially? I'm having a crisis right now.

Re: Twitter was down

#317

Post by kgraves »

akhilcacharya wrote: post #274
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

The other possibility is intern season (I'm 99.99% joking)

As much as I don't like interns, I am sure that they wouldn't even touch a system of the scale like Twitter's in my opinion. /s

Re: Twitter was down

#320

Post by dqpb »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

3) Come on man, you can't just go around opening parentheses and then not closing them.
Post reply on HN