Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

191–200 of 544 posts

Re: Twitter was down

#191

Post by AnIdiotOnTheNet »

the_af wrote: post #61
lukey_q wrote: post #25

A lot of high-profile outages recently. Can't actually remember the last time Twitter went fully down. Have to confess I immediately assumed an issue with my own connection, even though every other site is working. Unrelated, but for some reason the phrase "I have no mouth and I must scream" just popped into my head

> Unrelated, but for some reason the phrase "I have no mouth and I must scream" just popped into my head That phrase was coined by Harlan Ellison in his classic scifi short story to represent a situation of complete despair and powerlessness. I don't think a lack of Twitter, Whatsapp, Instagram, Facebook or Gmail -- however inconvenient -- would fill me with that kind of existential dread :)

Not those things specifically, but take a step back and look at the Rube Goldberg inspired complexity of modern technology and despair.

Re: Twitter was down

#192

Post by isostatic »

fzort wrote: post #134
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

This is hilarious, thanks for sharing. I used to work at companies like this, except they weren't worth billions of dollars.

Neither was twitter in 2008, it didn't reach $1b until the end of 2009

Re: Twitter was down

#193

Post by ryandrake »

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

> The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer

Never understood this mentality but have seen it at many companies. Rewarding someone or some team for heroically fixing something after a catastrophic failure. Talk about misaligned incentives! Reminds me of the Cobra Effect [1]. When you reward “fixing a bad thing” you will get more of the bad thing to be fixed.

1: https://en.wikipedia.org/wiki/Cobra_effect

Re: Twitter was down

#194

Post by idlewords »

cambalache wrote: post #140
lukey_q wrote: post #25

A lot of high-profile outages recently. Can't actually remember the last time Twitter went fully down. Have to confess I immediately assumed an issue with my own connection, even though every other site is working. Unrelated, but for some reason the phrase "I have no mouth and I must scream" just popped into my head

It makes me wonder if a powerful malicious agent can devise a complex operation (planting people in several key places) and wipe out most of the databases of one these places. It would be interesting to see what would it happen if suddenly all of twitter of facebook is deleted , puff!, gone in a second.

Actually deleting stuff at these scales takes many, many days if you don't want it to be trivially recoverable.

Re: Twitter was down

#197

Post by sizzle »

idlewords wrote: post #114
dubcanada wrote: post #106

Earlier quoted context omitted.

"I would intercept these rewards and put them in my backpack for the bus ride home, in order to avoid creating perverse incentives for the operations team. But did anyone call me 'hero'?" Wait so you stole rewards for a team that was spending time (I assume extra or stressful) on something you didn't do or have any part in. And you want a cookie? I mean I get it, the company was probably not great in it's infancy. Bu…

I don't want a cookie; I want more $24/bottle Belgian beer.

Just wanted to say that I enjoy reading your blog.

Re: Twitter was down

#198

Post by godarni »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

Lots of people on vacation this time of year. Would be interesting to see if there is a seasonal component to the reliability of these services.

Re: Twitter was down

#199

Post by thomasjudge »

qrush wrote: post #88
neom wrote: post #9

I miss fail whale. :( https://www.theatlantic.com/technology/archive/2015/01/the-s...

Me too! :)

What is the new fail mascot called? It looks like a cartoonish-alien with a PacMan/snipper hand and another hand that looks like a burning fuse standing next to a bomb with a fuse lit that is split open so it also looks like a PacMan

Re: Twitter was down

#200

Post by hn_throwaway_99 »

sandworm101 wrote: post #186
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

>> he hit Control-C, and Twitter came back up. Monolithic architecture. When I did security work I fought this every day. Moving away from it is a nightmare of technical debt and heated debate about who should control what. I'm reminded of a story from the early days of MSN. The legend goes that in the late 90s MSN ran out of one cabinet, a single server. The server had redundant power supplies, but only one physical…

> Monolithic architecture.

This particular problem had nothing to do with a monolithic architecture. Your app can be a monolith, but that still doesn't mean your BI team can't have a separate data warehouse or at least separate read replicas to run queries against.

Post reply on HN