Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

241–250 of 544 posts

Re: Twitter was down

#241

Post by kreetx »

fzort wrote: post #134
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

This is hilarious, thanks for sharing. I used to work at companies like this, except they weren't worth billions of dollars.

The story is most probably not true. Love the taco tunnel though :)

Edit: apparently the stories actually are true.

Re: Twitter was down

#242

Post by lordnacho »

whatshisface wrote: post #69

I remember once we were at three outages, someone posted that they thought three was a reasonably-sized random cluster given the rate at which services go down. How many outages have we had in the last 30 days, how many do we have per month on average, and how strongly can we reject the null hypothesis? The formula for computing how unlikely this is is the Poisson distribution: `λ^k * e^-λ / k!`, where λ is the avera…

The outages might not be independent. Chances are these services are cross integrated at some level.

Re: Twitter was down

#243

Post by ibdf »

The real question is, if twitter were to go down permanently, what social media tool would the president use? Would he switch to something else or not use anything at all? I can imagine whatever tool he were to choose would become popular over night.

Re: Twitter was down

#244

Post by sandworm101 »

hn_throwaway_99 wrote: post #200
sandworm101 wrote: post #186

Earlier quoted context omitted.

>> he hit Control-C, and Twitter came back up. Monolithic architecture. When I did security work I fought this every day. Moving away from it is a nightmare of technical debt and heated debate about who should control what. I'm reminded of a story from the early days of MSN. The legend goes that in the late 90s MSN ran out of one cabinet, a single server. The server had redundant power supplies, but only one physical…

> Monolithic architecture. This particular problem had nothing to do with a monolithic architecture. Your app can be a monolith, but that still doesn't mean your BI team can't have a separate data warehouse or at least separate read replicas to run queries against.

If your website crashes because a single person ran a query, your system is too monolithic. You can have thousands of little microservices running all over the place, but a single query causing a fault proves that a vital system is running without redundancy or load sharing and that other systems cannot handle the situation. You have too many aspects of your service all tied together within a single system. It is too monolithic.

Re: Twitter was down

#245

Post by wedn3sday »

colpabar wrote: post #166
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

Can someone explain the joke (about the beer) because I genuinely don't understand edit: pretty please

If everytime the site was brought back up (because it had gone down), and the ops guys got free fancy beer, then the message pretty quickly turns into, "if the site goes down, I get rewarded."

Re: Twitter was down

#247

Post by slackfan »

treis wrote: post #236
colpabar wrote: post #166

Earlier quoted context omitted.

Can someone explain the joke (about the beer) because I genuinely don't understand edit: pretty please

If the ops team got beer every time the servers went down (as a reward for fixing them) then they'd have an incentive for the servers to go down.

What the hell are all of you smoking, some moderately expensive alcohol is nowhere near enough reward to take down a service.

Re: Twitter was down

#248

Post by julienfr112 »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

What about raising temperatures ?

Re: Twitter was down

#249

Post by idlewords »

georgehotelling wrote: post #223
jayflux wrote: post #176

Earlier quoted context omitted.

> We got so engrossed in the details of this topic that half an hour passed before we noticed that everyone had stopped working and was running around in a frenzy. Someone finally ran over and asked him if he was doing a query, he hit Control-C, and Twitter came back up. This would not be out of place as a scene in Silicon Valley

idlewords, the user you're replying to, was listed as a consultant on the show

For a later season. This was one of my favorite scenes on the show.
Post reply on HN