Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

211–220 of 544 posts

Re: Twitter was down

#211

Post by listic »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

I (don't) like how you exclude Russia, China, Iran and somebody from your definition of 'us'.

Re: Twitter was down

#212

Post by patmcguire »

eljimmy wrote: post #93
lukey_q wrote: post #25

A lot of high-profile outages recently. Can't actually remember the last time Twitter went fully down. Have to confess I immediately assumed an issue with my own connection, even though every other site is working. Unrelated, but for some reason the phrase "I have no mouth and I must scream" just popped into my head

Yesterday, Stripe went down for a half hour, and the later yesterday, Google's Android Payment Validation API went down for more than 2 hours. Stripe at least acknowledged their downtime. Google was oblivious, made no update to any of their status pages. Really horrendous awareness and support from Google per usual.

Odd coincidence, seems like it might have been some upstream banks having a bad SCA rollout

Re: Twitter was down

#213

Post by mrep »

jayflux wrote: post #176
idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

> We got so engrossed in the details of this topic that half an hour passed before we noticed that everyone had stopped working and was running around in a frenzy. Someone finally ran over and asked him if he was doing a query, he hit Control-C, and Twitter came back up. This would not be out of place as a scene in Silicon Valley

Not the best quality, but there is a scene just like that: https://www.youtube.com/watch?v=Dz7Niw29WlY

Re: Twitter was down

#214

Post by [deleted] »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

[deleted]

Re: Twitter was down

#215

Post by [deleted] »

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

[deleted]

Re: Twitter was down

#216

Post by busterarm »

idlewords wrote: post #82

So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…

This is the same group of folks who wrote the infamous ranty blog shitting all over Rails back in...'11(?) when it was pretty clear that their workload wasn't suited to an RBDMS and ActiveRecord. They wrote their own message queue twice despite suggestions to use known tools before eventually giving up.

Re: Twitter was down

#217

Post by sp332 »

cagrimmett wrote: post #102

I hope Jack went rogue and nuked it after his most recent hot yoga vision. That's the kind of shakeup we need in this world.

They have been teasing/warning about a new version of Twitter that was due to launch soon. I wonder if this is it.

Re: Twitter was down

#218

Post by sho »

hn_throwaway_99 wrote: post #200
sandworm101 wrote: post #186

Earlier quoted context omitted.

>> he hit Control-C, and Twitter came back up. Monolithic architecture. When I did security work I fought this every day. Moving away from it is a nightmare of technical debt and heated debate about who should control what. I'm reminded of a story from the early days of MSN. The legend goes that in the late 90s MSN ran out of one cabinet, a single server. The server had redundant power supplies, but only one physical…

> Monolithic architecture. This particular problem had nothing to do with a monolithic architecture. Your app can be a monolith, but that still doesn't mean your BI team can't have a separate data warehouse or at least separate read replicas to run queries against.

And of course this is standard practise. I've contracted on largish apps before (rails! Shock!) and of course we provided read-only replicas for BI query purposes. I wouldn't have provided production access even if asked.

Anything else is simple incompetence and the macro-organisation of the code and/or services is irrelevant.

Re: Twitter was down

#219

Post by jonprobably »

rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

It's summer time and everyone who knows how stuff works is halfway through a drink right now. Probably with their families. Is it a trend year over year for 7/4 +/- a week?
Post reply on HN