All I can think, smugly, is that DNS, SMTP, HTTP, etc. don't "go down". Twitter should be a protocol, not a website.
DNS absolutely -goes down-, though in much more entertaining ways.
All the DNS in the world can't fail at once. All the Twitter can.
Twitter was down
501–510 of 544 posts
Re: Twitter was down
#502Earlier quoted context omitted.
Just a quick note not directed at OP but for any other engineers that may be unaware, these days AWS makes provisioning a read replica painless, and you can point the CEO to up-to-the-minute data while essentially firewalling the queries from customer operations.
how?
[deleted]
Re: Twitter was down
#503Earlier quoted context omitted.
The logic of the GP still applies though. Sites have outages every day so it is inevitable that some large sites will fail around the same time. Also, we know that Cloudflare and Twitter outages were attributed to configuration changes, probably others have benign explanations as well.
Sure, but "configuration changes" does not exclude several of these options. For example, is it harder to predict/deal with the consequences of configuration changes than it used to be?
Well, the options above cover pretty much every possibility, including the one I'm suggesting.
Re: Twitter was down
#504Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
There's also the HN filter bubble which could be presenting a misleading picture of "outage" frequency.
Re: Twitter was down
#505Earlier quoted context omitted.
> We notice errors more now. Mistakes are instantly news. Heck, just look at Twitter itself from its original "Fail Whale" days where there was so much downtime, to now where even this relatively small amount of downtime is the top story on HN for hours.
So, when it went down, was there a Fail Whale displayed during this most recent incident?
[deleted]
Re: Twitter was down
#506Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…
Yes, but it always seems to come down to a very small change with far reaching consequences. For this ongoing twitter outage, it's due to an "internal configuration change"... and yet the change has wide reaching consequences.
It seems that something is being lost over time. In the old days of running on bare metal, yes servers failed for various reasons, then we added resiliency techniques whose sole purpose was to alleviate downtime. Now we're at highly complex distributed systems that have failed to keep the resiliency up there.
But the fact that all the mega-corps have had these issues seems to indicate a systemic problem rather than unconnected ones.
Perhaps a connection is the management techniques or HR hiring practices? Perhaps it's due to high turnover causing the issue? (Not that I know, of course, just throwing it out there). That is, are the people well looked after and know the systems that are being maintained? Even yourself who's 'been around the traps' with high profile companies: you have moved around a lot... Were you unhappy with those companies that caused you to move on? We've seen multiple stories here on HN about how those people in the 'maintenance' role get overlooked for promotions, etc. Is this why you move around? So, perhaps the problem is systemic and it's due to management who've got the wrong set of metrics in their spreadsheets, and aren't measuring maintenance properly?
Re: Twitter was down
#507Earlier quoted context omitted.
#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…
FAANG(+T)(-N)(+M)
I think we 'bumped heads' at Middlebury in '94, and I think you are in store for an "ideological reckoning" w/in 3 years.
Pinboard is a great product, so thanks for that. I am surpised you don't have your own Mastodon instance (or do you?).
Re: Twitter was down
#508Earlier quoted context omitted.
Rule one of having interns and retaining your sanity is that interns get their own branch to muck around in.
Rule one of having a useful intern experience is to get them writing production code as quickly as possible. They check in their first change? Get that thing into production immediately. (If it's going to destabilize the system, why did you approve the CL? You two probably pair programmed the whole thing together.)
I completely agree- even if it's something small.
I'm an intern in a big company with an internal robotics and automation group, and I recently got to wire up a pretty basic control panel, install it, and watch workers use it. That was so cool, and made me appreciate what I was doing a lot more.
Re: Twitter was down
#509Earlier quoted context omitted.
Why not have it run against a replicated copy? I did that in the past, works amazingly, they can f * up all they want without any implications.
It's probably easy to do if you know it's an issue to begin with. I've run into this scenario before (running sql queries to read data that turned out to lock everything) and it caught me by surprise. Why would a read query cause the database to lock anything? I thought databases did stuff like multiversion concurrency control to make locks like that unnecessary.
Doing large queries on a Postgres standby had the potential to mess up the master, depending on configuration settings.
Re: Twitter was down
#510Earlier quoted context omitted.
I was just polishing my bit. Not in a bad mood today so much as a bored mood. You seem like you know what you are talking about (yes, I was bored enough to stalk you, too)
If you are bored one day and around Menlo Park, come have a coffee or ice cream at FB campus. You can troll me in person.
Isn't it interesting where this is going? We all want to meet our accusers? I don't care for FB myself, but I appreciate what you all are doing in the larger sense. Cloudflare is my fave of your former employers (since you shared that in this discussion).