Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

561–570 of 772 posts

Re: Inside the longest Atlassian outage

#561

Post by calsy »

buscoquadnary wrote: post #489
systemvoltage wrote: post #486

Earlier quoted context omitted.

This is an engineering problem. They should own it and improve things, make sure it doesn't happen again. Also, GP's quote > Engineering mistakes happen. I don't like this statement because it offers consolation at the expense of unintentional normalization.

And coders that say all code has bugs are just defeatists that are trying to make excuses for being lazy. Sometimes manure will always hit the fan. Being robust means being able to handle that.

It's the exact opposite, any coder who blindly believes that a piece of software is flawless is kidding themselves.

It's delusional to think software can be flawless in the real world when it's used by an untold amount of people, on all manner of devices, possibly running different OS's with different versions on networks that can be configured all sorts of ways. Thats not to mention all the dependencies involved in creating high level software, from the third party libraries to external services like cloud storage.

You anticipate there will be problems and make sure there are processes in place to manage them when they inevitably occur. Thats the exact opposite of laziness.

Re: Inside the longest Atlassian outage

#562

Post by gjvc »

benreesman wrote: post #496
benreesman wrote: post #492

Earlier quoted context omitted.

Jesus, if there was ever an example of the internet making the world smaller. When do execs living it up at the fucking Wynn Encore while the house burns down start to not get another job? They’ll keep pulling this shit until it cost money.

For clarity: I went through a period where some combination of self-indulgence and legitimate life crisis caused me to take my eye off the ball when it mattered. I’m still trying to kickstart a second act years later, because I’m trailer trash and it’s hard work when you’re that.

Everyone deserves a second chance. You'll be on top again before you know it, and when you look back it'll have been easier than it seems now. Fair wind and God speed!

Re: Inside the longest Atlassian outage

#563

Post by mrlinx »

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

She replied to that in the comments of that post:

> Yes Andy it was and I didn't realize that my scheduled posts were still going out. They have now stopped.

Re: Inside the longest Atlassian outage

#564

Post by jamesfinlayson »

nicoburns wrote: post #468
yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

We switched from JIRA to Shortcut https://shortcut.com/ (formerly Clubhouse), and I'd highly recommend them. It's much better than JIRA ever was, both from a UX perspective and an implementation/performance perspective.

I used to use Clubhouse and moving a ticket between columns was so snappy. With Jira it is a long wait.

Re: Inside the longest Atlassian outage

#565

Post by systemvoltage »

syshum wrote: post #521
systemvoltage wrote: post #486

Earlier quoted context omitted.

This is an engineering problem. They should own it and improve things, make sure it doesn't happen again. Also, GP's quote > Engineering mistakes happen. I don't like this statement because it offers consolation at the expense of unintentional normalization.

The deletion of customer data was engineering mistake, that is not what I was talking about The Negative fall out was not due to the deletion of customer data, as the Story and multiple customers have stated the negative fall out was the SILENCE / lack of communications, which is Sales / Customer Service not engineering As the comment I was replying to noted while engineering was trying to recover from what might pos…

Ok I agree with these failures, but don't you think that its a PR people problem? Perhaps executives and upper management? Sales people are just doing what they are supposed to do. Sell Atlassian products.

Re: Inside the longest Atlassian outage

#566

Post by wly_cdgr »

opensrcken wrote: post #540
kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

Unless you know, with certainty, who is responsible for customer-facing communications , how can you publicly call on a specific person to be fired? Even if you did know, and are correct in your implication, how do you know that she didn't want to do the right thing but was overruled by people above her in the chain? Overall, this is a pretty irresponsible comment, and reflects the kind of mob mentality that I would…

They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with

Re: Inside the longest Atlassian outage

#567

Post by mdoms »

mh- wrote: post #261
mdoms wrote: post #234

Earlier quoted context omitted.

This is the case. I won't comment on your "hundreds of thousands" figure because the number of Cloud customers was a closely guarded secret at least when I worked there, but yes one DB per tenant, dozens to hundreds of DBs per server, and some complicated shuffling of tenant DBs when you run into noisy neighbours.

That makes this prolonged restore process all the more confusing, then. I (and many others) assumed they had to graft in data from backups since a full restore would clobber newer changes from unaffected customers. If they're all isolated in their own logical per-tenant DBs, I'm really at a loss for what is making restoration take 3 weeks for 400 tenants. I understand if you'd rather not venture into it, but care to…

To be honest I'm at a bit of a loss too. My speculation is that since they went all-in on microservices and utilizing various AWS services (something that was underway when I worked there) their data stores have become very much more disparate.

For example, they have the main PostgreSQL data store. Surely that's easy to restore. But the users in that DB have a "foreign key" (in a logical sense, not physical) to the Identity service. This is a real life example that occurred while I was there. So now we have a mixture of multi and single tenancy. So perhaps the identity records are also tied to this app ID and deletes were propagated to that service. And perhaps there is an SQS queue and a serverless function to handle, say, outgoing mail from Jira. Where does this data go? I dunno maybe some Go-powered microservice with its own DocumentDB store. Do deletes propagate here too? Who knows. You can see how this gets complicated and how issues multiply with more services.

Again, this is only speculation. But "decomposing the monolith" was a big deal and it was coming from the top.

Re: Inside the longest Atlassian outage

#568

Post by mdoms »

miketria wrote: post #396
bluedino wrote: post #145

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…

Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.

400 tenants doesn't seem like that much scale though...? What will happen if there's an incident affecting more than 0.18% of tenants?

Re: Inside the longest Atlassian outage

#569

Post by astura »

yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

I haven't seen Trac mentioned yet.

Re: Inside the longest Atlassian outage

#570

Post by sverhagen »

iancarroll wrote: post #552
theteapot wrote: post #541

Earlier quoted context omitted.

Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.

You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.

Why not restore the tenants to a different environment that is not otherwise active? At Atlassian's scale you would expect them 1) not to be running all the things on one server anyway, 2) have some existing ability to move tenants between environments for legal or performance reasons, 3) have the ability to backup/restore single tenants, and so on. I am not arguing that the predicament they are in now is real, for them, now. But I have worked at much smaller outlets where this worked fine, not because they were smaller, but because they had that particular shit in order, which is entirely a matter of priorities.

Nothing, and I mean absolutely nothing, that Atlassian has to offer is rocket surgery-kind of hard... yet, here we are... not being particularly surprised at all.

Post reply on HN