Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

451–460 of 772 posts

Re: Inside the longest Atlassian outage

#451

I remember finding out one of the senior managers from my company ended up as head of software at Atlassian. It was at that point I was convinced Atlassian has no idea what the hell they're doing. I think this demonstrates the point nicely.

After this they might have to boomerang back to your company lol

Re: Inside the longest Atlassian outage

#452

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…

In these cases the best thing to do is just give every customer the full month refund; don't make them ask for it.

Not every business can afford to go one month without income. What's the best thing for customers? Have the business go bankrupt and irremediably lose access to the service?

Re: Inside the longest Atlassian outage

#453
post #234

Earlier quoted context omitted.

This. It would be an impossible nightmare for every account to have their own DB. Hundreds of thousands of accounts and databases....

This is the case. I won't comment on your "hundreds of thousands" figure because the number of Cloud customers was a closely guarded secret at least when I worked there, but yes one DB per tenant, dozens to hundreds of DBs per server, and some complicated shuffling of tenant DBs when you run into noisy neighbours.

If they had multi-tenant databases for SaaS it would mean either the self-hosted jira instances also had the same multi-tenant database schema or they'd have to maintain two almost entirely different data access layers for cloud vs. on-prem. Since their cloud offering came from a historically on-prem codebase, I would expect the easiest way to offer cloud stuff is to do a DB per tenant. Otherwise there would a shit-ton of new code that only applies for cloud stuff....

Re: Inside the longest Atlassian outage

#454
post #442

Earlier quoted context omitted.

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

I actually did this once with Dropbox, though it wasn't a feature they actually published. I clobbered my Dropbox directory accidentally, but I was able to find a script someone wrote to roll it back to a previous point in time and it worked quite well. After that I also took my own snapshots just in case.

Dropbox support can rollback your Dropbox account to a previous point in time too.

Re: Inside the longest Atlassian outage

#455

Earlier quoted context omitted.

Free software has zero acquisition cost, but non-zero TCO, which can measure in millions USD (recurring salary of dedicated IT team), depending on the size of organization and complexity of the setup. You will need to maintain on-premise infrastructure, automate backups and recovery, automate security, automate updates (including testing and rollbacks) etc etc, basically doing all the jobs of the people responsible f…

in thirty years of experience, I see this talking point straight from Microsoft anti-Open Source days.. > Free software has zero acquisition cost, but non-zero TCO, which can measure in millions USD Often a primary driver is exactly the opposite -- for-profit companies are accustomed to paying money for a good or service, with a billing pattern and legal obligations. The company financial deciders do not want a setup…

To be honest, I do not understand your comment.

> The company financial deciders do not want a setup that does not have a billing pattern and clear legal obligations.

I haven’t ever met a CTO or CIO, who would make budget decisions like that, neither I do it this way myself. The reality in 2022 is the same as it was in 2012 or in 2002: when you choose a solution, you consider all long term costs. In 2022 TCO for the server software includes everything that I mentioned in my comment and more. There’s a lot of use cases for OSS in corporate environment, for sure, but not every OSS solution is cheap or even affordable. Running on-premise open source collaboration tool is certainly not cheap if you do it right.

Re: Inside the longest Atlassian outage

#456
post #442
post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

I wouldn’t expect them to advertise such a thing, but the question is “can they recover from their own mistakes” not “can they recover from mine.” I don’t care if this is with an “account-level restore” or whatever; it shouldn’t be my concern.

Re: Inside the longest Atlassian outage

#457
post #442
post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

I wouldn't expect it if I just asked. I think it's reasonable as part of their disaster recovery though.

Re: Inside the longest Atlassian outage

#458
Engineering mistakes happen.

The most inexcusable thing is not communicating with the paying customers who have been affected for over a week.

Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061

Actually reading a bit more, it seems like their customer team was partying in Las Vegas instead of taking care of business: https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22

Priorities.

Re: Inside the longest Atlassian outage

#459
post #434

Earlier quoted context omitted.

It's it though? You can hop onto any of a constellation of other IM platforms, FOSS and not fairly quickly for an instant comms channel, even if you're missing the history. Having all your issue tickets missing is something you can't really deal with unless you have a very recent dump, and even then you can't just fire up Bugzilla and get something working without a lot of migration and administrative effort. You can…

We have all sorts of slack channels set up to coordinate activity, so that internal customers can talk to engineers easily, or engineers can engage with each other. If slack goes down, we'd have to work all that out. For many days, it would be a huge drag on the process, slowing down interactions. Other IM platforms wouldn't solve that just by existing. Sure, in principle one could set up such channels elsewhere, but…

Sounds like having a fallback pre-defined would be prudent if it's that important and you don't feel you could collectively extemporise something. "If Slack goes down, the plan is to use WhatsApp/Teams/Jeff's Matrix homeserver in his garage until service comes back. A list of group channels will be emailed if that happens."

Then if it does go down, you don't have to waste the first day arguing about the plan.

Post reply on HN