I remember finding out one of the senior managers from my company ended up as head of software at Atlassian. It was at that point I was convinced Atlassian has no idea what the hell they're doing. I think this demonstrates the point nicely.
Inside the longest Atlassian outage
451–460 of 772 posts
Re: Inside the longest Atlassian outage
#452Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…
In these cases the best thing to do is just give every customer the full month refund; don't make them ask for it.
Re: Inside the longest Atlassian outage
#453Earlier quoted context omitted.
This. It would be an impossible nightmare for every account to have their own DB. Hundreds of thousands of accounts and databases....
This is the case. I won't comment on your "hundreds of thousands" figure because the number of Cloud customers was a closely guarded secret at least when I worked there, but yes one DB per tenant, dozens to hundreds of DBs per server, and some complicated shuffling of tenant DBs when you run into noisy neighbours.
Re: Inside the longest Atlassian outage
#454Earlier quoted context omitted.
Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.
I actually did this once with Dropbox, though it wasn't a feature they actually published. I clobbered my Dropbox directory accidentally, but I was able to find a script someone wrote to roll it back to a previous point in time and it worked quite well. After that I also took my own snapshots just in case.
Re: Inside the longest Atlassian outage
#455Earlier quoted context omitted.
Free software has zero acquisition cost, but non-zero TCO, which can measure in millions USD (recurring salary of dedicated IT team), depending on the size of organization and complexity of the setup. You will need to maintain on-premise infrastructure, automate backups and recovery, automate security, automate updates (including testing and rollbacks) etc etc, basically doing all the jobs of the people responsible f…
in thirty years of experience, I see this talking point straight from Microsoft anti-Open Source days.. > Free software has zero acquisition cost, but non-zero TCO, which can measure in millions USD Often a primary driver is exactly the opposite -- for-profit companies are accustomed to paying money for a good or service, with a billing pattern and legal obligations. The company financial deciders do not want a setup…
> The company financial deciders do not want a setup that does not have a billing pattern and clear legal obligations.
I haven’t ever met a CTO or CIO, who would make budget decisions like that, neither I do it this way myself. The reality in 2022 is the same as it was in 2012 or in 2002: when you choose a solution, you consider all long term costs. In 2022 TCO for the server software includes everything that I mentioned in my comment and more. There’s a lot of use cases for OSS in corporate environment, for sure, but not every OSS solution is cheap or even affordable. Running on-premise open source collaboration tool is certainly not cheap if you do it right.
Re: Inside the longest Atlassian outage
#456This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…
Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.
Re: Inside the longest Atlassian outage
#457This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…
Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.
Re: Inside the longest Atlassian outage
#458The most inexcusable thing is not communicating with the paying customers who have been affected for over a week.
Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061
Actually reading a bit more, it seems like their customer team was partying in Las Vegas instead of taking care of business: https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22
Priorities.
Re: Inside the longest Atlassian outage
#459Earlier quoted context omitted.
It's it though? You can hop onto any of a constellation of other IM platforms, FOSS and not fairly quickly for an instant comms channel, even if you're missing the history. Having all your issue tickets missing is something you can't really deal with unless you have a very recent dump, and even then you can't just fire up Bugzilla and get something working without a lot of migration and administrative effort. You can…
We have all sorts of slack channels set up to coordinate activity, so that internal customers can talk to engineers easily, or engineers can engage with each other. If slack goes down, we'd have to work all that out. For many days, it would be a huge drag on the process, slowing down interactions. Other IM platforms wouldn't solve that just by existing. Sure, in principle one could set up such channels elsewhere, but…
Then if it does go down, you don't have to waste the first day arguing about the plan.