I am very surprised that there aren't more people/companies announcing "offsite backup capabilities" for JIRA or Confluence, etc.. I did just search for scripts that can do this, I'd probably pay a few bucks for something like that at this point.
Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
121–130 of 260 posts
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#122As the person responsible for running Jira and Confluence on premises at my employer I‘m looking forward for the next time one of their sales droids contacts me to make us move to their cloud services (despite me stating that we are not interested multiple times)…
It's insane. Since they've all but killed off on-premise Jira and Confluence, they've been spamming me regularly trying to convince me to "upgrade" to the cloud. Eventually I gave in and replied to them asking two simple questions. One is that we've had many bad reports from partners that Jira Cloud is incredibly slow, even when compared to the already underperforming Jira Server and I wonder what their performance g…
My employer recently switched from on-prem to cloud. The cloud service is insanely slow, or maybe it's my aging Macbook, but every single component on the page seems to have to load separately. (It's a newer UI versus what we had on-prem).
Thankfully we haven't been impacted by this outage.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#123Earlier quoted context omitted.
Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…
How does that work around GDPR and other "Right to be forgotten" legislation? Aren't we required to hard-delete this kind of data?
As long as the data is deleted within a month there should not be any GDPR concerns.
> The data subject shall have the right to obtain from the controller the erasure of personal data concerning him or her without undue delay
> “Undue delay” is considered to be about a month
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#124As someone who is impacted, this is obviously immensely frustrating. Worse, outside of "we have rebuilt functionality for over 35% of the users", I haven't seen any reports from the people who have ostensibly been recovered. Next, their published RTO is 6 hours, so obviously they must have done something that completely demolished their ability to use their standard recovery methods: https://www.atlassian.com/trust/s…
RTO is so hard to state properly even with regular testing. If someone blows away a critical database, sure you can meet your published RTO. What if we lose 300 of our databases and need to copy snapshots from another region. AWS limits you to 20 concurrent snapshot copies cross region. Which of those databases should you do first? Do you know your entire dependency graph for all 1000 of your services to make the right call? Meanwhile tick-tock your "6 hours" is slipping away. And what if someone nukes our entire AWS account with all of our prod resources? Databases, load balancers, S3 (no such thing as snapshotting there), EC2 instances, etc, etc
Those last two are examples are very unlikely but no company is going to say RTO = "probably 6 hours but it could be three weeks if we get ransomwared"
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#125Earlier quoted context omitted.
Bigger lesson is: do not utilize cloud solutions. Your organization's competency floor is set to the least competent employee of the cloud provider.
I'd disagree with this lesson. Saying "do not utilize cloud solutions" period is nuts. Google and Microsoft are way better at email hosting and delivery than your on-prem server is unless you spend a ton more money on hardware and engineers to keep it up, which is simply not worth it for many companies. Dropbox is going to have better uptime and lower TCO than your self-hosted owncloud instance. What I will say is it…
There's something humorous when better uptime is cited as a reason to use cloud software instead of on-prem in a thread about a major cloud outage.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#126Earlier quoted context omitted.
I've actually done exactly that many years ago for a self-hosted Jira installation that didn't have any backups. You can bet we had backups with regular testing after that.
I would have just quit if I had that landed on me.
Honestly it doesn't sound too difficult and like a challenge to script something fun. To me. If you ever find yourself in that situation, email is in my profile and we'll work something out :)
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#127Reminds me of the time a group I worked for at a National Lab decided to call the root folder for their project “core”. I bet you can’t guess what filename the backup scripts were configured to ignore…
This is an expensive lesson I think everyone gets to learn at some point. There's no such thing as a file worth excluding from a backup, it always fucks you, some (like me) more than others. Have to buy twice as much disk? Fine, at least you know you actually have a backup
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#128Earlier quoted context omitted.
It's insane. Since they've all but killed off on-premise Jira and Confluence, they've been spamming me regularly trying to convince me to "upgrade" to the cloud. Eventually I gave in and replied to them asking two simple questions. One is that we've had many bad reports from partners that Jira Cloud is incredibly slow, even when compared to the already underperforming Jira Server and I wonder what their performance g…
My employer recently switched from on-prem to cloud. The cloud service is insanely slow, or maybe it's my aging Macbook, but every single component on the page seems to have to load separately. (It's a newer UI versus what we had on-prem). Thankfully we haven't been impacted by this outage.
I've also heard that the cloud service is slow; you can easily check if it's your machine or the server by watching devtools -> network tab and seeing how many requests are `(waiting)`, because chances are it's Atlassian's server speed.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#129Earlier quoted context omitted.
They are discontinuing on prem after 2022 though. What is your plan if not to migrate to their SaaS offering?
We will probably bite the sour apple (not sure if this is correct English but you’ll get the meaning even if it’s not) and switch to the data center edition which is still on prem but costs approximately twice as much.
You can probably use this 1 week (3 scheduled) outage to ask for a discount, "your cloud offering is a bucket of shit, your data center edition is too expensive, my higher-ups told me to find something else...".
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#130I wish companies would stop with this "small number of customers" messaging. It always seems disingenuous and, besides, that matters for your internal estimation of business impact but means absolutely nothing to the customers affected.
I think they add that because otherwise you read about the problem, panic, and then spend hours digging through your own data to make sure it's all there. Unaffected customers like being told they're not affected.