Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

Post by Zircom » Mon, Apr 11, 2022, 11:51 PM UTC

stonogo wrote: post #150

RL_Quine wrote: post #118

Earlier quoted context omitted.

Tapes are not that slow, 360MB/s per drive, and on a large scale swapping them is completely automatic.

In support of your point, 360MB/s is an extremely conservative estimate. I'd expect that from LTO-6, which is around ten years old, and I would certainly hope their backups are on more modern gear than that.

Post by bigiain » Tue, Apr 12, 2022, 12:05 AM UTC

mdeeks wrote: post #124

Rantenki wrote: post #46

As someone who is impacted, this is obviously immensely frustrating. Worse, outside of "we have rebuilt functionality for over 35% of the users", I haven't seen any reports from the people who have ostensibly been recovered. Next, their published RTO is 6 hours, so obviously they must have done something that completely demolished their ability to use their standard recovery methods: https://www.atlassian.com/trust/s…

RTO is so hard to state properly even with regular testing. If someone blows away a critical database, sure you can meet your published RTO. What if we lose 300 of our databases and need to copy snapshots from another region. AWS limits you to 20 concurrent snapshot copies cross region. Which of those databases should you do first? Do you know your entire dependency graph for all 1000 of your services to make the rig…

Post by bigiain » Tue, Apr 12, 2022, 12:12 AM UTC

manquer wrote: post #149

mdeeks wrote: post #124

Earlier quoted context omitted.

RTO is so hard to state properly even with regular testing. If someone blows away a critical database, sure you can meet your published RTO. What if we lose 300 of our databases and need to copy snapshots from another region. AWS limits you to 20 concurrent snapshot copies cross region. Which of those databases should you do first? Do you know your entire dependency graph for all 1000 of your services to make the rig…

For what you suggest some combination of these things should have happened. - Some employee has root access to AWS account and uses it operationally - Given wildcard S3 permissions to an IAM user and allowing delete bucket - Not enabled object versioning - Cross Region replication not enabled - no large bucket protection - don't have basic security monitoring and setup of Cloudtail alerts - have not invested in full…

Post by nvr219 » Tue, Apr 12, 2022, 12:21 AM UTC

WorldMaker wrote: post #170

nvr219 wrote: post #87

Earlier quoted context omitted.

Last time I checked (a few months ago), Planner still could not be backed up. Like, at all. If someone went in and deleted a whole bucket you can't recover it, not natively, and not with third party. So that's a big fat no from me.

I seem to recall dumping Planner to JSON easily enough. I don't know if you can easily restore directly from its JSON, but the JSON was nice enough to work on for what little I needed to do with it. My current employer shut off Trello and forced us over to Jira and is threatening to disable Planner, so I'm "not allowed" to rely on Planner enough day-to-day so it's possible it is either better or worse than I remember…

Post by duxup » Tue, Apr 12, 2022, 12:24 AM UTC

xenadu02 wrote: post #29

btgeekboy wrote: post #10

Earlier quoted context omitted.

Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…