Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-incident review on the Atlassian April 2022 outage

atlassian.com

Post reply on HN
Print view View original

61–70 of 157 posts

Re: Post-incident review on the Atlassian April 2022 outage

#62

Post by doctor_eval »

dopylitty wrote: post #2

I can only imagine the way the person who pulled the trigger on the deletion script felt the moment they realized what had happened. I’ve been there with much less significant incidents when a “routine” change turned into a potentially resume generating event. It’s not fun. Ultimately the responsibility is with the organization that made it possible for an event of that scale to happen rather than the individual pers…

I was on call “back in the day” when a customer rang to tell me that she’d meant to type:

    # find /tmp -exec rm {} /;
But instead she’d typed

    # find / tmp -exec rm {} /;
I learned a lot about Unix (and, as it happens, X windows) file systems that week. Note that this was long before tools like sudo existed, where cleaning up tmp was often a manual process.

There is no way we could blame a person for a single keystroke error - it’s crazy that the command worked - but I now look at the find command with an enormous amount of fear.

Re: Post-incident review on the Atlassian April 2022 outage

#64

Post by ekimekim »

baskethead wrote: post #34

Oooof. Passing in Application IDs will delete applications and passing in site IDs will delete sites. That's a really really bad design. I'm bookmarking this so that I can use it as a showcase going forward. Just this week, I changed a spec in one of our proposed endpoints that did exactly that. We passed in ids of various types of objects to perform actions, and I changed the api so that it would be forced to pass i…

This reminds me of how useful it is to make your ids include the type of the thing they're identifying as part of the value. This could be as simple as something like "u" for users, "p" for projects, etc. Or it could be a full-blown URN scheme like some of the big cloud providers use (though please, whatever you pick, BE CONSISTENT about using that form and only that form. If there's an operation to exact "just the id" from your larger value, and you need to use the full thing in some places and "just the id" in others, then you've missed the point).

Re: Post-incident review on the Atlassian April 2022 outage

#65

Post by ripper1138 »

martin_drapeau wrote: post #63

Implementing soft deletes is a lessons every developer learns early in their career. The fact that Atlassian did not implement that in their cloud, is mind boggling. Great case study of a monumental fuck up!

So many opportunities missed to avoid this! Look at one ID and ensure it is what you expect it to be. Run the script in a dryrun mode. Run the script for 1 customer. Probably more!

Re: Post-incident review on the Atlassian April 2022 outage

#67

Post by nreilly »

ilayn wrote: post #24

I am very curious if they used a Jira board during this crisis for issue tracking. Because then they would have more than 4 lessons learned.

From the article:

"To manage the restoration progress we created a new Jira project, SITE, and a workflow to track restorations on a site-by-site basis across multiple teams (engineering, program management, support, etc). This approach empowered all teams to easily identify and track issues related to any individual site restoration."

Re: Post-incident review on the Atlassian April 2022 outage

#70

Post by nomilk »

eyelidlessness wrote: post #53

I sincerely hope all of the people who worked on the recovery effort are okay, and are being well supported and strongly encouraged to tend their mental health. I have no personal investment in Atlassian products—if anything, unrelated to this or any incident, I could happily never use them again. But the people who work there are human, and I know what kind of a toll a protracted recovery effort can take. I know it…

As someone who's worked on similar situations, I don't expect any pat on the back, but see it as my responsibility to make sure it doesn't happen in the first place, and when it does, my responsibility to fix it without complaint.

Some Atlassian customers might have had much more severe (mental health and other) problems than the Atlassian staff, so I think it can be perceived as mildly solipsistic to be praising the staff and (being perceived to) forget about the customers' wellbeing (I know you didn't say anything about forgetting about the customers, but affected customers may perceive it that way).

Post reply on HN