Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-incident review on the Atlassian April 2022 outage

atlassian.com

Post reply on HN
Print view View original

141–150 of 157 posts

Re: Post-incident review on the Atlassian April 2022 outage

#141

Post by lultimouomo »

Griffinsauce wrote: post #116
helmholtz wrote: post #94

Earlier quoted context omitted.

I'm curious about your take on the cheapening or dilution of words in language nowadays. One isnt hurt but traumatized, one isn't "once bitten twice shy" but suffering from PTSD. It's assault if one gets in a fight and so on. I sometimes feel we're running out of language to distinguish the truly extreme from the quotidian. Same thing with how GP phrased it. A hard week of work comes across as having gone to battle a…

> nowadays Any argument that uses this qualification should be accompanied with data. You haven't specified anything, not even a rough time frame. Your statement is so vague nearly anything can be projected onto it. Also a sidenote: if you're going to make an argument about language and definitions it seems like a good idea to actually know what PTSD is before using it as an example.

> Any argument that uses this qualification should be accompanied with data.

https://trends.google.com/trends/explore?date=all&q=Traumati...

https://trends.google.com/trends/explore?date=all&q=PTSD

Assault is going the other way though:

https://trends.google.com/trends/explore?date=all&q=assault

Re: Post-incident review on the Atlassian April 2022 outage

#142

Post by throwaway787544 »

giraffe_lady wrote: post #85
throwaway787544 wrote: post #80

Earlier quoted context omitted.

It should be shouted from the rooftops: don't switch services just to save money if the result is potentially worse business outcomes. Why save a tiny bit of cash if it puts your business at risk?

Presumably no one who made that decision is fucking stupid and thought they were putting their business at risk. Good lord. I wasn't affected by this in the slightest and just found out opsgenie exists from the parent comment but even I can understand that this decision would almost certainly be driven by things like "we're already using atlassian for everything else and will benefit from the interop" and "we already…

There's no need to get that upset over it...

When we switched to OpsGenie it was also to consolidate billing with other Atlassian products, but we all knew PagerDuty worked better and that OpsGenie was still pretty new and rough around the edges. We certainly didn't gain any business advantage by switching, but we did have to do a lot of work to switch which took away from other things we needed to get done. But ultimately we had no say because somebody just wanted to save a little money. I also doubt that there was a thorough vendor assessment before we picked it up, since it was a vendor we already used.

Re: Post-incident review on the Atlassian April 2022 outage

#143

Post by Bender »

doctor_eval wrote: post #62
dopylitty wrote: post #2

I can only imagine the way the person who pulled the trigger on the deletion script felt the moment they realized what had happened. I’ve been there with much less significant incidents when a “routine” change turned into a potentially resume generating event. It’s not fun. Ultimately the responsibility is with the organization that made it possible for an event of that scale to happen rather than the individual pers…

I was on call “back in the day” when a customer rang to tell me that she’d meant to type: # find /tmp -exec rm {} /; But instead she’d typed # find / tmp -exec rm {} /; I learned a lot about Unix (and, as it happens, X windows) file systems that week. Note that this was long before tools like sudo existed, where cleaning up tmp was often a manual process. There is no way we could blame a person for a single keystroke…

I ran into a similar one a few times from the same team that kept doing things in their scripts like

  rm -rf ${basedir}/*
without setting

  set -u
to exit if there are undefined variables. I suggested a few other sanity checks but was told I was adding friction.

Re: Post-incident review on the Atlassian April 2022 outage

#144

Post by hobs »

solatic wrote: post #139
bigbluedots wrote: post #111

> During this incident, we missed our RTO but met our RPO. You missed your recovery time objective by ~2 weeks and did not properly communicate the issue from senior management until around a week into the outage. It is great to hear about how the company plans to do better, let's see how the next outage improves things.

Noting that they met their RPO is technically correct but tone-deaf. Nobody cares that you lost very little customer-submitted data if customers rely on you (mission-critical) to continue to accept data, and the outage prevented that.

That's simply not true, its just not good enough - if you lost customer data as well people would be livid.

Re: Post-incident review on the Atlassian April 2022 outage

#145

Post by therealcamino »

nreilly wrote: post #67
ilayn wrote: post #24

I am very curious if they used a Jira board during this crisis for issue tracking. Because then they would have more than 4 lessons learned.

From the article: "To manage the restoration progress we created a new Jira project, SITE, and a workflow to track restorations on a site-by-site basis across multiple teams (engineering, program management, support, etc). This approach empowered all teams to easily identify and track issues related to any individual site restoration."

I read "empowered" there and thought, what a ridiculous misstep. They can't even stop themselves from selling their product in a document that's supposed to be a mea culpa, where you can guarantee that many of the readers are pissed off and absolutely not receptive to your marketing.

Re: Post-incident review on the Atlassian April 2022 outage

#146

Post by tetha »

SnowHill9902 wrote: post #11
dopylitty wrote: post #2

I can only imagine the way the person who pulled the trigger on the deletion script felt the moment they realized what had happened. I’ve been there with much less significant incidents when a “routine” change turned into a potentially resume generating event. It’s not fun. Ultimately the responsibility is with the organization that made it possible for an event of that scale to happen rather than the individual pers…

That moment when you see DELETE 18388272773 0

Reasons why I don't enter writing queries without starting a transaction first. ABORT, good sir, ABORT.

Re: Post-incident review on the Atlassian April 2022 outage

#147

Post by tetha »

Too wrote: post #127
pianoben wrote: post #108

Earlier quoted context omitted.

One of the favorite tricks tricks I've ever seen is how Twilio uses human-readable prefixes[0] on their various identifiers - you will never mistake a device (HSxxxxxxx) for an account (ACxxxxxxxx). It's prevented us (Twilio customer) from making similar mistakes in the past. [0]: https://www.twilio.com/docs/glossary/what-is-a-sid#common-si...

I like the idea of human readable identifiers. But generally feels like this class of error could be prevented with more type safety in the api and data model? Like deleteDevice(123) and deleteAccount(123), rather than delete(123). This is how REST is designed, the type of resource is already baked into the url.

Those go hand-in-hand. Tell me if deleteDevice(123) or deleteAccount(123) is wrong, opposed to deleteDevice(A123) and deleteAccount(D123).

Re: Post-incident review on the Atlassian April 2022 outage

#148

Post by Aeolun »

pixeltopic wrote: post #115
Aeolun wrote: post #90

Earlier quoted context omitted.

But you don’t need a paging system if your services don’t go down. Isn’t the fact you can’t deal without one for two weeks an indictment of your own practices?

Well said - you don't need to write tests for your code if you don't write bugs!

That’s not equivalent. Code changes every day (potentialy quite extensive), but the same thing is not true for infra, which hopefully stays mostly the same from day to day.

If your incident reporting is down, hopefully you completely stop changing anything about your infra.

Re: Post-incident review on the Atlassian April 2022 outage

#149

Post by yakshaving_jgt »

PeledYuval wrote: post #133
yakshaving_jgt wrote: post #131

> The API used to perform the deletion accepted both site and app identifiers and assumed the input was correct Yet another case where using types would have prevented a massive problem.

How do you imagine types would prevent this? No matter how I think about types in this context, I think a runtime check of "is this ID an app ID?" is mandatory in order to truly prevent this (alternatively- different URLs/parameter names to delete an app and a site)

Yes, you're certainly right that this check would have to happen at runtime (unless Atlassian have written one big project which is all checked by a compiler, which I highly doubt).

How I imagine this to work would be that the team who wrote the system (function, script, whatever) for deleting things would encode it in the types that their deletion system would only accept specifically app IDs, rather than both app IDs and site IDs (which I think reflects the specifics of their post-mortem).

In order to get a value into the deletion system for processing, the ID value would need to be parsed into the specific type that the deletion system accepts.

This is of course similar to just saying there should have been validation, but I think conceptually, validation and parsing into a narrower type are two different things. Gary Bernhardt calls this "functional core, imperative shell". Michael Feathers calls this "edge-free programming". Probably the best literature on this difference is by Alexis King, here: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

Re: Post-incident review on the Atlassian April 2022 outage

#150

Post by pixeltopic »

Aeolun wrote: post #148
pixeltopic wrote: post #115

Earlier quoted context omitted.

Well said - you don't need to write tests for your code if you don't write bugs!

That’s not equivalent. Code changes every day (potentialy quite extensive), but the same thing is not true for infra, which hopefully stays mostly the same from day to day. If your incident reporting is down, hopefully you completely stop changing anything about your infra.

It's naive to expect that things will just stay fine and dandy if you "stop changing" your infra. Consider scenarios such as traffic spikes, network outages, and under-provisioned resources.
Post reply on HN