Sure. But I hear they still only offer one type of cheese....
Post-incident review on the Atlassian April 2022 outage
51–60 of 157 posts
Re: Post-incident review on the Atlassian April 2022 outage
#52Post-incident reviews are really a means for (black) PR and nothing more than covert advertising. Few people if any care at all about these things in the busy landscape of devops and whatever. Eventually everyone is let-go, because every sane software engineer knows that software works by magic on a very large scale and outages and incidents are everyday life.
Re: Post-incident review on the Atlassian April 2022 outage
#53I sincerely hope all of the people who worked on the recovery effort are okay, and are being well supported and strongly encouraged to tend their mental health. I have no personal investment in Atlassian products—if anything, unrelated to this or any incident, I could happily never use them again.
But the people who work there are human, and I know what kind of a toll a protracted recovery effort can take. I know it from chronic pain two years after the burnout started. No one should experience that.
I know it’s probably not the thing Atlassian the business thinks they should communicate to customers/general public, and it’s not surprising it’s absent, but it’s a shame they did not address the well being of their staff. Especially those who undoubtedly put exhausting effort into recovering from this incident.
And sure, that would be unusual as a business communication. But should it be? Even before this severe burnout, I would have looked at this list of lessons learned and thought “a lot of people are going to be overworking the same as they’ve been through all of this, but now they’ll be doing it invisibly.” And that definitely doesn’t inspire confidence that there won’t be another catastrophic mistake in the near future—it doesn’t make me more confident in the business.
None of this is a strong criticism, just some observations from being on the other side of a marathon incident recovery.
Re: Post-incident review on the Atlassian April 2022 outage
#54Earlier quoted context omitted.
I tend to agree. However, there's one point that makes me skeptical: there are no organizational changes, or changes to leadership, or anything in that direction. This sounds like "the tech guys screwed up, culture and management is fine here". Which it might be, or it might not. I would have loved to see 5. We will stop pushing customers so hard towards using our cloud for one, but that wouldn't be convenient for At…
Note that the ToS also forbid Cloud users from “disseminating information” about “the performance of the products”. So you can’t say it’s slow or unperforming.
Is that real? That is so bad. Queue up next “Shall not disclose or notice our incompetence” clause.
Re: Post-incident review on the Atlassian April 2022 outage
#55So what else is being done for impacted customers? Are they getting money back? What's Atlassian's SLA for these services and what can be recouped under it?
Re: Post-incident review on the Atlassian April 2022 outage
#56I can't say that I've ever been a fan of Atlassian or their products, but this blog post makes it sounds like they've at least learned the right lessons from this: 1. Establish universal "soft deletes" across all systems. 2. Better DR for multi-site, multi-product incidents. 3. Fix their incident-management process for large-scale incidents. 4. Fix their incident communications. Regarding #4 in particular: "Rather th…
I am actually pleasantly surprised at the openness of this response, and their taking responsibility of mistakes and detailing what will change in the future. It's not just corporate speak. I think that speaks well for the company and it improved my view of them.
Why do you assert that this is not just corporate pr?
Re: Post-incident review on the Atlassian April 2022 outage
#57I sincerely hope all of the people who worked on the recovery effort are okay, and are being well supported and strongly encouraged to tend their mental health. I have no personal investment in Atlassian products—if anything, unrelated to this or any incident, I could happily never use them again. But the people who work there are human, and I know what kind of a toll a protracted recovery effort can take. I know it…
I was thinking of this - working on this incident sounds like an exhausting amount of work for so many people.
Re: Post-incident review on the Atlassian April 2022 outage
#58Earlier quoted context omitted.
I am actually pleasantly surprised at the openness of this response, and their taking responsibility of mistakes and detailing what will change in the future. It's not just corporate speak. I think that speaks well for the company and it improved my view of them.
Why do you assert that this is not just corporate pr?
Of course, this is a part of public relations for a corporation.
But usually when we people talk about "corporate pr" they talk about weasel words, non-committal statements, and deflecting blame away from them. I think this PIR does a decent job of acknowledging the mistakes they make and how they can improve.
Re: Post-incident review on the Atlassian April 2022 outage
#59Earlier quoted context omitted.
Datomic treats all data as immutable, so it can wind back to any version. When new data is written, the entire block is copied and rewritten rather than changing the data in-place.
This is similar to SQL checkpoints in that a rollback is all or nothing. It wouldn’t be segmented by tenant unless the each tenant has its own transactors.
The difference is that, unless you’re using a SQL database with direct/extended support for write-once immutability or a data model designed for it, once the transaction exits every successful committed change is ~final.
A write-once immutability design (whether in the DB itself, or an extension, or a userspace implementation) lets you reconsider and rework mistakes after they’re committed, not unlike how you might do with git rebase etc.
Re: Post-incident review on the Atlassian April 2022 outage
#60Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…
In MVCC systems like PostgreSQL, if you don't vacuum (garbage collect the old tuples), your database is append-only and you can query as if your transaction was started at some time in the past. I don't know how to set auto-vacuum to have a fixed delay, e.g. keeping 24h of changes, but I bet it can be added if it's not built-in.
I don’t understand why you’re being downvoted and I hope someone will explain.