Inside the longest Atlassian outage • Hacker News Board

Re: Inside the longest Atlassian outage

#521

Post by syshum » Wed, Apr 13, 2022, 10:55 PM UTC

systemvoltage wrote: post #486

syshum wrote: post #480

Earlier quoted context omitted.

sales never takes the blame. If anyone is fired it will be scapegoats in engineering once they have busted their ass to restore their reward will be the door

This is an engineering problem. They should own it and improve things, make sure it doesn't happen again. Also, GP's quote > Engineering mistakes happen. I don't like this statement because it offers consolation at the expense of unintentional normalization.

The deletion of customer data was engineering mistake, that is not what I was talking about

The Negative fall out was not due to the deletion of customer data, as the Story and multiple customers have stated the negative fall out was the SILENCE / lack of communications, which is Sales / Customer Service not engineering

As the comment I was replying to noted while engineering was trying to recover from what might possibly be the biggest outage in the history of the company Sales was partying and not handling customer communications

That (the failure to communicate with customers) should be a resume generating event of all leadership customer service / sales. It will not be because sales will simply redirect their failure on to engineering in the exact same manner you just have

Re: Inside the longest Atlassian outage

#522

Post by DannyBee » Wed, Apr 13, 2022, 10:56 PM UTC

femto113 wrote: post #239

tyingq wrote: post #175

Earlier quoted context omitted.

>I still don't understand the strangehold JIRA has on some clients. - Integrations with things like the source code repos, incident management systems, confluence or other wikis, Slack, etc. Moving away from Jira creates a bunch of dead links. - Internal dependence on complex workflows and state transition rules that are implemented in Jira. - Various very customized reports that leaders depend on to make decisions,…

Many years worth of source code filled with comments like // if we don't toggle bit 7 here 10% of transactions will fail on Thursdays // see JIRA issue BIGPROJ-12654 for detailed discussion

Having migrated bug systems for very large, very old code bases before, it's pretty easy to make the URls and links like this still go to the right place.

This is actually the least difficult thing, i would say ;)

Re: Inside the longest Atlassian outage

#523

Post by doliveira » Wed, Apr 13, 2022, 10:56 PM UTC

akie wrote: post #132

doliveira wrote: post #95

Earlier quoted context omitted.

By segregating as much as you can. Definitely not by putting everything in a single table. At the very least separate databases/schemas with proper permissions so there's not any chance of data intermiBy segregating as much as you can. Definitely not by putting everything in a single table. At the very least separate databases/schemas with proper permissions so there's no chance of data intermixing. The best would be…

Now every time you run a database migration, you have to adjust N tables - and in Atlassian's case, N is 200000. Is that better? It depends. There is no "best" way of doing multitenancy.

There is a worst way of doing multitenancy, and that is sharing a single big table.

Re: Inside the longest Atlassian outage

#524

Post by doliveira » Wed, Apr 13, 2022, 10:57 PM UTC

hnlmorg wrote: post #379

robertlagrant wrote: post #371

Earlier quoted context omitted.

The bug we are mitigating was also just an automation issue.

It’s also pretty easy to foobar up a single DB instance if you don’t have proper guardrails in place. Automation wasn’t the issue here. It’s the symptom not the cause.

Way easier, actually.

Re: Inside the longest Atlassian outage

#525

Post by iancarroll » Wed, Apr 13, 2022, 10:57 PM UTC

madmulita wrote: post #511

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

They claim they test backups quarterly yet they don't have a procedure in place to restore the operation. We all know your backup is not tested until you restored everything successfully. This is not an engineering mistake, it is a flat out lie.

Well, their explanation makes sense. These are multi-tenant environments where not every tenant was affected; sensibly, the backups appear divided by environment, not tenant. You can’t blindly revert to an environment’s last backup in this scenario, although you’d think they would have done it before.

Re: Inside the longest Atlassian outage

#526

Post by lovehashbrowns » Wed, Apr 13, 2022, 11:00 PM UTC

deckard1 wrote: post #446

hu3 wrote: post #19

Earlier quoted context omitted.

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 cu…

I don't want to assume too much, since the details are sparse. But I know for a fact that few of my current coworkers know a thing about writing tooling code. It's becoming a bit of a lost art. Here's the way such a script should be done. You have a dry-run flag. Or, better yet, make the script dry-run only . What this script does is it checks the database, gathers actions, and then sends those actions to stdout. You…

That’s how I did one of my more impactful deduplication/deletion scripts. It had to reach across environments to do its work. But there was no way to send any flags to it to do stuff. The environment names were hard coded, so like dev-uw2 reaching out to stg-ue1. It would output a dry run result by default. And you could look and see what was going to get deleted and from what environment.

Because the names were hard coded, I had to get changes approved in GitHub. Then the script would run on Jenkins.

That script was also only for that purpose and nothing else. It made a mess because I needed a ton of functionality around creation and querying, too. I just copied the script to folders and modified them as needed but a better solution would’ve been to make a python module. I just liked the code itself being highly specific to what the script was doing to help reduce mistakes. If I’m running a script to delete repos, I need to go to the delete-repos directory.

Re: Inside the longest Atlassian outage

#527

Post by a-dub » Wed, Apr 13, 2022, 11:01 PM UTC

miketria wrote: post #469

bombcar wrote: post #167

Earlier quoted context omitted.

Yeah, soft delete is the way to go in 99.99% of the cases, with a system setup to eventually hard delete on some schedule (preferably don't hard delete until X number of backups have caught the soft deleted data safely, for example).

Hi, this is Mike from Atlassian Engineering. Strongly agree with this. I'd say that if you can afford it, don't do the hard deletes on a schedule though. You never know when there's a system out there referring to soft deleted data that fails once the data is hard deleted. Hard deletes should feel frightening because they are frightening.

i disagree for one reason. you really don't want the tooling or the process to rot. running it automatically normalizes the scary. otherwise you have bespoke tools in indeterminate states being run by people who are learning how to run them again. that's when i believe things get dangerous.

if it forces additional fail safes or backups to be able to do so safely, then that's probably a good thing to have anyway, no?

Re: Inside the longest Atlassian outage

#528

Post by scottlamb » Wed, Apr 13, 2022, 11:05 PM UTC

Delitio wrote: post #425

notatoad wrote: post #392

Earlier quoted context omitted.

No, I don't think that's disturbing. That's the point of backups - even when something is permanently and completely erased in the production database, it's still in the backup. Eventually it will get rotated out as the backups expire. Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.

Nope it's not ridiculous. If you are only allowed to store data for x month that's it. It's your job to use technics which allow you to do this like using encryption on your backup and deleting the keys for it, for example.

Delitio> If you are only allowed to store data for x month that's it.

Exactly. I'm not aware of any laws saying "you must delete this data immediately". More like "within X days or months". The permanently delete thing presumably skips some cooling-off period in the online database but not the backup, which seems perfectly appropriate, provided your backup retention is compliant.

Google has a nice page describing out their deletion process. [1] It doesn't go into product-specific technical details/steps (like marked as deleted within the product, row deleted from Bigtable/Spanner, major compaction guaranteed to happen, backups guaranteed to be deleted or unusable) but it says this:

Google> We then begin a process designed to safely and completely delete the data from our storage systems. Safe deletion is important to protect our users and customers from accidental data loss. Complete deletion of data from our servers is equally important for users’ peace of mind. This process generally takes around 2 months from the time of deletion. This often includes up to a month-long recovery period in case the data was removed unintentionally.

This is a best practice.

Delitio> It's your job to use technics which allow you to do this like using encryption on your backup and deleting the keys for it, for example.

If they'd thrown away the encryption key immediately, this would have been much worse. Instead of "we're down for 2 weeks?!?" (already quite bad) it'd be "our data is gone forever?!?". You never want to delete anything too quickly for exactly this reason.

[1] https://policies.google.com/technologies/retention?hl=en-US

Re: Inside the longest Atlassian outage

#529

Post by fphhotchips » Wed, Apr 13, 2022, 11:10 PM UTC

syshum wrote: post #480

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

sales never takes the blame. If anyone is fired it will be scapegoats in engineering once they have busted their ass to restore their reward will be the door

Reformed salesperson here: bullshit.

Atlassian famously eschews the exact sales teams whose job it would be to manage direct customer comms in an outage like this one, and to be the lightning rod for the understandable customer frustration. In the past, I've been the guy that gets the angry text message from the customer and has to carefully paper over the gaps in communication from higher ups. It's not fun being the neck that gets choked.

The complete lack of meaningful communication for so long indicates to me that Atlassian doesn't have a meaningful feedback mechanism from the field back up to the executive suite - the exact feedback mechanism that Sales and Sales Engineering teams fill in most SaaS orgs. Customer Success should fill that role, but IME don't have the same incentives, pressure or influence as sales teams watching half their yearly comp go down the pipes.

Re: Inside the longest Atlassian outage

#530

Post by jacquesm » Wed, Apr 13, 2022, 11:11 PM UTC

callalex wrote: post #491

jacquesm wrote: post #412

Earlier quoted context omitted.

True, but likely not this kind of data.

Yes, this kind of data. Your OkCupid account has all kinds of information about who you associate with.

That's true but it was stored there with your explicit consent. The GDPR is first and foremost concerned with data that is stored about you without your consent or with data that continues to be stored about you after your explicit request for deletion. Or incorrect data that you have requested to be removed. See the wikipedia page on the GDPR or a bunch of articles that I wrote about this subject.

If they had obtained the data without you supplying it freely then that would have been an entirely different matter, especially if it was used in ways that you did not consent to. But since that does not appear to be the case here the GDPR applies like it does to all data that is directly related to a data subject but continuing to store it on behalf of the user(s) that supplied it is not a problem.

Note that the user here is disappointed that their data which they consented to be kept is no longer there. This is a pretty clear indication that as far as they are concerned their expectation was the even with the GDPR up and running that such data would continue to be preserved as it is in almost every service that existed prior to may 2018.

It is precisely this kind of panicky thinking around the whole subject of the GDPR that gives these irrational responses, companies that suddenly no longer dare to mail you but you have to log in to their portal, which is secured by your email address and more of these totally weird constructs.

If they wanted to delete this data the better way would have been to positively contact the user (so that you know that they have received your message) to ask if their data should be deleted or not. That's good stewardship, just tossing it isn't.