Inside the longest Atlassian outage • Hacker News Board

Re: Inside the longest Atlassian outage

#411

Post by beachy » Wed, Apr 13, 2022, 9:02 PM UTC

This is extremely poor for a large SaaS company.

A standard RFP question for SaaS should be:

- Can you restore data for a single customer, and if so, what is the RTO for that operation?

A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival.

But for any decent size multi-tenanted SaaS, it's imperative that you have the ability to selectively restore individual customers.

The usual approach is to do a full database restore into a separate instance, then run your pre-prepared "restore customer" scripts to extract a single customer's data from there and pump it across your prod instance. In Oracle for example you might use database links to give your restore code access to prod and also the restore instance at the same time.

Atlassian - MUST DO BETTER.

Re: Inside the longest Atlassian outage

#412

Post by jacquesm » Wed, Apr 13, 2022, 9:02 PM UTC

dangrossman wrote: post #409

nemo1618 wrote: post #399

Earlier quoted context omitted.

After I met my now-fiancée on OkCupid, I deactivated my profile, turned off notifications and forgot about it for a while. A while later, I thought it be nice to revisit the first messages we sent to each other, only to find that... OkCupid had deleted both of our accounts. They didn't give me any advance warning, either, because I turned off notifications, remember? :^) I'm still kinda salty about it. I understand w…

With GDPR, privacy regulations and data breach regulations sweeping the globe, holding onto unnecessary data is a huge liability. Getting rid of data you no longer have clear consent to store, or which you're unlikely to have a clear business need to continue storing, is a sign of a good company these days.

True, but likely not this kind of data.

Re: Inside the longest Atlassian outage

#413

Post by dangrossman » Wed, Apr 13, 2022, 9:02 PM UTC

hallway_monitor wrote: post #358

hinkley wrote: post #189

Earlier quoted context omitted.

Someone somewhere got a promotion sooner because they lowered the slope of a line a little bit.

Or some overzealous engineer said hey guys let's delete all data 7 days after an account is canceled. This is called over optimizing.

Such a decision is just as likely to have come from the legal/compliance team as an engineer. Data you no longer have clear consent or a legitimate business need to store is a liability, and if you operate in Europe, potentially illegal to continue storing.

Re: Inside the longest Atlassian outage

#414

Post by nindalf » Wed, Apr 13, 2022, 9:03 PM UTC

yebyen wrote: post #408

notatoad wrote: post #392

Earlier quoted context omitted.

No, I don't think that's disturbing. That's the point of backups - even when something is permanently and completely erased in the production database, it's still in the backup. Eventually it will get rotated out as the backups expire. Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.

I think that depends on what you mean by compliance. Some regulations require you to irreversibly destroy data when they prescribe the destruction of that data. That can mean as much as "you have to encrypt everything with a separate key, so that you can destroy the key for the given (say, personally identifiable) dataset making its retrieval irrecoverable" I'm not saying that's the particular compliance reason they…

This is why regulations specify that data must be destroyed within a time period, typically 90 days. It gives enough time for backups to rotate out.

If this weren’t a concern, regulations would demand immediate deletion of data.

Re: Inside the longest Atlassian outage

#415

Post by __app_dev__ » Wed, Apr 13, 2022, 9:03 PM UTC

jmondi wrote: post #110

What blow's my mind is that Atlassian stock has barely taken a hit...

Yeah, I place a Put option order yesterday. By end of day I was up over 50% and now down to 50% of what I original purchased the Put at because it went up 5% today.

Oh well, better luck next time.

Re: Inside the longest Atlassian outage

#416

Post by miketria » Wed, Apr 13, 2022, 9:03 PM UTC

politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

Hi, this is Mike from Atlassian Engineering. For the customers impacted by this incident covered by an SLA, we will adhere to our contractual terms. However, given the long duration of this outage, we are planning to go above and beyond for our impacted customers. We are currently focused on restoring service, but after that will be discussing how we can make it right for each impacted customer.

Re: Inside the longest Atlassian outage

#417

Post by __app_dev__ » Wed, Apr 13, 2022, 9:05 PM UTC

devmunchies wrote: post #134

jmondi wrote: post #110

What blow's my mind is that Atlassian stock has barely taken a hit...

it was at $317 on the day of the outage and now at $278.5. A ~12% drop. You're right, not much of a drop for such a large outage.

The outage did not impact the stock, most major tech stocks have taken a large hit in the past week and a half (until today).

This even is not even showing on any financial news site. I'm still hoping it does and the stock goes down because I place an option order yesterday betting that it goes down by next Friday. Seems like it won't now but the risk was worth taking in my book.

Re: Inside the longest Atlassian outage

#418

Post by duxup » Wed, Apr 13, 2022, 9:06 PM UTC

adamc wrote: post #403

duxup wrote: post #8

Earlier quoted context omitted.

As this continues I suspect that this might be one of the few times where a lack of transparency / good communication really ... might not be better or worse because the situation is so bad that transparency would be horrible just the same. Granted that's how all lies start / what sometimes people assume and they're wrong but ... maybe this is that time? Maybe it is in fact so bad that honesty would be a push or wors…

If so, that itself would be a huge red flag for dealing with Atlassian.

I think it is…either way.

Re: Inside the longest Atlassian outage

#419

Post by spookthesunset » Wed, Apr 13, 2022, 9:06 PM UTC

Cederfjard wrote: post #260

PSA, because I’m seeing a lot of JIRA in this thread: Since the 2017 rebranding, Jira is no longer officially written in all caps: https://community.atlassian.com/t5/Feedback-Forum-articles/A... (You can argue how successful it was when people are still using the old style in 2022). It also makes more sense, since Jira is not an acronym, it’s a truncation of Gojira, inspired by Bugzilla/Mozilla.

Yeah, I've never typed it as anything but JIRA. Pretty sure my auto-complete will vouch for that.

Re: Inside the longest Atlassian outage

#420

Post by luhn » Wed, Apr 13, 2022, 9:07 PM UTC

a2800276 wrote: post #389

Alex3917 wrote: post #44

A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

It's generally recognized that deleting data from a backup would violate the integrity of the backup, so allowances are made. Usually you have to make sure the data is deleted as part of the restore process. For example, from CCPA:

> If a business stores any personal information on archived or backup systems, it may delay compliance with the consumer's request to delete, with respect to data stored on the archived or backup system, until the archived or backup system relating to that data is restored to an active system or next accessed or used for a sale, disclosure, or commercial purpose.