Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

551–560 of 772 posts

Re: Inside the longest Atlassian outage

#551

Post by [deleted] »

voxic11 wrote: post #517
notatoad wrote: post #465

Earlier quoted context omitted.

"permanently delete" does not mean the same thing as "immediately delete". deleting from the live database is the first step of a permanent deletion, as long as the data exists somewhere the deletion process is still in-progress. there's a whole lot of people in here who are way too quick to assume that just because one part of a permanent deletion process was inadvertently triggered and then caught while they still…

https://ico.org.uk/for-organisations/guide-to-data-protectio... You seem to be right-ish, while the gdpr in certain circumstances allows you to keep backups of data that should have been deleted it seems like they are trying to discourage it in the future. > ...It is, however, important to note that where data put beyond use is still held it might need to be provided in response to a court order. Therefore data contr…

[deleted]

Re: Inside the longest Atlassian outage

#552

Post by iancarroll »

theteapot wrote: post #541
iancarroll wrote: post #525

Earlier quoted context omitted.

Well, their explanation makes sense. These are multi-tenant environments where not every tenant was affected; sensibly, the backups appear divided by environment, not tenant. You can’t blindly revert to an environment’s last backup in this scenario, although you’d think they would have done it before.

Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.

You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.

Re: Inside the longest Atlassian outage

#553

Post by twistedpair »

a2800276 wrote: post #389
Alex3917 wrote: post #44

A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

One assumes the backups will rotate out and be discarded after a fixed time, so everything is "truly deleted" for compliance reasons.

Re: Inside the longest Atlassian outage

#554

Post by devoutsalsa »

madmulita wrote: post #511
kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

They claim they test backups quarterly yet they don't have a procedure in place to restore the operation. We all know your backup is not tested until you restored everything successfully. This is not an engineering mistake, it is a flat out lie.

They'd test monthly, but a full restore take a quarter XD

Re: Inside the longest Atlassian outage

#555

Post by notatoad »

jacobsenscott wrote: post #518
a2800276 wrote: post #424

Earlier quoted context omitted.

"permanently delete" strongly suggests to me that it was the "medical and financial data" kind of compliance. If data can be restored, it's not permanently deleted. But this was a statement from the CEO, so words can have arbitrary meaning :)

A better way to do this sort of thing is not an actual "delete", but a "cryptographic delete". The data should be encrypted, and you just delete the key. The data is then unrecoverable everywhere, including backups. Of course you probably don't want to just nuke the key, but disable it for some period of time, and then nuke it.

i don't see how that really changes anything - your keys should be backed up just as much as, if not more than your data. and any process for deleting the encryption keys should allow for restoring from backups for some period of time just the same as your process for deleting data should allow restoring from backups for some period of time. either way, permanently rendering data as unrecoverable takes time.

Re: Inside the longest Atlassian outage

#556

Post by jacobsenscott »

notatoad wrote: post #555
jacobsenscott wrote: post #518

Earlier quoted context omitted.

A better way to do this sort of thing is not an actual "delete", but a "cryptographic delete". The data should be encrypted, and you just delete the key. The data is then unrecoverable everywhere, including backups. Of course you probably don't want to just nuke the key, but disable it for some period of time, and then nuke it.

i don't see how that really changes anything - your keys should be backed up just as much as, if not more than your data. and any process for deleting the encryption keys should allow for restoring from backups for some period of time just the same as your process for deleting data should allow restoring from backups for some period of time. either way, permanently rendering data as unrecoverable takes time.

As an example, if you are using Amazon's KMS for key management and you destroy a key it gives you 7 days to undo before permanently destroying the key. Or you can disable they key and destroy it later as your retention policy permits. Surely they have some kind of key backup, but KMS users have no access to those backups.

Re: Inside the longest Atlassian outage

#557

Post by broknbottle »

iamtheworstdev wrote: post #477
kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

Can confirm. Saw them there while I was on vacation.

Thanks narc, now our proper holiday to Malia Greece was put on indefinite hold. Vegas was only supposed to be the pre game warm up.

Re: Inside the longest Atlassian outage

#558

Post by twistedpair »

bluedino wrote: post #145

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…

This is why I love GCP Cloud Storage. The "colder" tiers are cheaper, and reads simply cost a lot more from there, but they don't slow them down and take days to restore. You pay with dollars, not time for restoring those GCS backups. e.g. Coldline [1] simply has reduced availability in exchange for being cheaper (99.9-99.95% availability, so 43min/mo, way less than "two days").

[1] https://cloud.google.com/storage/docs/storage-classes#coldli...

Re: Inside the longest Atlassian outage

#559

Post by jamesfinlayson »

nemo1618 wrote: post #399
Alex3917 wrote: post #141

Earlier quoted context omitted.

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

After I met my now-fiancée on OkCupid, I deactivated my profile, turned off notifications and forgot about it for a while. A while later, I thought it be nice to revisit the first messages we sent to each other, only to find that... OkCupid had deleted both of our accounts. They didn't give me any advance warning, either, because I turned off notifications, remember? :^) I'm still kinda salty about it. I understand w…

I kind of agree - a company I used to work at used free Slack for years, then HipChat (until Atlassian killed it - with good reason), then converted to paid Slack, and all of our chat history was still there - even the old stuff that gets hidden as part of the free plan.

Re: Inside the longest Atlassian outage

#560

Post by [deleted] »

Kwpolska wrote: post #428
Alex3917 wrote: post #141

Earlier quoted context omitted.

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

> why would you ever delete someone's data when it's literally only 160 KB of text? Compliance? The contract has expired, so there’s no legal basis for them to keep your data?

[deleted]
Post reply on HN