Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

421–430 of 772 posts

Re: Inside the longest Atlassian outage

#421

Post by mc4ndr3 »

politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

I've yet to work at an office that paid sufficient attention to regular backup & restore validation, to scalable design, or proper unit testing, or to basic security updates. Upper management is repeatedly incentivized to produce vaporware, not reliable service.

Suits think a crummy Flash quiz on PII is enough to stop leaks. The automotive industry couldn't stop airbags from acting as claymores. It's even harder to get good code approved in tech.

Re: Inside the longest Atlassian outage

#423

Post by __app_dev__ »

radicaldreamer wrote: post #75
1970-01-01 wrote: post #63

Interesting note: Atlassian stock (NASDAQ: TEAM) is up 4% as of noon today.

It might be a good short opportunity... I imagine a lot of customers are kicking off their own internal process for migrating away from JIRA. By the time they actually do, it'll be at least a couple of quarters from now, which is when the customer hit will start materializing in quarterly results for the company. Maybe time to throw a few chips at some long term puts?

I bought puts yesterday morning. Was up 50% by the end of day but now down to 50% of what I paid.

Mine expire 4/22 but I have more calls open at the moment anyways so if I had to choose between this going down or the market up I'll take a full loss on these puts (seems likely at the moment)

Re: Inside the longest Atlassian outage

#424

Post by a2800276 »

yebyen wrote: post #408
notatoad wrote: post #392

Earlier quoted context omitted.

No, I don't think that's disturbing. That's the point of backups - even when something is permanently and completely erased in the production database, it's still in the backup. Eventually it will get rotated out as the backups expire. Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.

I think that depends on what you mean by compliance. Some regulations require you to irreversibly destroy data when they prescribe the destruction of that data. That can mean as much as "you have to encrypt everything with a separate key, so that you can destroy the key for the given (say, personally identifiable) dataset making its retrieval irrecoverable" I'm not saying that's the particular compliance reason they…

"permanently delete" strongly suggests to me that it was the "medical and financial data" kind of compliance. If data can be restored, it's not permanently deleted. But this was a statement from the CEO, so words can have arbitrary meaning :)

Re: Inside the longest Atlassian outage

#425

Post by Delitio »

notatoad wrote: post #392
a2800276 wrote: post #389

Earlier quoted context omitted.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

No, I don't think that's disturbing. That's the point of backups - even when something is permanently and completely erased in the production database, it's still in the backup. Eventually it will get rotated out as the backups expire. Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.

Nope it's not ridiculous. If you are only allowed to store data for x month that's it.

It's your job to use technics which allow you to do this like using encryption on your backup and deleting the keys for it, for example.

Re: Inside the longest Atlassian outage

#426

Post by darkwater »

inopinatus wrote: post #290
seanwilson wrote: post #277

Earlier quoted context omitted.

> shard your customer databases 1:1 What are the downsides to this?

* makes it much harder to distribute your tables by any other factor, for whatever reason (usually performance, sometimes archival) * disaggregates data that the SaaS might be interested in querying/updating as an aggregate * not all ORM frameworks handle this case well, if at all * dumps are more than a single trivial command basically all your data operations gain an additional dimension of complexity, and you may…

All of your points (minus maybe the first one) should be "easily" solved/implemented in a company the size of Atlassian, and maybe there are newer costumers sharded like this already. IMO what happened in this case is basically tech debt that is now being paid with loooot of interests.

Re: Inside the longest Atlassian outage

#427

Post by miketria »

gtm1260 wrote: post #38
hu3 wrote: post #19

Earlier quoted context omitted.

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 cu…

Right? The way this reads it seems like one person set a flag incorrectly, something I'm sure we've all done numerous times. And there were no checks down the line to catch it.

Hi, this is Mike from Atlassian Engineering. You are right that the checks need to improve to reduce human error, but that's only half of it. I don't see this as human error though. It's a system error. We will be doing some work to make these kind of hard deletes impossible in our system.

Re: Inside the longest Atlassian outage

#428

Post by Kwpolska »

Alex3917 wrote: post #141
jacquesm wrote: post #124

Earlier quoted context omitted.

Did you continue as their customer after that?

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

> why would you ever delete someone's data when it's literally only 160 KB of text?

Compliance? The contract has expired, so there’s no legal basis for them to keep your data?

Re: Inside the longest Atlassian outage

#429

Post by elxr »

yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

If you're hosting your code on GitHub, then GitHub projects is definitely worth using.

Does everything I used to use Jira for, but feels more modern and lightweight. Also, it has dark mode.

Re: Inside the longest Atlassian outage

#430

Post by __app_dev__ »

eli wrote: post #80
radicaldreamer wrote: post #75

Earlier quoted context omitted.

It might be a good short opportunity... I imagine a lot of customers are kicking off their own internal process for migrating away from JIRA. By the time they actually do, it'll be at least a couple of quarters from now, which is when the customer hit will start materializing in quarterly results for the company. Maybe time to throw a few chips at some long term puts?

Aren't most customers in 12+ month contracts? A migration seems like it would take many months to select a new vendor and migrate regardless. Be careful about the date on those puts. It's pretty hard to out-think the market on this kind of stuff. I'd just as soon bet the other way: few customers will actually churn and in 6 months this won't really matter.

They might even get some new customers after people who never used it look at their site and offerings.

Disclaimer I have puts that expire 4/22 (purchased yesterday) so I hope they go down in the short term. Seems like a total loss now after being up 50% yesterday.

Post reply on HN