Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

401–410 of 772 posts

Re: Inside the longest Atlassian outage

#401

Post by mountainriver »

NineStarPoint wrote: post #187
jmondi wrote: post #110

What blow's my mind is that Atlassian stock has barely taken a hit...

Unless their revenue takes a long term hit over the outage, no reason for the stock market to care. There isn't news of people actually planning to stop using Atlassian products over this. The only direct consequence is going to be the one time payment of SLA credits. So I guess the part I find surprising is how little impact this looks like it will have on people using their products more so than I am that the stock…

Yeah Atlassian is a corporate leech, you don’t get away that easy

Re: Inside the longest Atlassian outage

#402

Post by snark42 »

crummy wrote: post #391
originalvichy wrote: post #221

Earlier quoted context omitted.

I am biased but I can tell you what works best for mid-large companies: having a solution provider. Basically a partner that hosts and maintains the instance and has enough Atlassian certified people to help you with any question so that you will never have to hire people to just maintain the beasts or tell you about features, tricks or plugins that could solve problem X. Experienced people hosting and tuning Atlassi…

I can't see the difference between a "solution provider" that hosts your Jira and just getting Atlassian to do it. What's stopping the solution provider from accidentally running a script that deletes some customer's files and struggling to do a partial backup restore?

I would assume the MSP is running a dedicated instance and can do a full/backup restore just for the user they're supporting.

If it's some multi-tenant solution it's no better.

Re: Inside the longest Atlassian outage

#403

Post by adamc »

duxup wrote: post #8
nitinagg wrote: post #2

Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.

As this continues I suspect that this might be one of the few times where a lack of transparency / good communication really ... might not be better or worse because the situation is so bad that transparency would be horrible just the same. Granted that's how all lies start / what sometimes people assume and they're wrong but ... maybe this is that time? Maybe it is in fact so bad that honesty would be a push or wors…

If so, that itself would be a huge red flag for dealing with Atlassian.

Re: Inside the longest Atlassian outage

#404

Post by miketria »

nitinagg wrote: post #2

Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.

Hi, this is Mike from Atlassian Engineering. You are right the communications from us have not lived up to our standard. We will focus on this specifically once we restore service and get the post incident review out there. More details here: https://www.atlassian.com/engineering/april-2022-outage-upda...

Re: Inside the longest Atlassian outage

#405

Post by deckard1 »

inopinatus wrote: post #290
seanwilson wrote: post #277

Earlier quoted context omitted.

> shard your customer databases 1:1 What are the downsides to this?

* makes it much harder to distribute your tables by any other factor, for whatever reason (usually performance, sometimes archival) * disaggregates data that the SaaS might be interested in querying/updating as an aggregate * not all ORM frameworks handle this case well, if at all * dumps are more than a single trivial command basically all your data operations gain an additional dimension of complexity, and you may…

> not all ORM frameworks handle this case well, if at all

typically this is probably for internal reporting/metrics. But yeah, a custom script with direct SQL is in order. Personally my opinion is avoid ORM at all costs. Never seen a benefit that wasn't trivially done in SQL, and the downsides are incredibly painful.

The big downside of sharding out, per customer, is that's a lot of databases to migrate on upgrades. Or rollback if shit hits the fan.

The upside? You can have customers on different versions of your app if you really wanted to do such a thing.

In any case, proper tooling goes a long way to making it the difference between wonderfully manageable and torturous nightmare. Think idempotent backup scripts that are capable of failing at any time and resuming where they died, etc.

Re: Inside the longest Atlassian outage

#408

Post by yebyen »

notatoad wrote: post #392
a2800276 wrote: post #389

Earlier quoted context omitted.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

No, I don't think that's disturbing. That's the point of backups - even when something is permanently and completely erased in the production database, it's still in the backup. Eventually it will get rotated out as the backups expire. Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.

I think that depends on what you mean by compliance. Some regulations require you to irreversibly destroy data when they prescribe the destruction of that data.

That can mean as much as "you have to encrypt everything with a separate key, so that you can destroy the key for the given (say, personally identifiable) dataset making its retrieval irrecoverable"

I'm not saying that's the particular compliance reason they had here, or that the analysis you're giving is wrong, either. There is an interpretation where either of these ideas could be the correct one.

Re: Inside the longest Atlassian outage

#409

Post by dangrossman »

nemo1618 wrote: post #399
Alex3917 wrote: post #141

Earlier quoted context omitted.

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

After I met my now-fiancée on OkCupid, I deactivated my profile, turned off notifications and forgot about it for a while. A while later, I thought it be nice to revisit the first messages we sent to each other, only to find that... OkCupid had deleted both of our accounts. They didn't give me any advance warning, either, because I turned off notifications, remember? :^) I'm still kinda salty about it. I understand w…

With GDPR, privacy regulations and data breach regulations sweeping the globe, holding onto unnecessary data is a huge liability. Getting rid of data you no longer have clear consent to store, or which you're unlikely to have a clear business need to continue storing, is a sign of a good company these days.

Re: Inside the longest Atlassian outage

#410

Post by indymike »

omoikane wrote: post #146
indymike wrote: post #112

Earlier quoted context omitted.

There is nothing non-technical that matters. If we start acting like it does, we incredibly poor decisions that in fact have nothing to do with physical reality, and quickly arrive at unworkable technology.

Non-technical reasons include "legal" and "compliance", which often matters a fair bit. I am not disagreeing that non-technical requirements occasionally lead to poor decisions, for some value of poor.

I live is a state that once tried to legislate that pi = 3.15. The results were tragic, and the attempt to legislate a ratio was a failure, much like systems created by regulation and laws often are. Math is much less forgiving than legal prose. Making database decisions based on criteria that don't make any engineering sense one way or the other is not far off from legislating the value of PI.
Post reply on HN