Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

111–120 of 772 posts

Re: Inside the longest Atlassian outage

#111

Earlier quoted context omitted.

I suppose that’s why you don’t combine a tazer and gun into 1 device with 2 triggers.

Instead, you make it with one trigger and a PRNG that decides which gets activated. Just hope you've chosen the right PRNG!!

I will then write a script calls your script with the PRNG of my choice: PRNG1 always returns "trigger 2", and PRNG2 always returns "trigger 1". This detail will be documented in Confluence.

Re: Inside the longest Atlassian outage

#112

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

Not being able to selectively restore data for a subset of users might also indicate that users are not fully isolated from each other, which is worrying for technical and nontechnical reasons.

There is nothing non-technical that matters. If we start acting like it does, we incredibly poor decisions that in fact have nothing to do with physical reality, and quickly arrive at unworkable technology.

Re: Inside the longest Atlassian outage

#113
post #85

i hate deleting things. prefer flags that hide things instead (like a boolean deleted flag in an rdbms table). prevents data integrity issues in relational databases, makes debugging easier and prevents disasters. ideally also include a timestamp, both for bookkeeping and safe tools that only remove things that have been soft deleted for some time and are safe to delete without compromising integrity of anything that…

The GDPR and various things have made companies more skittish in doing things this way, because they get scared. Perhaps an effective measure would be to create a key that encrypts a customer's data, and give them a copy of the key, and let them know that after a certain point your copy of the key will be deleted, and if they want a restore past that point they'll need to provide the key.

You may as well just delete it, then. I guarantee a high percentage of users won't save that key and be able to find it later. GH (edit: or similarly nerdy sites) might (might!) be able to get away with that, but as soon as part of your process is "give the user a cryptographic key" you've just guaranteed yourself a support nightmare, with normal users. It's why the only cryptographic person-to-person communication systems that've been broadly successful haven't involved keeping track of anything, and don't have a setup process more complex than "point camera at QR code".

Re: Inside the longest Atlassian outage

#114

Earlier quoted context omitted.

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

I have built multiple multi-tenancy platforms and I never create separate databases for each customer. If you have separate databases, it's almost impossible to run meaningful queries across all of them. That architectural choice creates far more headaches than it solves. Usually people end up with the split-database architecture when they want a quick retrofit for a system that wasn't designed with multiple tenants.…

I can't believe anyone would do separate databases.

Just wait until a migration doesn't run on 2 of your 400+ customer databases. Or multi-hour migrations.

Re: Inside the longest Atlassian outage

#115

Heh. We have a Confluence account. That no one uses. So we didn't notice.

You probably wouldn't if you weren't in the affected subset of customers who were. This wasn't a total outage, but rather it affected a group of users who had been running a legacy standalone app called "Insight – Asset Management".

Re: Inside the longest Atlassian outage

#117

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

> Are SLAs even real?

Tommy: Here's the way I see it, Ted. Guy puts a fancy guarantee on a box 'cause he wants you to fell all warm and toasty inside.

Ted Nelson: Yeah, makes a man feel good.

Ted Nelson: But why do they put a guarantee on the box?

Tommy: Because they know all they sold ya was a guaranteed piece of shit. That's all it is, isn't it? Hey, if you want me to take a dump in a box and mark it guaranteed, I will.

Re: Inside the longest Atlassian outage

#118
post #70
post #23

Earlier quoted context omitted.

> Communication gap. First, there was a communication gap between the team that requested the deactivation and the team that ran the deactivation. Instead of providing the IDs of the intended app being marked for deactivation, the team provided the IDs of the entire cloud site where the apps were to be deactivated. So what they are saying is that they are not testing scripts at some staging server before running them…

it seems that it worked as intended, thus they have a QA process. The problem was in the wrong IDs provided and I doubt that at their scale they have a staging environment that duplicates the customer data.

> I doubt that at their scale they have a staging environment that duplicates the customer data.

If there is no feasible way of replicating their production environment somewhere else, then there should be some sanity checks in place. Something like "if an abnormally high amount of customer sites go down during the script's execution, kill the script". This is a 20/20 hindsight approach though and if Atlassian engineers can't solve I doubt a random HN user like me can.

Re: Inside the longest Atlassian outage

#120
I suspect - pure speculation - that they can't restore the backups, because if they could then they could easily do this in a way that accounts affected could be restored selectively. In other words: test your backups, if you don't they won't be there for you when you need them.
Post reply on HN