Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

541–550 of 772 posts

Re: Inside the longest Atlassian outage

#541

Post by theteapot »

iancarroll wrote: post #525
madmulita wrote: post #511

Earlier quoted context omitted.

They claim they test backups quarterly yet they don't have a procedure in place to restore the operation. We all know your backup is not tested until you restored everything successfully. This is not an engineering mistake, it is a flat out lie.

Well, their explanation makes sense. These are multi-tenant environments where not every tenant was affected; sensibly, the backups appear divided by environment, not tenant. You can’t blindly revert to an environment’s last backup in this scenario, although you’d think they would have done it before.

Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.

Re: Inside the longest Atlassian outage

#542

Post by kikki »

spookthesunset wrote: post #444
kikki wrote: post #359

Earlier quoted context omitted.

There are _many_ databases on each server, last I checked there was around 8 servers (or: "clusters") - and we have it so the traffic is somewhat evenly distributed across each server. There are reasonable capacity limits, and when servers get full we spin up a new one and start accepting new wikis there. I am not in OPS, and they do a lot of work behind the scenes to make this all run smoothly - but from an eng pers…

How do you update the schema on 300,000 databases?

It doesn't change often, if we do we just have large automated rollout plans - but we've done mass changes enough times there are good procedures around large DB migrations.

Re: Inside the longest Atlassian outage

#543

Post by theteapot »

Karrot_Kream wrote: post #537
syshum wrote: post #480

Earlier quoted context omitted.

sales never takes the blame. If anyone is fired it will be scapegoats in engineering once they have busted their ass to restore their reward will be the door

Why should sales take the blame when it's engineering's problem? If Sales promised a feature to a customer that was infeasible _then_ it should be Sales problem, but engineers made a mistake so only engineers can clean up the mistake.

What? Because sales interacts with customers and the customer doesn't give a F*$% who's fault it is or even know engineers exist.

Re: Inside the longest Atlassian outage

#545

Post by Karrot_Kream »

Animats wrote: post #472
beachy wrote: post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Rather, customers must stop using Atlassian cloud services.

This is a great opportunity to be an Atlassian competitor. I'm sure more than one business added Atlassian cloud services as a business risk in the meantime, even if they weren't affected.

Re: Inside the longest Atlassian outage

#546

Post by cerved »

jacquesm wrote: post #131
a-dub wrote: post #85

i hate deleting things. prefer flags that hide things instead (like a boolean deleted flag in an rdbms table). prevents data integrity issues in relational databases, makes debugging easier and prevents disasters. ideally also include a timestamp, both for bookkeeping and safe tools that only remove things that have been soft deleted for some time and are safe to delete without compromising integrity of anything that…

Better still: a field that registers at what date a record was supposedly marked as deleted. Because otherwise you still can't bulk recover from an error.

to bulk recover only x >= y ?

Re: Inside the longest Atlassian outage

#547

Post by Karrot_Kream »

theteapot wrote: post #543
Karrot_Kream wrote: post #537

Earlier quoted context omitted.

Why should sales take the blame when it's engineering's problem? If Sales promised a feature to a customer that was infeasible _then_ it should be Sales problem, but engineers made a mistake so only engineers can clean up the mistake.

What? Because sales interacts with customers and the customer doesn't give a F*$% who's fault it is or even know engineers exist.

This depends on your org structure. Sales does not focus on ongoing relationships at our company, instead we have dedicated account managers who handle relationships with large customers and have a general process to publish status updates during an outage. The outage status update process is guided by an internal outage management team and is a big deal; our C-suite and VPs immediately action on gaps managing status updates. We promise SLAs so managing expectations is key to hemorrhaging money from outages.

Without knowing Atlassian's org structure all I, an outsider, know is that engineering had an issue and remediation status communication has been lacking. If anyone should take "blame" it should be the bad engineering practices (usually a lack of funding or inability for upper management to prioritize engineering risk) which led to an outage of this magnitude and the status update mechanisms in place. Even then, I question how much better status communication would even get Atlassian given the sheer length of this outage, which is why I think the blame should ultimately land on engineering practices. Hopefully Atlassian learns from this but I'm afraid the damage is done. A SAAS system which takes this long to heal is unacceptable in 2022.

Re: Inside the longest Atlassian outage

#548

Post by cerved »

a-dub wrote: post #155
jacquesm wrote: post #131

Earlier quoted context omitted.

Better still: a field that registers at what date a record was supposedly marked as deleted. Because otherwise you still can't bulk recover from an error.

yep. but at least in the rdbms case, and probably in all cases, a flag (and an index on it) tends to be essential for query performance since the state of the flag will appear in most, if not all queries. that's okay though, queries that reference the timestamp can be slow since they're housekeeping.

if the goal is the ability to rollback a certain delete, it's probably better to make it a nullable fk on a delete event with a timestamp.

But a nullable timestamp could also be viable, just can't tell two different delete, at the same time, apart

Re: Inside the longest Atlassian outage

#549

Post by jamesfinlayson »

scottlamb wrote: post #504
beachy wrote: post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Is it standard for a RFP to have a long list of questions like this? I've never been involved in an RFP from either side. Is it standard to (in addition or instead) to have something more general/forward-looking like: how do you watch other providers' postmortems and apply the lessons to your own system? > - Can you restore data for a single customer, and if so, what is the RTO for that operation? If I were to aim so…

I work on a couple of reasonably small products but generally government customers have a long list of questions (mostly but not entirely about security).

Re: Inside the longest Atlassian outage

#550

Post by twistedpair »

drsim wrote: post #442
beachy wrote: post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

I know plenty of places (small/med startups) with "undelete" and "restore" account/data options built into their admin panels. Engineers shouldn't be flipping bits by hand, under duress.

I really wonder what these Altassian restore tools look like it if takes "hundreds of engineers across the company" to restore 400 accounts. Are backups siloed across many teams?

Post reply on HN