Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

701–710 of 772 posts

Re: Inside the longest Atlassian outage

#701

Post by max599 »

originalvichy wrote: post #461
pphysch wrote: post #346

Earlier quoted context omitted.

It seems like if you are going to pay for a bunch of SaaS seats AND a team of technicians/engineers for make it work, you might as well just do the latter and roll your own solutions... A lot of these SaaS are just glorified Rails apps with a patina of professional "security" and "reliability", and loads of extra junk that your co will never use.

Trust me, if someone could clone Jira and its functionality they would have done so already. Truth is that if you build one product for 20 years you have a giant lead in features. If all it took was having a Kanban board then Jira would have died years ago.

isn't it one of those "No one ever got fired for buying XXXXXX" type of situation?

Maybe i'm wrong, but the impression I had of Jira is that just like using sharepoint for file storage, the C-level people want it because they were told that's what big enterprise are using. And if it doesn't fit the need of the company and everyone hates it, they just blame the employees or lack of training.

Re: Inside the longest Atlassian outage

#702

Post by gorgoiler »

mdoms wrote: post #571
wly_cdgr wrote: post #566

Earlier quoted context omitted.

They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with

You seem to be implying that customer success is a customer support role. It isn't. Customer success is about helping customers get the greatest business value from your product. They are not going to be munging databases and wrangling backups. Support engineers (among others) do that work.

You’re talking about the outage.

They’re talking about the outage lasting for two weeks without any communication as to what’s going on.

Re: Inside the longest Atlassian outage

#703

Post by fletchowns »

fishnchips wrote: post #499
scottlamb wrote: post #99

Gmail had a vaguely similar outage years ago. [1] tl;dr: 1. Different root cause. There was a bug in a refactoring of gmail's storage layer (iirc a missing asterisk caused a pointer to an important bool to be set to null, rather than setting the bool to false), which slipped through code review, automated testing, and early test servers dedicated to the team, so it got rolled out to some fraction of real users. Onlin…

Funny enough, most of what we restored then was spam (ex gTape SRE, remember the outage).

Even though you are no longer there...I had a friend who recently had her gmail inbox mysteriously emptied, all emails seemingly permanently deleted. She paid for Google One to be able to talk to support, and they said that the data is gone. Do you know if there's a way to recover this data? She is quite heartbroken at all the attachments that she will never get to see again.

Re: Inside the longest Atlassian outage

#704

Post by donavanm »

a2800276 wrote: post #389
Alex3917 wrote: post #44

A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

> Anyone else find it disturbing that they are able to restore data that they deleted permanently for "compliance" reasons?

An overarching theme with these things is “legitimate business need” and “no indefinitely retained customer data. Having backups, system event logs, etc are all legitimate business needs. Based on the data type that business need may be days or years with things like financial and legal requirements.

Youre conflating permanent, immediate, and irrevocable. These are usually handled in different aspects. Think of accounts having multiple states like active, suspended, closed, terminated, purged. Some examples;

suspended: credentials/authnz immediately disabled, all data online, charges continue to accrue, can be restored in minutes.

Closed: credentials disabled, data online, processing stopped, charges stopped, may take manual intervention (hours) to return to active.

Terminated: creds & account irrevocably unavailable, online data deleted, offline data (backups) remains available.

Purged: all online and offline customer data irrevocably unavailable. This generally happens after a defined retention period for things like logs, backups, etc.

You can apply similar concepts to individual resources more granularly than the account.

Disclaimer: principal at AWS but the above is my own opinion/observation and does not represent my employer.

Re: Inside the longest Atlassian outage

#705

Post by alias_neo »

scottlamb wrote: post #504
beachy wrote: post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Is it standard for a RFP to have a long list of questions like this? I've never been involved in an RFP from either side. Is it standard to (in addition or instead) to have something more general/forward-looking like: how do you watch other providers' postmortems and apply the lessons to your own system? > - Can you restore data for a single customer, and if so, what is the RTO for that operation? If I were to aim so…

It's very common to have several pages of questions like this with particular customers, and though they may often come in to sales and marketing people, they'll contain highly technical or operational questions relating to a variety of things such as security algorithms, programming languages (type safety etc) all the way to disaster recovery.

Regarding bulk restore, a big customer doesn't care if you can restore all of your customers' data, they care if you can restore _their_ data, and fast, hence the question of "can you restore data for a single customer?".

Re: Inside the longest Atlassian outage

#706

Post by robertlagrant »

spookthesunset wrote: post #449
robertlagrant wrote: post #368

Earlier quoted context omitted.

You can write the code to do 1000 schema migrations, but the problem is if you've migrated 40% of them and hit an issue. What do?

One of the many reasons to put good constrains on fields and use referential integrity! If you don't let the database enforce data validity you are gonna get fucked at some point! source: every single place I've worked at that poo-poos referential integrity has a database that is full of bullshit that "the application code" never cleaned up Always use referential integrity. The people who are against it almost always…

That's one thing yes. What if there's a transient network error, or the DB runs out of memory, and now you have some data in an old state and some in a new.

You're lecturing about table design. I'm talking about more general transactionality over any errors.

Re: Inside the longest Atlassian outage

#707

Post by bzzzt »

JohnBooty wrote: post #612
jimbokun wrote: post #596

Earlier quoted context omitted.

How do you implement per tenant backups? Not every db system cleanly separates where each tenant’s data is stored.

I don't have a ton of experience doing this, but with a lot of multitenant you just give each tenant their own database in the first place. It solves multiple classes of problems. On a single Postgres instance you can (at least theoretically) have 4 billion databases per instance.

It also creates multiple classes of problems. There are known issues with PostgreSQL's handling of databases containing tens of thousands of tables. (you will need a disproportionate amount of memory to handle that use case on a busy db server). Using a single database with tenant isolation by a discriminator key (preferably enforced by row level security) is a lot more efficient.

Re: Inside the longest Atlassian outage

#708

Post by Cthulhu_ »

It kinda reads like their user's data is not separated very cleanly; I've never worked at a SaaS before, but reading this, especially given the size of some customers, I'd want each customer to have their own independent instance, with its own backup pipeline. I was thinking of "just" giving them their own database, but there's been plenty of instances where authentication got botched allowing one user to see another user's data; this should be impossible if things are running on their own instances.

Note that I'm pretty naïve and armchair on this subject, I'll see myself out.

Re: Inside the longest Atlassian outage

#709

Post by dx034 »

beachy wrote: post #693
dx034 wrote: post #689

Earlier quoted context omitted.

I still don't get why they didn't separate clients on a database level. Sure, put many clients on one database server to save resources. But why not use different databases? They cost nothing and provide perfect separation. It also drastically lowers the attack surface as you can set all permissions via database software. And if they had done that, this would've never been a multi-day outage. If Jira was a product us…

There are lots of downsides to doing absolute partitioning of tenants (along with lots of upsides as you point out). Really annoying things that slash your velocity. You can't easily run pan-customer queries, can't aggregate data. Ironically too using a single database makes full backup and restore much easier.

Wouldn't you extract data anyway to another system for analytics? Running analytics queries on production databases seems a bit risky in any setting?
Post reply on HN