Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

611–620 of 772 posts

Re: Inside the longest Atlassian outage

#611

Post by PragmaticPulp »

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

> Actually reading a bit more, it seems like their customer team was partying in Las Vegas instead of taking care of business: https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22

This was a conference they hosted, not just some Atlassian team members partying in Vegas: https://events.atlassian.com/team22

Thousands of Atlassian customers bought tickets, flights, and hotels for the event. It's unreasonable to suggest that they should cancel it all days before the event.

> Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061

She responded that it was a scheduled post. This is normal on LinkedIn: Marketers write a lot of posts all at once and schedule them to come out periodically. They're not literally sitting down in front of LinkedIn and typing out their daily puff piece.

Please, let's drop the pitchforks. It's not a good look for us on HN, especially when the facts are being completely misinterpreted.

Re: Inside the longest Atlassian outage

#612

Post by JohnBooty »

jimbokun wrote: post #596
theteapot wrote: post #541

Earlier quoted context omitted.

Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.

How do you implement per tenant backups? Not every db system cleanly separates where each tenant’s data is stored.

I don't have a ton of experience doing this, but with a lot of multitenant you just give each tenant their own database in the first place. It solves multiple classes of problems.

On a single Postgres instance you can (at least theoretically) have 4 billion databases per instance.

Re: Inside the longest Atlassian outage

#613

Post by dimgl »

yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

I've used Linear and Shortcut (formerly Clubhouse). I was a huge Shortcut proponent, but there were a couple of concepts that weren't fully fleshed out.

Linear has none of these issues. I've been super impressed with it.

Re: Inside the longest Atlassian outage

#614

Post by nijave »

R0ger wrote: post #13

I guess this is wake call for the people rushing to SaaS solutions.

On prem is worse in some cases. If you don't have access to the source code or ability to modify it, you're still at the mercy of someone else and now there's likely additional hoops to jump through.

If your Oracle DB or Cisco router has a software big, you can always restore/rebuild but that doesn't guarantee you won't hit it again and in both cases you're still at the mercy of the company producing it.

Even if you're on OSS are you able to fix a data corruption bug yourself?

You get more control over maintenance windows and backups, but it doesn't automatically guarantee better uptime.

Re: Inside the longest Atlassian outage

#615

Post by dodobirdlord »

mdoms wrote: post #571
wly_cdgr wrote: post #566

Earlier quoted context omitted.

They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with

You seem to be implying that customer success is a customer support role. It isn't. Customer success is about helping customers get the greatest business value from your product. They are not going to be munging databases and wrangling backups. Support engineers (among others) do that work.

Assuming that the "Global Head of Customer Success" is a leadership role that covers support engineering teams (no clue, but it appears to be some sort of executive position), then it's sorta beside the point whether or not the person is personally munging databases and wrangling backups. If an organization fails it is the fault of its leadership. The buck has to stop somewhere. Whether the "root cause" was risky engineering practices, careless employees, or just back luck, the blame lies with leadership, who should have established safer practices, not hired careless employees, and had a plan to mitigate the unlucky circumstance.

Re: Inside the longest Atlassian outage

#617

Post by nijave »

escot wrote: post #284

When doing bulk deletes like this what safe guards do you put in place, other than testing the script up/down in another environment, turning off app servers etc (which Im guessing they did not do)?

Canary deploys i.e. start with a couple customers and do manual validations, wait a little bit of time (maybe a few days) before incrementally rolling it out to larger amounts of customers.

It's not clear I'd the issue affected all tenants where the script ran--which it sounds like it did. It wouldn't be as effective if it only effected certain tenants (maybe with a specific config)

Re: Inside the longest Atlassian outage

#618

Post by Rantenki »

I've repeatedly asked Atlassian if:

1. They can confirm that they have backups of our data (about a thousand stories, substantial confluence, opsgenie history, and three service desks).

2. Will our integrations, configuration, and customizations also be recovered, or will we need to rebuild those once our data is recovered?

I have received no response, and no human is even willing to acknowledge those questions. The service desk staff ignore them as if I never asked. Repeatedly.

Also, I've been asking around, and haven't been able to find a single story from somebody that can confirm that they were down, who has had their data recovered.

Re: Inside the longest Atlassian outage

#619

Post by Rantenki »

miketria wrote: post #396
bluedino wrote: post #145

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…

Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.

Hey Mike; Not dumping on you personally, but the RTO claims to be 6 hours. I can understand that being a target, but we're at 32X that RTO target, with a communicated target date of another 12 or so days IIRC. That's literally two orders of magnitude longer than the RTO. I don't think any rational person would take that document seriously at this point.

I'll also ask (since nobody else has answered, I may as well ask you as well):

1. Are the customers actually being restored from backups (and additionally, by a standard process)?

2. Will the recovery also include our integrations, API keys, configuration and customization?

Re: Inside the longest Atlassian outage

#620

Post by [deleted] »

madmulita wrote: post #511
kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

They claim they test backups quarterly yet they don't have a procedure in place to restore the operation. We all know your backup is not tested until you restored everything successfully. This is not an engineering mistake, it is a flat out lie.

[deleted]
Post reply on HN