Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

twitter.com

Post reply on HN
Print view View original

81–90 of 260 posts

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#81

Post by daigoba66 »

laurent92 wrote: post #42
Nextgrid wrote: post #26

I bet they screwed up royally, deleted some data and are down to either rebuilding it from logs, caches or other side-effects, or using data recovery software on the storage drives (which might involve third-party companies). I can't see many other reasons why this should take 2 weeks.

Let me bet: Rebuilding from Jira email notifications. Yes, the diffs in the notifications.

I've actually done exactly that many years ago for a self-hosted Jira installation that didn't have any backups. You can bet we had backups with regular testing after that.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#82

Post by izacus »

justin_oaks wrote: post #64
xenadu02 wrote: post #29

Earlier quoted context omitted.

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

I agree that data must never be "deleted and forever gone" unless you've already been very sure about it a few time. But I would like to warn people about certain implementations of database "soft deletes" that I'm not a fan of. To be clear, I'm talking about the idea of having a "deleted" and/or a "date_deleted" column and using those columns in the WHERE clause to filter out rows that shouldn't be visible. That pat…

A "deleted" field type deletion is also how you get a massive fine from a GDPR agency when they find out that you're not actually deleting PII properly.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#83

Post by Nextgrid »

perlgeek wrote: post #66

So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…

My theory in my other comment is that they've deleted some data and are waiting on third-party data recovery specialists. That would explain the timescale.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#84

Post by uuyi »

laurent92 wrote: post #47
justin_oaks wrote: post #38

Earlier quoted context omitted.

There are two ways this can go: 1) This outage will get their organization to prioritize work such that it never happens again. 2) This outage is representative of a dysfunctional organization that can't prioritize work correctly. If you've been using Atlassian software for a while and are used to how they prioritize tickets then one of those options seems far more likely than the other.

> 1) This outage will get their organization to prioritize work such that it never happens again It has already happened in the past.

2012. https://www.theregister.com/2012/05/09/atlassian_cloud_stora...

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#85

Post by nvr219 »

polski-g wrote: post #74
rglover wrote: post #72

I love stuff like this happening, not for the schadenfreude but for the "what not to do" lessons. #3,198,191 Don't automate deletion scripts w/o sufficient recovery options.

Bigger lesson is: do not utilize cloud solutions. Your organization's competency floor is set to the least competent employee of the cloud provider.

I'd disagree with this lesson. Saying "do not utilize cloud solutions" period is nuts. Google and Microsoft are way better at email hosting and delivery than your on-prem server is unless you spend a ton more money on hardware and engineers to keep it up, which is simply not worth it for many companies. Dropbox is going to have better uptime and lower TCO than your self-hosted owncloud instance.

What I will say is it's important for the customer to HAVE THEIR OWN BACKUPS. Don't rely on the vendor - that's the lesson here. If you have all your stuff in AWS back that data up someplace that's not AWS, etc.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#86

Post by uuyi »

planb wrote: post #68

As the person responsible for running Jira and Confluence on premises at my employer I‘m looking forward for the next time one of their sales droids contacts me to make us move to their cloud services (despite me stating that we are not interested multiple times)…

I would love to listen to that call.

There’s nothing that makes me happier than the fearsome squealing noises that enterprise sales drones make when you drop the sales equivalent of a Paveway IV on their pitch.

My favourite one was running some software supply chain compliance software on itself and explaining how it was constructed on top of a CVE riddled garbage dump.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#87

Post by nvr219 »

WorldMaker wrote: post #65
whiteboardr wrote: post #34

Earlier quoted context omitted.

Honest question: Since Trello is part of Atlassian aswell - what are good, reliable and above all lightweight alternatives for managing projects without the “pseudo-agile” rabbit holes of functionality?

Microsoft Planner is included in most Microsoft 365 plans. Pretty much if you've got Teams, you've got Planner (and you can just add Planner as Tab in a Teams channel). At this point it has surprising feature parity with Trello.

Last time I checked (a few months ago), Planner still could not be backed up. Like, at all. If someone went in and deleted a whole bucket you can't recover it, not natively, and not with third party. So that's a big fat no from me.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#88

Post by mey »

Rantenki wrote: post #46

As someone who is impacted, this is obviously immensely frustrating. Worse, outside of "we have rebuilt functionality for over 35% of the users", I haven't seen any reports from the people who have ostensibly been recovered. Next, their published RTO is 6 hours, so obviously they must have done something that completely demolished their ability to use their standard recovery methods: https://www.atlassian.com/trust/s…

Don't have a way to verify this, since you know, but I don't believe we had this plugin installed/enabled, but are still caught in the blast.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#89

Post by nvr219 »

throwawayboise wrote: post #54
api wrote: post #2

It's funny that they recently killed self-hosted Jira. If you'd self-hosted you'd be fine.

> If you'd self-hosted you'd be fine. Maybe. But you're counting on your sysadmin(s), who are also managing dozens of other things, to keep up to speed on Jira and its quirks, and apply patches and new versions as they become available without missing any steps or screwing something up. On average, you're still probably better off having a company that knows the product also host it for you, but obviously they can ma…

> and the downside is that when they do it might affect all clients, not just one.

This is also potentially an upside. For example when us-east-1 went down recently, customers were somewhat understanding because it was "amazon's fault" and everyone was down - it was in the news, etc. If we ran our own data center and that went down, our customers would've just said "why did you morons roll your own data center instead of just using aws?"

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#90

Post by beachy »

yabones wrote: post #71
beachy wrote: post #49

Earlier quoted context omitted.

Why would you continue with that plan? You couldn't get a clearer warning signal.

To be quite honest, the problem is historical. We have over a decade of project plans, support tickets, change control logs, etc in our Jira instance. There's simply no painless way to export that into another product that will have approximately the same functionality and features. There's a few that come close, but all fall short of a drop-in replacement. The only options now are the $$$$$ "datacenter" license, mig…

I totally understandthat. But at the same time, a multi-week outage is really a sign of an org that simply does not have their shit together at all.

But the lack of transparency is the worst. Another post speculated that Atlassian has lost data, doesn't even have backups, and is re-creating it by munging their emails and diffing them to re-create history. I can't really imagine that's tue - but what if it is, and Atlassian is concealing things?

Post reply on HN