Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

twitter.com

Post reply on HN
Print view View original

41–50 of 260 posts

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#41

Post by jdbernard »

bpp wrote: post #33
xenadu02 wrote: post #29

Earlier quoted context omitted.

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

Works until you get a bug in the deletion job. I've seen exactly this happen.

Yeah, the idea is that by expecting the deletion logic you can make it simpler and more rigorously tested than regularly changing business logic or application code.

If you organizationally cannot prioritize quality then nothing can help you.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#42

Post by laurent92 »

Nextgrid wrote: post #26

I bet they screwed up royally, deleted some data and are down to either rebuilding it from logs, caches or other side-effects, or using data recovery software on the storage drives (which might involve third-party companies). I can't see many other reasons why this should take 2 weeks.

Let me bet: Rebuilding from Jira email notifications. Yes, the diffs in the notifications.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#43

Post by danuker »

xenadu02 wrote: post #29
btgeekboy wrote: post #10

Earlier quoted context omitted.

Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

> or bad SQL WHERE clause

Tip: Begin an SQL session with BEGIN TRANSACTION; at the end you can either COMMIT or ROLLBACK.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#44

Post by lazide »

schrismartin wrote: post #39
xenadu02 wrote: post #29

Earlier quoted context omitted.

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

How does that work around GDPR and other "Right to be forgotten" legislation? Aren't we required to hard-delete this kind of data?

You are within a certain period of time, not ‘instantly’ (depending on the exact situation you are referring to). The script could take that into account (using a shorter period of time or the like)

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#45

Post by danuker »

Nextgrid wrote: post #24
encryptluks2 wrote: post #14

I'm honestly surprised their stock hasn't plummeted yet. There is no way I could ever invest in a company that was this terrible.

If technical competency had any bearing on stock prices they should've been at 0 since long ago. Their stock price is tied to the amount of clueless/shitty companies that will still cling onto their products regardless of what happens, and I don't think this incident is going to change much.

If the market did not react instantly, it will certainly do so at the next financial report.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#46

Post by Rantenki »

As someone who is impacted, this is obviously immensely frustrating.

Worse, outside of "we have rebuilt functionality for over 35% of the users", I haven't seen any reports from the people who have ostensibly been recovered.

Next, their published RTO is 6 hours, so obviously they must have done something that completely demolished their ability to use their standard recovery methods: https://www.atlassian.com/trust/security/data-management

Finally, there have been some hints that this is related to the decommissioning of a plugin product Atlassian recently acquired (Insight asset management) which is only really useful to large organizations. I suspect that the "0.18% impacted" number is relative to ALL users of Atlassian, including free/limited accounts, and that the percentage of large/serious organizations who are impacted (and who would have a use for an asset management product), is much higher.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#47

Post by laurent92 »

justin_oaks wrote: post #38
yabones wrote: post #22

We're mere weeks aware from migrating to their could platform after the self-hosted rugpull. This really doesn't give me confidence in their ability to not break my stuff.

There are two ways this can go: 1) This outage will get their organization to prioritize work such that it never happens again. 2) This outage is representative of a dysfunctional organization that can't prioritize work correctly. If you've been using Atlassian software for a while and are used to how they prioritize tickets then one of those options seems far more likely than the other.

> 1) This outage will get their organization to prioritize work such that it never happens again

It has already happened in the past.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#49

Post by beachy »

yabones wrote: post #22

We're mere weeks aware from migrating to their could platform after the self-hosted rugpull. This really doesn't give me confidence in their ability to not break my stuff.

Why would you continue with that plan? You couldn't get a clearer warning signal.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#50

Post by vbezhenar »

xenadu02 wrote: post #29
btgeekboy wrote: post #10

Earlier quoted context omitted.

Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

What's wrong with restoring from backups? This is one reason they exist after all. I don't think that making a mistake in delete statement is something you would do every week.
Post reply on HN