Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

twitter.com

Post reply on HN
Print view View original

111–120 of 260 posts

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#111

Post by hyperman1 »

tetsusaiga wrote: post #97
hyperman1 wrote: post #62

Earlier quoted context omitted.

You don't even need a bug. Just a wrong system clock. We had a few windows laptops where something caused them to time travel to 8000 years in the future. Then, they'd slowly spend a few hours deleting every local profile, as nobody had logged in to them for 8000 years. Then, they'd do something to their time zone database and travel back 8000 years. When they started the process, it was unstoppable. Trying to modify…

that is almost cartoonishly nightmarish

It happened specifically to 1 type of laptop and we only had about 30 of them. So we pulled all of them out of roulation. Then covid struck, so I reformatted most of them with Debian and we gave them away for home schooling. I wonder if I managed to linuxify some kid in the process.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#112

Post by m0llusk »

perlgeek wrote: post #66

So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…

If you are not regularly testing restores then you don't have backups.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#113

Post by dylan604 »

k8sToGo wrote: post #98
bdefore wrote: post #78

Engineers, do yourself a favor and add to your CV/resume: I will not work for companies that use Atlassian software.

So like 90% of the companies?

Yeah, it's not really a good piece of advice. If anyone actually reads that bit of information, they'll just shrug and toss the resume in the bin and move to one of the other hundreds of applicants.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#114

Post by [deleted] »

Rantenki wrote: post #46

As someone who is impacted, this is obviously immensely frustrating. Worse, outside of "we have rebuilt functionality for over 35% of the users", I haven't seen any reports from the people who have ostensibly been recovered. Next, their published RTO is 6 hours, so obviously they must have done something that completely demolished their ability to use their standard recovery methods: https://www.atlassian.com/trust/s…

[deleted]

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#115

Post by mh- »

ivanvas wrote: post #107

Looking at the company financials and timing, the wooden headquarters might have to wait. I don't envy engineers there right now, some dpts. Stay strong, don't burn out! https://architectureau.com/articles/worlds-tallest-hybrid-ti...

Wow, those renderings are gorgeous. (If you click the top photo, there's a gallery of 5 more you can scroll through)

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#116

Post by davio »

danuker wrote: post #43
xenadu02 wrote: post #29

Earlier quoted context omitted.

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

> or bad SQL WHERE clause Tip: Begin an SQL session with BEGIN TRANSACTION; at the end you can either COMMIT or ROLLBACK.

I always liked LIMIT when doing dangerous stuff in prod

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#117

Post by m0llusk »

xenadu02 wrote: post #29
btgeekboy wrote: post #10

Earlier quoted context omitted.

Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

This is also true for people cleaning house or going minimal. Put it in a well labeled box and tuck it away somewhere. If it matters then you'll fetch it, otherwise just chuck it when the box gets in the way.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#118

Post by RL_Quine »

jlbooker wrote: post #93
perlgeek wrote: post #66

So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…

Restore from off-site tape backup. The kind of service where you ship them ~dozen new tapes in a lockbox each week and they ship you the oldest dozen back. It's supposed to be the "if all of a data centers happen to burn to ashes simultaneously" option. If you say "give us all of our tapes, asap" and then have some pour souls swapping them out as fast as the data can be read... it would probably take a few weeks.

Tapes are not that slow, 360MB/s per drive, and on a large scale swapping them is completely automatic.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#120

Post by oldshatterhand »

senorsmile wrote: post #99
planb wrote: post #68

As the person responsible for running Jira and Confluence on premises at my employer I‘m looking forward for the next time one of their sales droids contacts me to make us move to their cloud services (despite me stating that we are not interested multiple times)…

They are discontinuing on prem after 2022 though. What is your plan if not to migrate to their SaaS offering?

Switching platforms
Post reply on HN