Live data from Hacker News

Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

twitter.com

181–190 of 260 posts

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#181
post #177
post #167

Earlier quoted context omitted.

this is a scary thought. I need to start being more aggressive about backing things up that are "in the cloud."

The cloud is just someone else's servers. 3-2-1-0 Applies to all data, at all time, in all places

Normally it's 3-2-1, what does the 0 represent here?

3 copies

2 different formats (ex. HDD, cloud)

1 offsite

0 lost data? :D

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#182
post #63

Earlier quoted context omitted.

Same situation. Absolutely no one even knew this was happening and doesn’t give a shit now because it’s a project death march. JIRA as a whole has been a fucking shit show of a product over the last decade even on-prem.

I will say that back when Bugzilla was it, JIRA rocked. It was amazing the new power you had and the functionality it provided. We self-hostd JIRA from 2008ish to 2014ish. (memory is fading on exact dates) By the time we decided to stop using JIRA, we fracking hated JIRA and would never return. Since then, GitHub Issues, Trello, Clubhouse (neé Shortcut) all provide less friction in day to day use. As an Enterprise, I…

Not trying to nitpick, just for your information

> Clubhouse (neé Shortcut)

It's the opposite: Shortcut (née Clubhouse), as "née" means "born", so it's the name it had at birth, the old name

The more you know!

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#183

So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…

If you would permanently delete data for selected customers from a large multitenant system, it could actually take some time to restore it - even with proper backups. You can’t just do a full recovery as that would mess those customers who were not affected (it likely takes time to notice the mistake - others have continued to use the system). You might need to write some tools to migrate the data from backups. Also…

What a nightmare situation. Makes you wonder if some kind of 1 database per customer setup would be preferable here since you could restore only affected customers.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#184

Earlier quoted context omitted.

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

I agree that data must never be "deleted and forever gone" unless you've already been very sure about it a few time. But I would like to warn people about certain implementations of database "soft deletes" that I'm not a fan of. To be clear, I'm talking about the idea of having a "deleted" and/or a "date_deleted" column and using those columns in the WHERE clause to filter out rows that shouldn't be visible. That pat…

>I have dealt with the rare "Oops! I deleted important data!" by restoring from backups and it has worked fine.

Usually this goes along with "Oh and the other team did some important work at the same time" so you can't just restore a backup. You either tell them to deal with it or start writing custom scripts to copy out only the data you want to restore.

A more sane solution would be soft delete for x days and after that it becomes a real delete.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#185

Earlier quoted context omitted.

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

What's wrong with restoring from backups? This is one reason they exist after all. I don't think that making a mistake in delete statement is something you would do every week.

Because you lose all the work done since the issue happened. It's very rarely acceptable to just give up and do a full backup restore unless literally everything is gone. If there was just some bug that cause partial issues, you have to find some other way to fix it.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#186

Earlier quoted context omitted.

How does that work around GDPR and other "Right to be forgotten" legislation? Aren't we required to hard-delete this kind of data?

What is hard deletion? You can restore rows from database files before vacuum runs. You can often restore data from disk sectors. Some people say SSD can remap sectors under your chair and you won't even know that your deleted data is there.

The law isn't a technical specification. You have to follow the spirit of the law. A soft deleted_at timestamp wouldn't be following the law in good faith. Having some data stuck in an unmapped section of an ssd would be within the spirit.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#187

So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…

If you would permanently delete data for selected customers from a large multitenant system, it could actually take some time to restore it - even with proper backups. You can’t just do a full recovery as that would mess those customers who were not affected (it likely takes time to notice the mistake - others have continued to use the system). You might need to write some tools to migrate the data from backups. Also…

how would that require hundreds of engineers though? Would one not build a script and then just run it for each customer or each dept build a script? I honestly have no idea, never been in that sort of recovery situation before.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#188
post #85

Earlier quoted context omitted.

I'd disagree with this lesson. Saying "do not utilize cloud solutions" period is nuts. Google and Microsoft are way better at email hosting and delivery than your on-prem server is unless you spend a ton more money on hardware and engineers to keep it up, which is simply not worth it for many companies. Dropbox is going to have better uptime and lower TCO than your self-hosted owncloud instance. What I will say is it…

There's something humorous when better uptime is cited as a reason to use cloud software instead of on-prem in a thread about a major cloud outage.

On prem outages and data loss happens constantly. Much more than cloud hosted issues. They just affect a smaller group each time. It's like how people view the countless car crash deaths as non issue but freak out over a rare train crash.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#189
post #177

Earlier quoted context omitted.

The cloud is just someone else's servers. 3-2-1-0 Applies to all data, at all time, in all places

Normally it's 3-2-1, what does the 0 represent here? 3 copies 2 different formats (ex. HDD, cloud) 1 offsite 0 lost data? :D

0 data leaving the system unencrypted would be my formulation

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#190
post #67

Earlier quoted context omitted.

Bonus tip: don't even type COMMIT until after you've run a few select queries to verify the data has in fact changed how you expected it to change.

super duper bonus tip: Don't execute SQL in a production environment that you just typed out on the fly.

I fucked up once and lost 2 hours of customer data. I was so lucky we were a small startup and had daily backups AND the backup was only 2 hours old. I would have been royally fucked. Never making that mistake again.

Always use a copy of prod on a staging server and run your queries there for testing.

Post reply on HN