[deleted]
Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
241–250 of 260 posts
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#242Earlier quoted context omitted.
Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...
Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…
> Never ever ever be in a situation where a rogue script or bad SQL WHERE clause means restoring from backups.
As a second step, restore from the backup at a set frequency. This would force orgs to automate and optimize not just the backup flow but also the restore flow. Tear-down and restore entire systems from backups. Of course, doing so enormously adds to the cost, but when there's an outage, it will pay itself over.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#243Earlier quoted context omitted.
The cloud is just someone else's servers. 3-2-1-0 Applies to all data, at all time, in all places
Normally it's 3-2-1, what does the 0 represent here? 3 copies 2 different formats (ex. HDD, cloud) 1 offsite 0 lost data? :D
0 Errors.
Technically alot of Backup Planning has moved to 3-2-1-1-0
3 Copies
2 Different Media
1 - Offsite
1 - Offline / Immutable
0 - Errors from Verification Tests
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#244Earlier quoted context omitted.
Normally it's 3-2-1, what does the 0 represent here? 3 copies 2 different formats (ex. HDD, cloud) 1 offsite 0 lost data? :D
>2 different formats (ex. HDD, cloud) That's not what "format" means, it's more like DB-Dump and DB-VM-Dump, or pure Files and VM-Dump, or something like restic-repo and rsync(pure files).
2 Media, but the HDD / Cloud is correct
The idea is to not have the backups stored on the same hardware or even same type of hardware. Same hardware is obvious but same type of hardware is listed because if a manufacturing defect or a known vulnerability is present it would make all of your backups at risk. So you want to have backups stored on 2 desperate types of storage media. HDD and Tape, or Cloud etc...
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#245Earlier quoted context omitted.
>2 different formats (ex. HDD, cloud) That's not what "format" means, it's more like DB-Dump and DB-VM-Dump, or pure Files and VM-Dump, or something like restic-repo and rsync(pure files).
2 Media, but the HDD / Cloud is correct The idea is to not have the backups stored on the same hardware or even same type of hardware. Same hardware is obvious but same type of hardware is listed because if a manufacturing defect or a known vulnerability is present it would make all of your backups at risk. So you want to have backups stored on 2 desperate types of storage media. HDD and Tape, or Cloud etc...
for me it was always 2 different formats (db-dump, vm-dump) because i trust (good) storage-media more then backup-software, for example old veeam-backups cannot be restored with new version, old veeam-software runs not on new esxi etc...
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#246So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…
If you would permanently delete data for selected customers from a large multitenant system, it could actually take some time to restore it - even with proper backups. You can’t just do a full recovery as that would mess those customers who were not affected (it likely takes time to notice the mistake - others have continued to use the system). You might need to write some tools to migrate the data from backups. Also…
I was in a similar situation many years ago (different ticket software though). What we did was spinning up a spare server with the backup data and script an extraction/injection tool to populate the production multitenant sass.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#247Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#248Earlier quoted context omitted.
2 Media, but the HDD / Cloud is correct The idea is to not have the backups stored on the same hardware or even same type of hardware. Same hardware is obvious but same type of hardware is listed because if a manufacturing defect or a known vulnerability is present it would make all of your backups at risk. So you want to have backups stored on 2 desperate types of storage media. HDD and Tape, or Cloud etc...
for me it was always 2 different formats (db-dump, vm-dump) because i trust (good) storage-media more then backup-software, for example old veeam-backups cannot be restored with new version, old veeam-software runs not on new esxi etc...
Veeam backup chains are backwards compatible with older version of veeam so I am not sure where you get that.
As far as running veeam on esxi, you would need to elaborate more on that
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#249Earlier quoted context omitted.
What's wrong with restoring from backups? This is one reason they exist after all. I don't think that making a mistake in delete statement is something you would do every week.
This whole debacle is Exhibit A. They’re lucky they have a sound backup strategy in place, and that the amount of data lost is appearing to be minimal.
Yea, and this whole thread is very informative for a non-db specialist like me!
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#250Earlier quoted context omitted.
that is almost cartoonishly nightmarish
It happened specifically to 1 type of laptop and we only had about 30 of them. So we pulled all of them out of roulation. Then covid struck, so I reformatted most of them with Debian and we gave them away for home schooling. I wonder if I managed to linuxify some kid in the process.
Oh man, so this was relatively recent no less. Hopefully you did!