I'm utterly shocked at how many people assume that "The Cloud" is both run by people infinitely more intelligent and more competent than they are and that everything there is backed up.
What a crock of shit.
Canon's cloud platform has lost users' files and can't restore them
341–350 of 419 posts
Re: Canon's cloud platform has lost users' files and can't restore them
#342Earlier quoted context omitted.
I've designed a storage system for an image storage app for a client, and I had an extensive spreadsheet with the cost and risks of various combinations of erasure coding and data centres, as well as other mitigations. So yes. But this data loss highlights that all too often it ends there, with calculating the risk of loss due to hardware or data centre failures. You need to also do the same in terms assessing how yo…
But was this framed as easy business from gambling with users artefacts ? or just the usual "shit happens" preliminary thinking ?
I'd guess in this case nobody had thought about it as a real risk. People are often woefully unaware of the risks of accidental effects of unnecessary privileged.
Re: Canon's cloud platform has lost users' files and can't restore them
#343Earlier quoted context omitted.
Reading their description of the problem, it wasn't that they lost one of their redundant servers, it was that their code that cleans up temporary storage accidentally was also cleaning up long-term storage. Increasing the redundancy of their storage wouldn't have helped.
If you have regular (daily, hourly) tape backups that are shipped off-site then you greatly mitigate the potential damage of software bugs. It sounds like they weren’t making backups and just treating failover replica servers as a “backup” (which they clearly aren’t).
This sounds like a case of a bug in their software uncovered a huge gaping hole in their architecture.
Put another way, while they seem to have fixed the immediate problem, no word on whether they are putting real BACKUP plans in place to avoid a similar problem happening and whipping out everything.
Re: Canon's cloud platform has lost users' files and can't restore them
#344Earlier quoted context omitted.
If memory serves, a class of Glacier storage (AWS) isn't practical to use in the EU because the inability to delete makes it impossible to serve "Right to Erasure" GDPR requests.
I see memory served me incorrectly! Apologies if I have spread misinformation. It was an offhand comment from an AWS rep a few years ago and things may have changed.
When GDPR passed there was a bit of a scramble to understand it, and people had all sorts of different interpretations that they were sharing.
Re: Canon's cloud platform has lost users' files and can't restore them
#345Earlier quoted context omitted.
Where I worked, weekly point in time backups were required. Those backups were put onto tape drives, those tape drives were set on a pallet and then driven by truck to an offline second location. IMO _that’s_ how it’s done properly.
When we used Iron Mountain at my last job, there were two interesting additional concerns: 1) That nobody in our company actually knew where they were physically stored (insider risk?), but 2) That we had assurance that the physical storage was far enough away that a physical disaster in our area wouldn't also touch where the offline storage was located. There's a concomitant jump in RTO if that's what you do this, b…
Definitely a commitment jump, and not always necessary. We ran our own data centers, off the topic of backups, and it was wild to me that they were so disaster proof. There were different power lines from different power companies coming in in case either of the power companies had an outage, and some giant diesel power generator that could last a long time while fully powering the data center.
Re: Canon's cloud platform has lost users' files and can't restore them
#346This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…
> It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones. Ironically my experience has been exactly the opposite. It's the demosaic-ed, fully developed copies of my photos that are larger and harder to preserve than the original RAWs. And these f…
For storing lossless rgb you might consider FLIF given you've demonstrated some flexibility in your setup, https://flif.info it supports 16-bit channels
Re: Canon's cloud platform has lost users' files and can't restore them
#347Earlier quoted context omitted.
This can be just calculated risk. Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option. Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be…
This. Outside of FAANG, Gov and finance, very few consumer facing companies have the budgets to engineer redundancies.
I suspect this is more a view from the trenches of "modern" SAS companies hosting their offerings on cloud providers charging a premium for storage+transit.
Because I can assure you, from the perspective of someone who worked for >10 years in a backup related part of the storage industry, our product penetration covered a huge portion of public companies, and untold numbers of non public businesses. Given that we were one of a half dozen or so similar products and our competitors also had non-insignificant footprints in places we didn't most companies were taking backup of their core systems very seriously.
And even then, today, there are dozens of online backup services. There are plans which can backup a TB or two of data a year off a NAS/etc and provide a read only historical snapshot for the price of a couple starbucks coffees a month (and a reasonable internet connection). And then there are the dozens of mom & pop places which just use two or three ~$100 USB disks on a rotation with windows backup/time machine/whatever.
So, no, lack of a proper backup (and recovery) strategy isn't really a question of cost. Its more a question of the competencies of the individuals charged with configuring/maintaining the IT systems.
And sure, if you have .5PB+ of data that needs backing up, a tape library, or replicating dedupe appliance, software license, etc is going to cost $$$$ but overwhelmingly when compared with the overall IT budget for machines+storage+administrators those devices rarely crack double digit percentages of the yearly IT budgets. Further, there are ways to lower those costs even more with a bit of hunting on the resale markets where picking up a three year old piece of equipment for a tiny fraction of its original cost isn't unusual.
Re: Canon's cloud platform has lost users' files and can't restore them
#348This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.
This can be just calculated risk. Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option. Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be…
[deleted]
Re: Canon's cloud platform has lost users' files and can't restore them
#349Earlier quoted context omitted.
But they did backup the original data - the article states that the problem was with deploying wrong cleanup code on the long-term storage (that was supposed to be only deployed for the short-term storage). How do you guard against that? You could store everything in backup storage & never delete it, but that has implications: - privacy - legal (GDPR) - cost etc. It's basically a "black swan" event, they designed the…
> privacy Encryption? > legal (GDPR) You dont have to delete offline backups...never heard of such a case > cost etc. Cheaper than loose your customer's data wich they trust you with. EDIT: If you can delete a backup it's not a backup.
Under GDPR "right to be forgotten", you have to delete backups, can't keep them forever - see https://ico.org.uk/for-organisations/guide-to-data-protectio...
There are alternatives (one of which is to encrypt the backed up data & just throw away the encryption keys instead) but of course those incur additional costs
> Cheaper than loose your customer's data wich they trust you with.
This is just opinion. And also it's heavily context-dependent(cost of losing customer data is not the same for all data or all businesses).
> EDIT: If you can delete a backup it's not a backup.
If it gets destroyed, is it a backup? How's that different from "delete"? Those were not intentionally deleted, they were accidentally destroyed. Can your backups be accidentally destroyed?
Re: Canon's cloud platform has lost users' files and can't restore them
#350Earlier quoted context omitted.
Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must…
I haven't looked at dropbox in a while, but they at least used to have an option for keeping revisions. It did come with an extra cost.
Unfortunately Dropbox's revision history isn't reliable enough for a backup. If you notice, their marketing carefully avoids use of the word "backup" despite the features seemingly implying it.
I had a client where a glitch in the Dropbox sync caused 300k+ files to be deleted when we added a new PC. Dropbox support was unable to successfully undo all the file changes, and I had to get a ticket with CS escalated to a special team to get everything restored. Even when it was finished, they could not give a guarantee that every single file deleted was restored.
Maybe things got better since they've launched Dropbox Rewind, but given that revision history has been a feature for years and it still didn't work right, I no longer trust them as a backup.