Canon's cloud platform has lost users' files and can't restore them • Hacker News Board

Re: Canon's cloud platform has lost users' files and can't restore them

#411

Post by virgilp » Thu, Aug 20, 2020, 5:55 AM UTC

nix23 wrote: post #393

virgilp wrote: post #349

Earlier quoted context omitted.

Under GDPR "right to be forgotten", you have to delete backups, can't keep them forever - see https://ico.org.uk/for-organisations/guide-to-data-protectio... There are alternatives (one of which is to encrypt the backed up data & just throw away the encryption keys instead) but of course those incur additional costs > Cheaper than loose your customer's data wich they trust you with. This is just opinion. And also it'…

>you have to delete backups, can't keep them forever >>The key issue is to put the backup data ‘beyond use’, even if it cannot be immediately overwritten. It's clear that you cannot delete WORM-tapen with many other data on it. >This is just opinion. Not for a file hosting service, the customer trusts you to do your job. >If it gets destroyed, is it a backup? Destroyed means physikal interaction, since it is not poss…

Why did you stop quoting after "immediately overwritten" ?

> ie that the backup is simply held on your systems until it is replaced in line with an established schedule

Again, you can't keep backups forever, you are _required_ to delete them.

Also, not sure why you keep you backups in the bank (do you, really?) but even banks can suffer from fire/arson, explosion (accidental or not), or a number of other natural disasters (earthquake, flooding, etc).

[later edit] Also, sort of by definition, in all businesses customers trust you to do your job. Can you give an example of a business where customers don't typically trust you to do your job?

Re: Canon's cloud platform has lost users' files and can't restore them

#412

Post by deepstack » Thu, Aug 20, 2020, 9:49 AM UTC

chongli wrote: post #202

jefftk wrote: post #189

Earlier quoted context omitted.

Reading their description of the problem, it wasn't that they lost one of their redundant servers, it was that their code that cleans up temporary storage accidentally was also cleaning up long-term storage. Increasing the redundancy of their storage wouldn't have helped.

If you have regular (daily, hourly) tape backups that are shipped off-site then you greatly mitigate the potential damage of software bugs. It sounds like they weren’t making backups and just treating failover replica servers as a “backup” (which they clearly aren’t).

Always have a back that is write once! Storage are cheap now days. If this is your core business get your act together.

Re: Canon's cloud platform has lost users' files and can't restore them

#413

Post by nix23 » Thu, Aug 20, 2020, 4:08 PM UTC

virgilp wrote: post #411

nix23 wrote: post #393

Earlier quoted context omitted.

>you have to delete backups, can't keep them forever >>The key issue is to put the backup data ‘beyond use’, even if it cannot be immediately overwritten. It's clear that you cannot delete WORM-tapen with many other data on it. >This is just opinion. Not for a file hosting service, the customer trusts you to do your job. >If it gets destroyed, is it a backup? Destroyed means physikal interaction, since it is not poss…

Why did you stop quoting after " immediately overwritten" ? > ie that the backup is simply held on your systems until it is replaced in line with an established schedule Again, you can't keep backups forever, you are _required_ to delete them. Also, not sure why you keep you backups in the bank (do you, really?) but even banks can suffer from fire/arson, explosion (accidental or not), or a number of other natural dis…

Really depens on the industry and the country (Finance is 30years, Health-Insurance until the death of the person (in switzerland)) You cannot go to you finance institute and ask them to delete all you records...that will never happen.

>Also, not sure why you keep you backups in the bank

Business backup's, secure, really good fireprotection, flood protection and in case of a Bus-hit-scenario someone else has access to it.

>Can you give an example of a business where customers don't typically trust you to do your job?

Politic and Canon filehosting from now on, Airbus Max and Foxnews ;)

Re: Canon's cloud platform has lost users' files and can't restore them

#414

Post by RantyDave » Thu, Aug 20, 2020, 10:28 PM UTC

rsync wrote: post #328

crazygringo wrote: post #271

Earlier quoted context omitted.

Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must…

"Do any cloud providers create backups on top of replication though?" We do.[1] For exactly 1.75x our normal pricing, we will replicate your entire account, nightly, to a geo-redundant site which is not open for normal customer use (and, therefore, has a lower risk profile). This GR site is the he.net core datacenter, in Fremont. It's also worth pointing out that replication to rsync.net buys you malware/ransomware p…

I've been wondering about ransomware protection through snapshots. Presumably (and I do know more or less nothing about it) the malware aspect of it is present on the system significantly before the ransomware aspect is triggered - so restoring to yesterday's backup just puts you back in the position to get pwn3d again? How do companies get around this?

Re: Canon's cloud platform has lost users' files and can't restore them

#415

Post by RantyDave » Thu, Aug 20, 2020, 10:32 PM UTC

rsync wrote: post #332

sandworm101 wrote: post #279

Earlier quoted context omitted.

>> we're talking maybe exabytes I don't see the size as the big problem. If one cloud can handle the size, the backup storage system can too. For me, the real issue is timing. How do you backup a cloud full of constantly changing data? Do you draw a line in the sand, an image of the cloud state at a particular moment? Is that even possible? You have to do backups of smaller chunks, individual accounts, but eventually…

"For me, the real issue is timing. How do you backup a cloud full of constantly changing data? Do you draw a line in the sand, an image of the cloud state at a particular moment?" I can only describe what we do, and, of course, there is an enormous scale difference here, but ... Every single rsync.net account is it's own ZFS filesystem which means every single account gets snapshotted[1] nightly on a schedule. This m…

So it does not assert that the filesystem is in a consistent state before it snapshots. IE we could be midway through applying a database transaction; or there could be a reference in one file to another that doesn't exist (yet).

It could be argued that one needs to do a site wide "write cache flush", stop, and snapshot. Not that I think for one second the service provider should (or could) be in a position to detect when a good time to snapshot might be....

Re: Canon's cloud platform has lost users' files and can't restore them

#416

Post by virgilp » Fri, Aug 21, 2020, 4:54 AM UTC

nix23 wrote: post #413

virgilp wrote: post #411

Earlier quoted context omitted.

Why did you stop quoting after " immediately overwritten" ? > ie that the backup is simply held on your systems until it is replaced in line with an established schedule Again, you can't keep backups forever, you are _required_ to delete them. Also, not sure why you keep you backups in the bank (do you, really?) but even banks can suffer from fire/arson, explosion (accidental or not), or a number of other natural dis…

Really depens on the industry and the country (Finance is 30years, Health-Insurance until the death of the person (in switzerland)) You cannot go to you finance institute and ask them to delete all you records...that will never happen. >Also, not sure why you keep you backups in the bank Business backup's, secure, really good fireprotection, flood protection and in case of a Bus-hit-scenario someone else has access t…

Right to be forgotten does not apply in cases where you're legally required to keep the data, or have other legitimate reasons to require said data, or there's significant public interest. But for _my_ photos, on a hosting service, it most definitely applies - if I request erasure, you can't keep them 30years in backup without breaching the law, that much I guarantee.

> Politic and Canon filehosting from now on, Airbus Max and Foxnews ;)

I know it's tongue-in-cheek, but Foxnews viewers typically trust them, that's one of the problems, right? And customers (advertisers) obviously trust them to deliver message to audience.

Airbus is better trusted than ever, especially now after the 737 Max fiasco :P - but even Boeing was obviously trusted by their customers (whether that trust was misplaced is a different issue; my point was "customers must trust me" is in no way unique or particularly important for file hosting).

Re: Canon's cloud platform has lost users' files and can't restore them

#417

Post by nix23 » Fri, Aug 21, 2020, 9:42 AM UTC

virgilp wrote: post #416

nix23 wrote: post #413

Earlier quoted context omitted.

Really depens on the industry and the country (Finance is 30years, Health-Insurance until the death of the person (in switzerland)) You cannot go to you finance institute and ask them to delete all you records...that will never happen. >Also, not sure why you keep you backups in the bank Business backup's, secure, really good fireprotection, flood protection and in case of a Bus-hit-scenario someone else has access t…

Right to be forgotten does not apply in cases where you're legally required to keep the data, or have other legitimate reasons to require said data, or there's significant public interest. But for _my_ photos, on a hosting service, it most definitely applies - if I request erasure, you can't keep them 30years in backup without breaching the law, that much I guarantee. > Politic and Canon filehosting from now on, Airb…

>Airbus is better trusted than ever, especially now after the 737 Max fiasco :P

True true

Re: Canon's cloud platform has lost users' files and can't restore them

#418

Post by MaxBarraclough » Fri, Aug 21, 2020, 1:58 PM UTC

DaiPlusPlus wrote: post #199

rightbyte wrote: post #170

Earlier quoted context omitted.

I have bad experiences encrypting backup drives. If they corrupt a little the data might be gone. You also have the problem of keeping track of the key for years.

Protip: write the passphrase in sharpie on the slither of whitespace left available on the drive’s spec sticker

What's the point in encrypting the drive if the key is written on the drive?

Re: Canon's cloud platform has lost users' files and can't restore them

#419

Post by traveler01 » Mon, Aug 24, 2020, 11:13 AM UTC

tinco wrote: post #124

traveler01 wrote: post #117

Earlier quoted context omitted.

I use the Cloud to store all my work and it's my way of backup. I trust Google's capacity to save my files. But, this being said, you SHOULD have other ways of backing up your data, like another HDD, storing locally in your PC too. Being too careful is never enough.

Sure, Google can save your files. But will it? and will you never lose access to Google? The problem with these big cloud providers is there are employees and robots with a big "disable account button", but there's no one you can call to talk about reenabling your account. I don't know what the odds are that Google is going to hit that "disable account" button on your account, it's probably incredible small, but if i…

Well I always sync my files between two PCs so I always have them stored on these. If anything goes wrong with any of these PCs (both) I always have Google. I also do some backups on both my externals HDDs so yeah I don't see all this failing anytime soon.