Live data from Hacker News

Canon's cloud platform has lost users' files and can't restore them

digitalcameraworld.com

291–300 of 419 posts

Re: Canon's cloud platform has lost users' files and can't restore them

#291

Earlier quoted context omitted.

That’s a lot harder to pull off. What methods do you use to accomplish this?

Well, once upon a time, when I worked at a smaller company, I kept a revolving collection of tapes at home. A slightly larger company also used tapes, and stored them at an undisclosed (to me) offsite location. A much larger company kept it all in datacenters, but the "offline" backups were disconnected from the WAN when they weren't actively being used.

In some cases there is additional requirements as well... For some types of data... HIPPA compliant backups for example must be stored in multiple locations at least 500 miles apart.

https://lmgtfy.com/?q=hippa+rule+%22500+miles+apart%22

Re: Canon's cloud platform has lost users' files and can't restore them

#292

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

> It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones.

Ironically my experience has been exactly the opposite. It's the demosaic-ed, fully developed copies of my photos that are larger and harder to preserve than the original RAWs. And these files can't be trivially regenerated from the RAWs, either. Let me explain.

The most important issue is that the process of taking a raw image and turning it into an edited RGB-pixel image is not obvious, at all. Tons of steps have to happen in this process, and there's currently no way to describe this process in a way that's compatible with a single open standard or even multiple pieces of software. At all. The steps can be broken down roughly into a series of "instructions", but what those instructions mean (e.g. to Lightroom) is entirely opaque and even secret in the case of closed source programs. Even open source programs, like RawTherapee and Darktable, use entirely different and incompatible approaches, algorithms, and instructions.

What this means on a practical level is that you can preserve your raw files and the instructions for replicating the edits as carefully as you like, but without the exact same piece of software (often even the same version of the program with the same settings and defaults set), your edits are as good as gone forever.

As a result I've had to take fairly drastic steps to make sure my photos are safe. For my older photos, I keep an entire VirtualBox image with Windows and Lightroom backed up along with my raw images, so that I can be sure of restoring exactly the same output files if necessary. (This more-or-less has to be a hacked version of Lightroom because you can't take chances with licensing problems preventing the program from running now that Lightroom is subscription based.) And actually for newer photos, I've moved away from Lightroom to RawTherapee. Even though I feel it's usually inferior, I feel safer since I know the pipeline from raw to burned edit is essentially public. I keep a backed up copy of the RawTherapee source code, but even if that somehow failed someone could make a RawTherapee compatible raw converter from scratch.

So that's why edited images are actually harder to preserve: whereas with raw you can just save them in 2-3 different physical locations and storage devices, to keep the edits you have to take several additional steps and there are more points of failure. Why not just keep the static edited images too? Well, I do that for my most essential photos. But that gets into the other issue, that the edited images are actually larger than the original raws.

If the point is preserving my edits, not just having a copy that's good enough for Facebook, the images have to be lossless. This pretty much means PNG or TIFF, and in fact the latter seems to be necessary since PNG doesn't handle metadata anywhere nearly as well. Unfortunately, while the compression used on raw files tends to be pretty good (which is further aided by the images being mosaiced), the compression algorithms compatible with TIFF are pretty terrible. Add that to the fact that you almost certainly want 16 bit images, in order to preserve as much raw detail as possible (in case you want high quality prints or need to do further editing), and you end up with whopping huge TIFFs. I regularly see my TIFF output files 4-5 times larger than the corresponding raws.

In short, managing backups for a high quality photography workflow is actually a good bit more difficult than it seems at first sight - and how it looks at first is not that easy either!

Re: Canon's cloud platform has lost users' files and can't restore them

#293

Earlier quoted context omitted.

> Store customer data immutable and encrypted, with a unique AES encryption key per customer. Encrypt that AES key again with a single RSA key-pair and store the encrypted AES-key in a database. Now you have the worst of both worlds. You also now have 2 points of failure where data can get lost, because if either has a problem you lose data.

I can't think of a mechanism where you can lose the customer's database records but can still recover files associated with that customer ...

Total systen compromise. The backup for the files worked. the ones for the DB didn't.

Re: Canon's cloud platform has lost users' files and can't restore them

#294
post #63

Earlier quoted context omitted.

Git is also incompatible with GDPR, you can't simply delete a file from all history.

Don't store your users PII in a git repository then.

The use case is to do dataset versioning for ML. The dataset itself is updated frequently. It would be nice to use a tool that can store efficiently when small changes are made yet allow versioning for reproducibility.

Re: Canon's cloud platform has lost users' files and can't restore them

#295

Earlier quoted context omitted.

I would add to that, that the offsite one should also be off line .

That’s a lot harder to pull off. What methods do you use to accomplish this?

Where I worked, weekly point in time backups were required. Those backups were put onto tape drives, those tape drives were set on a pallet and then driven by truck to an offline second location. IMO _that’s_ how it’s done properly.

Re: Canon's cloud platform has lost users' files and can't restore them

#296
post #187

Earlier quoted context omitted.

Or they could like, not. I think there's a world where corporations aren't given a carte blanche pass on human decency. I believe that world is nicer and where possible I work to give it room to exist; I boycott Disney because I believe what they did to the copyright system is wrong. I don't use Google or Microsoft products because I don't think they treat users with respect. (Minus youtube which I use a mirror, and…

> there was a 10GB/user cap, so it's unlikely someone had been using it as their only copy I don't understand why a small total size implies people are less likely to use it for their only copy of something?

I think because video and photos have the potential to be huge, especially in higher-end cameras, so it wouldn't take very long to exhaust 10gb of data, implying that they might be using a service that allows more for longer-term stuff (e.g. S3).

Re: Canon's cloud platform has lost users' files and can't restore them

#297
Japanese camera companies are really bad at cloud software - Sony Playmemories probably the worst offender. Also the in-camera "mini-apps" tend to be terrible.

They make good hardware but horrible software. They should really buy a computational photography company that also knows how to make good web and native apps, like https://skylum.com/

Re: Canon's cloud platform has lost users' files and can't restore them

#298

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must…

I’m sure this is dated but here is a write up in Google’s backup system from High Scalability - http://highscalability.com/blog/2014/2/3/how-google-backs-up...

Re: Canon's cloud platform has lost users' files and can't restore them

#299
post #105

Earlier quoted context omitted.

>but not all photos at the original resolution Thats why you backup the original data and NOT the downscaled etc one. >But how much are you really willing to pay, per month, for that safety? You have to backup ALL your customers data, otherwise DONT go into file-hosting business.

But they did backup the original data - the article states that the problem was with deploying wrong cleanup code on the long-term storage (that was supposed to be only deployed for the short-term storage). How do you guard against that? You could store everything in backup storage & never delete it, but that has implications: - privacy - legal (GDPR) - cost etc. It's basically a "black swan" event, they designed the…

These "old" files are deleted to save disk space. Instead of deleting data automatically, they could give users a disk quota and block uploads until the user explicitly moves or deletes files, eliminating all failure modes in which automated deletions go wrong: it's far from a "black swan" case.

Re: Canon's cloud platform has lost users' files and can't restore them

#300

Earlier quoted context omitted.

S3 bucket with object lock is as offline and convenient as it gets

I don't know much about S3, but isn't S3, by definition, online, not offline?

I’ve always wondered if Amazon backs up S3. I don’t think they explicitly say but I get the impression that it is the user’s responsibility to replicate to a second region to guard against data loss so I am guessing not. Object Lock wouldn’t protect against an S3 failure.
Post reply on HN