Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Canon's cloud platform has lost users' files and can't restore them

digitalcameraworld.com

Post reply on HN
Print view View original

281–290 of 419 posts

Re: Canon's cloud platform has lost users' files and can't restore them

#281

Post by nix23 »

virgilp wrote: post #253
nix23 wrote: post #105

Earlier quoted context omitted.

>but not all photos at the original resolution Thats why you backup the original data and NOT the downscaled etc one. >But how much are you really willing to pay, per month, for that safety? You have to backup ALL your customers data, otherwise DONT go into file-hosting business.

But they did backup the original data - the article states that the problem was with deploying wrong cleanup code on the long-term storage (that was supposed to be only deployed for the short-term storage). How do you guard against that? You could store everything in backup storage & never delete it, but that has implications: - privacy - legal (GDPR) - cost etc. It's basically a "black swan" event, they designed the…

> privacy

Encryption?

> legal (GDPR)

You dont have to delete offline backups...never heard of such a case

> cost etc.

Cheaper than loose your customer's data wich they trust you with.

EDIT: If you can delete a backup it's not a backup.

Re: Canon's cloud platform has lost users' files and can't restore them

#282

Post by kevstev »

crazygringo wrote: post #271
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must…

Just using AWS S3 as an example, you can copy a bucket to another bucket (preferably in a different region) with a policy of retaining all versions- preventing the problem of a delete whether malicious or accidental, from being reflected in the backup copy.

There is also the option of using Glacier for lower cost long term storage.

If you are asking does regular S3 provide a true backup capability out of the box, it does not, or at least did not last time I actively looked at this about 2 years ago.

Re: Canon's cloud platform has lost users' files and can't restore them

#283

Post by phatfish »

visarga wrote: post #63
chrismatheson wrote: post #59

Earlier quoted context omitted.

I suppose that feature would be incompatible with user data in a GDPR world ??

Git is also incompatible with GDPR, you can't simply delete a file from all history.

Don't store your users PII in a git repository then.

Re: Canon's cloud platform has lost users' files and can't restore them

#284

Post by hellofunk »

hvidgaard wrote: post #257
qntmfred wrote: post #200

Earlier quoted context omitted.

Companies of Canon's size can be constrained for resources just as much as any other company. Canon is largely a legacy company at this point, with significantly declining revenue the last several years. https://i.redd.it/lrlfen2x7sm31.jpg

Even more reason to spend a little bit more to make this right. It's not expensive to get a proper expert on the matter to figure things out. The last thing they want is bad publicity for things that could earn them recurring revenue.

I don’t think they deliberately thought they were doing something inadequately, these problems are always obvious in hindsight. Issue is that eventually most companies will have this moment seeing through the lens of hindsight, the reasons are always different.

Re: Canon's cloud platform has lost users' files and can't restore them

#285

Post by njharman »

ibeckermayer wrote: post #226
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

What’s the difference between replication and backups? Is the distinction that backups must be stored on separate infrastructure, whereas replication might still be 1 or 2 points of failure?

Replication includes replicating deletes (and overwrites, and subtilier changes). Replication is READ / WRITE.

Backups are WRITE ONCE. They persist if the original is deleted or modified.

Separate infrastructure and geographical distribution are orthogonal.

Re: Canon's cloud platform has lost users' files and can't restore them

#286

Post by MaxBarraclough »

amiga_500 wrote: post #268
MaxBarraclough wrote: post #265

Earlier quoted context omitted.

> For stuff you don't need to restore in a rush, it's cheap That was my thinking. It seems a good fit as a last-resort backup. Low month-on-month storage costs, high retrieval costs. So we're essentially betting that we'll never retrieve the data. Which seems fine. Also, it apparently has strong assurances against data-loss. Lots of nines. [0] > how my non tech wife could possibly have restored the files as the inter…

If you retrieve the data slowly it's cheap. It's expensive if say you are a retailer who needs their database restored asap. I use a Linux perl client!

I suppose the biggest risk is failure-to-pay, as with all cloud backups/storage. If I allow my payment card to expire, Amazon aren't obligated to continue to store my data. If I drop off the grid for an extended holiday, that could be a real risk.

To my knowledge, Amazon offer no means of prepaying.

Re: Canon's cloud platform has lost users' files and can't restore them

#287

Post by nicoburns »

syshum wrote: post #273
nicoburns wrote: post #267

Earlier quoted context omitted.

The fact that you don't see things like healthcare as a right is precisely my criticism of your system. Of course everything is subject to limited resources and external constraints. Enforcing property laws is subject to limits on policing resources. How is "the right to not have your property stolen" any different from "the right to not have preventable diseases from killing or harming you"? In both cases there is s…

>> Enforcing property laws is subject to limits on policing resources. Incorrect, in the US I have no right to policing either, I have the right to defend my property myself from people that wish to take it. Our Courts have held continuously that no individual has the right to protection from the police force. We do not have Positive Rights in the US, never have and hopefully never will because in order to have posit…

Courts don't just mediate. Their judgements are backed by the power of the state (otherwise a thief could simply ignore them). And this is absolutely somebody having a positive right (to have their property ownership defended) that infringes on someone else's negative right (to take their property without the state interfering). It happens to be a right that I agree with. But I don't think you can differentiate it with the positive/negative distinction like you think you can.

Re: Canon's cloud platform has lost users' files and can't restore them

#288

Post by reaperducer »

skoskie wrote: post #270
mumblemumble wrote: post #243

Earlier quoted context omitted.

I would add to that, that the offsite one should also be off line .

That’s a lot harder to pull off. What methods do you use to accomplish this?

That’s a lot harder to pull off. What methods do you use to accomplish this?

Depending on how much data you're backing up, sneakernet works.

When I was still in the office, my company had me rotate a set of backup hard drives between the office lockup and a strongbox in my house. The notion was that it was unlikely that both the office building and my house 12 miles away would both burn down at the same time.

Of course, now I'm working from home, so all of the eggs are in one basket again.

Re: Canon's cloud platform has lost users' files and can't restore them

#289

Post by hiptobecubic »

tpolzer wrote: post #127
jacquesm wrote: post #113

Earlier quoted context omitted.

And there won't be at this rate. Besides, it is now a net negative for their users. 'The cloud is not a backup' should be a mantra that everybody that uses the cloud for their work and personal data be familiar with.

The cloud is a backup. Just like any other backup you should test restore or at least verify the data regularly. I trust Google more than a random hard drive (to not lose random bits) in this respect. And unless your admin configured it away, GSuite allows takeout just like GMail afaik?

It's Gmail, not GMail. :)

Also, I believe all Google products have to support takeout, just like they have to support deletion within whatever the legally mandated time frame is (months?).

Re: Canon's cloud platform has lost users' files and can't restore them

#290

Post by reaperducer »

randoramax wrote: post #272
skoskie wrote: post #270

Earlier quoted context omitted.

That’s a lot harder to pull off. What methods do you use to accomplish this?

S3 bucket with object lock is as offline and convenient as it gets

I don't know much about S3, but isn't S3, by definition, online, not offline?
Post reply on HN